Second, there were some upgrades made today to the libsyn.com stats display system to improve compatibility with Firefox 3.0. This code change caused some temporary issues for some users loading their stats. There was no interruption in stats collection or processing during this time- only an unusually slow load time.

Both issues have been resolved.

This issue is being addressed and we are working to detect the incomplete files and replace the corrupted or cut-off files from backup. Depending on the media type, sometimes the result of this bug will be a file that is just cut-off, or in the case of certain video codecs, the movie not playing at all. Quicktime reports "invalid public atom was found in the movie" in some cases, however in iTunes the file just doesn't show up.

While we are making a sweep of the terabytes of data we have stored, if you are experiencing "weird" issues like these please let us know immediately and we will investigate your issue specifically.

We are running some scripts now to recover as many of the files from backups as we can. Unfortunately there will likely be files which will need to be re-uploaded. We will contact the users directly who this pertains to via email.

There have been extra validation checks put into place to ensure this will never happen again.

Stay tuned for more updates to come.

The stats processing is currently going through a massive overhaul. Part of the upgrades will include some new hardware. We understand the importance of timely, accurate statistics, and second to reliable media delivery, it's the thing you all really care most about. We're on it. it's going to take a little time, thank you for understanding.

Fortunately (and unfortunately) the issue was the same as last time. The error we made this time, though, was that the fix was not properly incorporated into our build script for new media nodes. Just as we though we were helping the network by bringing on 30% more hardware, it turns around and bites us for not installing all the proper fixes.

The patch that had been applied to the previous nodes has now been applied. I don't know how many more times you're going to accept our apologizes on this (the most critical or critical) error. We now need to make sure the stickies around our monitors have prominence that say "Add to the new build script!!!" on every new fix and tweak.

Sorry to start off your day with this. Our hope is that we caught the issue in time as to not have affected too many downloads.

Again, your account will look like it's over quota and disabled from uploading, however you ARE still able to upload while we fix the archiving situation

Thank you and sorry for taking an extra couple days to resolve this problem

Since we introduced the public RSS feed option and the ability to redirect your-username.libsyn.com/rss to that address, some of you have taken advantage of this feature to redirect your libsyn feed URL to your feedburner feed. This caused a slight problem, since feedburner gets your feed information by checking your libsyn feed (thus creating an infinite loop).

We got together with the folks at feedburner to fix this problem as soon as we leanred of it. If anyone is still having problems with their feed, please drop an email to support.

Thanks.

The main libsyn media delivery network has been greatly tweaked and enhanced to provide seamless data delivery in the event of hardware or software failure on any of the individual nodes. Currently serving over 2 million media files a week, or approx. 7 terrabytes of data, its been pretty good for the most part. Outside issues dealing with databases, general network outtages, and file uploading aside, the libsyn media delivery network has had a pretty good track record with download speeds usually topping anything we see from other media websites. This mornings issues have caused us to reevaluate the handling of the archive network and upgrade it to the level of robustness we have in place for the main media network.

Bottom line, for those of you that can't reach libsyn send us your ip addresses. We are passing those off to our datacenter guy who is manually rerouting how our site gets to you. Also important, according to our charts your shows are being served to MOST of your audience.

This apparently has never happened before. It's not just us either, parts of the internet can't get to other parts of the internet. Big companies being Big Jerks. I need coffee. below is better explanations by my more eloquent buddies.

Here's the drafts:

Marty put it like this:

Hey everyone... libsyn is up and running and has been since the migration last Sunday night. What you're experiencing is the result of a corporate catfight between 2 major networks who facilitate large amounts of inter-connectivity across the net.

Dave C said this:

Dear libsyn customers-

Sorry to invade your inbox with more email, however we wanted to let everyone know we are ontop of the current issue going on in Internet land. "This is history in the making" says our ISP. http://ask.slashdot.org/askslashdot/05/10/05/2247207.shtml

We are working to get around this issue which mostly seems to be affecting TimeWarner Roadrunner users specifically by taking the IP addresses of the users being affected, looking at the IP block of their provider, and manually adjusting our BGP tables. If all that went right over your head, no problem. Bottom line: if you are having troubles connecting to libsyn.com, we are not offline. Please email support@libsyn.com with your IP address, and we will work our best to reroute the connection so you can get in.

Updated Wed 8/31 2:39am EDT

We were able to trace this problem back to FTP uploads (usually of large files). In rare cases, files were getting distributed out to our media nodes before the FTP upload was complete. The problem would eventually fix itself as the file completed and the completed version would then get propogated throughout the network, but the lag time for this to happen was undesirably high (upwards of 20 minutes).

We've taken steps to a) speed up the propogation of files throughout the libsyn media network, and b) not deliver partial uploads, but rather deliver the original file from the main libsyn server instead.

If anyone notices truncated downloads, or hears from their users that a file they downloaded from libsyn was cut off, please let us know right away. We beleive this issue to be resolved, but if there's something we missed, we'll want to fix it immediately.

We are getting occasional reports of file downloads being cut-off. We are working to try and figure out a connection between the incidents of file truncation. Please email support@libsyn.com right away if you are experiencing similar issues.