It appears that we're having difficulty connecting to our CDN, which is causing issues with newly uploaded files.
We have opened a ticket with them about this issue and will keep this Support blog posted with updates.
Please let us know if you run into any other problems by sending us an email at support@libsyn.com.
Thanks

___________________________
UPDATE 5:05pm
This issue should be resolved. If you're still having trouble with any files, please let us know at support@libsyn.com
Thanks

There were some hardware issues this weekend that were causing sporadic download errors.
The disks that were causing the problems should be all clear now and the errors should not be occurring any longer.
If you're still having trouble, please contact support@libsyn.com
Thanks

We are running into some more problems connecting to our CDN this afternoon. We are in contact with them and hope the issue will be resolved shortly.
We'll post updates as soon as we have them. Sorry for any delays.
----------

Update 1:50pm.
These issues should be resolved. Please contact support at libsyn.com if you're still having trouble.
Thanks

We noticed that there were problems connecting to the CDN. This will  affect uploads until they resolve the issue.
We have a ticket open with them and are working to get the problem solved.
Please let us know if you have any further questions or problems at support@libsyn.com
--------------------

Update 8:00pm 6/8/09
Issues with the CDN should be resolved. Please contact us if you're still having trouble support@libsyn.com.

We are looking into some recent reports of 500 Internal Errors for Libsyn downloads. Sorry for any frustration. We will post updates soon.
Thanks
___________________

I'm sorry for the frustration. There  was a problem with one of our disks that was causing these errors intermittently. We believe the problem should be resolved.
Please let us know if you have any other questions or problems.
Thanks

We unfortunately are experiencing more issues with our CDN. This has cause issues similar to the issues earlier in the week: failed uploads, erroneous files...
If you are experiencing trouble please email us at support@libsyn.com and we will work as quickly as possible to restore you files.
Thanks

---------------------
Monday March 30th (5pm)
We are continuing to monitor the situation and have kept communication open with our service provider. All indications point to these issues being resolved as we have not encountered any new problematic files in the last couple of days. If you are still experiencing trouble with any files, please continue to keep us posted at support@libsyn.com

We have received (and been able to confirm) reports of downloads not working properly for some files.  The symptoms are file downloads timing out or producing a "503 Service Unavailable" error.  This is an issue with our content-delivery network provider and we are working with them to get the issues resolved as quickly as possible.

---

8:00AM 1/17/2009 update- The CDN team has reported a full recovery of the network and repaired the failing hardware which was causing the issues.

Several user are reporting that they are unable to download their files.
This may be due to the same issue that caused the outage late last night. We have contacted the CDN and apologize for the inconvenience while we investigate this issue.

Our network team is currently investigating backbone issues which are causing media download requests to time out. We appreciate your patience while we work on the fix.

---

Update: 01/12/09 11:00am  An issue with one of our service providers has been resolved, and all media files are now being served.  Please email support@libsyn.com if you have any questions.

Over the weekend several users occasionally experienced and issue posting new files. You may have seen the following error:

* There was a problem adding ___file___ to the Libsyn Network. (false) Our team has been notified of this problem and will notify you if we need additional info to fix it.

If you were seeing that message, you can login and reupload now without any problems.

Even if you were not seeing that message, when you attempted to download new files, you may have been getting a 500 Internal Server Error message from time to time. That issue was also fixed yesterday.

Both issues were intermittent and it took a few days longer to track down with the holiday and everyone traveling. We were also slow to update this blog with the issue. Please accept our apologies.

An issue has just arisen with one of our providers which is causing a Redirect Loop error. We've contacted our provider and are working on the issue. Thanks for your patience.

---

Update: 11/25/08 1:30am EST

We have resolved the issue with one of our service providers that was causing the 'redirect loop' error.  All files are now being served.  There was no interruption in data collection.  Please contact support@libsyn.com if you have any other questions.

Between 3:30am and 10:30am on 4/24, requests for media files were intermittently resulting in "404 File Not Found" errors.  The outage was caused by human error / a mis-configured startup sequence on one of our media redirectors.  We apologize for the trouble.  The problem was discovered around 10am and resolved by 10:30.

We are currently investigating an error with recently uploaded media returning an XML error file instead of the media. The problem seems to be with our backend service provider.

We are working with the providers to resolve the issue as quickly as possible.

Update: problem seems to be very widespread and affecting other customers of the amazon S3 service: details from Amazon Forum

The "origin" server for the libsyn CDN service is held on S3. It has historically been very very solid, and I'm sure will have the service back up very quickly. As of 9:30 EST they have not provided an ETA.

---

Update: Files being uploaded will not be available for download righ away as a result of this issue. Once everything is resolved on amazon's end, we will reload all the recent uploaded files.

Thank you for your patience.

---

Update: As of 10:30 EST, the CDN is now available again, and all files should now be available. If you notice problems with any media files, please contact support@libsyn.com.

After spending the better part of the day yesterday trying to understand what was going on with the delivery network (for anyone who didn't notice, requests for newly uploaded files were failing with 503 and 504 errors), we were able to trace the problem to one of our service providers who had rolled out some new, less than stable code for our account.

Normally we would let you all know if we were planning an upgrade or anything that might interrupt service, but this particular upgrade happened without notification for us, so we couldn't notify you.  We apologize for the inconvenience  the problems caused, and you can be sure we are making it very clear that something like this cannot happen again.  The availability of your files is vital to both your livelihood and ours.

We finally were able to get the new code rolled back, which almost immediately fixed the problem, around 12:30am this morning.

If you find you are having problems with any of your files, please open a ticket with support, and make sure you include the URL of the file in question.

Thanks


 

We are currently experiencing issues with our file delivery CDN. You may experience intermittent issues as we are working with the data provider to fix the issue.

More updates to follow.

update: We have had been working all day between our origin data center and our CDN to resolve the issue. The problem continues to plague new files uploaded, and older files that may not have been downloaded in a while. We are looking to put some stop-gap fixes in place while the issues with the CDN get resolved.

We apologize for the inconvenience this issue has caused.

Some listeners may be noticing slow-ish downloads when requesting a file from the archives.  This is a capacity issue that is currently being addressed.  Basically a section of our archives is nearing its capacity, so we are taking steps to expand the infrastructure that handles that piece of the archives, which will resolve the current slowness and allow much more room for future growth.

This issue is intermittent and potentially affects about 20% of archived files. We expect to have a short-term fix in place sometime this afternoon.  Long term build-out of our infrastructure is continuing.

This was a bug in the new media delivery mechanism we recently rolled out.  We've addressed the immediate problem and are taking steps to prevent similar incidents in the future.  Our apologies to anyone this affected.

One of our datacenters is experiencing some downtime. Media delivery has been adjusted to compensate, but there may be intermittent download problems.
This will be remedied as soon as possible, and this post will be updated with further information.
Thank you for your patience.

We are experiencing some unusual traffic spikes which are causing some issues with downloads this morning.  We are working hard on the issue and hope to have a resolution very soon.

---

Update: We are back up to full capacity after the team spent the last couple hours fighting off a DDOS attack.  We are very sorry for the interruption this has caused.  It took us a little bit of time to discover the exact problem and then to combat it.  We will be watching everything very closely to ensure we've covered it all.

For some of the same users affected last night by the hardware outage, there appears to be some residual issues regarding incomplete uploads propagating around the network.  We are in the process of ensuring integrity across the entire delivery network and running some scripts by hand to ensure the full copies of all files are making it out to all the corners of the globe.

We understand the seriousness of this issue- delivering partial files is almost as bad-if not worse- then not being able to deliver at all.  This has our full attention and the team is working hard to get this done quickly.

There are a few issues this morning which are causing requests for media files to timeout.  The team has been working the issue for a few hours and are putting in some new hardware to handle the post-christmas load.  There are also some databases that need to be cleaned out.

We'll update when things are back to normal.

Sorry this post is late coming out.  The team is working on some issues surrounding a flood of requests on part of the network.  Many users are affected and we're working the issue.

Updates to be released as we figure it out.  Thank you for your patience.

---

Update:  The segment of our network seeing the problems today was caused by a propagation issue.  New files were not getting out to delivery nodes quick enough which was causing a backup in requests, slow downloads and 404- file not found errors for new files.  All the mounting traffic then started to cause older files to be hard to access as well, while the network was absorbing lots of requests it couldn't fill.

The nodes are "catching up" now with the new media, and will be turned back on as soon as that is complete.

The technical reason, for those who are interested, is being linked to the network cards on these media nodes.  We are checking into the reasons why they seem to have issues, and will replace the hardware, or update the os, etc as needed.

As always, we are sorry for the disturbance to your audience for these issues, and regret it took us so long to get to the bottom of the problem.

Hello.
We received some emails reporting slow downloads. We've noticed some congestion on one of our servers and are looking into the problem as this post is being written. We hope to have it cleared up soon.
Thanks for your patience.

Hey guys, this we are still working on this thing. We still do not have a handle on what exactly is happening, but basically we are being hammered by connections that are overloading a specific part of the network. We worked on this all yesterday, banning bad ips and clients and are contining to work on this today. Sorry for the lack of communication. Thats my fault as I have been running around all day but marty and the guys are glued to their command lines. We'll keep you updated and get this resolved asap.

dave mansueto

Update: We've made some major changes to the network that we feel will balance things out much better. The distribution and balancing technique we were using needed some updating. We're going to need to see tomorrow at peek times to see if we really have things tweaked perfectly.

As luck would have it, while doing some other general maintanence over the weekend, where we were adding some redundency to other key parts of the system, the back kinda broke on a segment of the network. Most new users (who've signed up in the last month or so) were affected, so we're not really starting out with our best foot forward with a lot of you. I hope you can understand that these kinds of things happen to everyone, and hope you stick with us- as you can be assured we're going to stick with you- through thick and thin. We're going to keep killing ourselves (and hopefully not each other) to ensure the best, most reliable, most stable system for all you content creators.
-Dave Chekan

It looks like we are still experiencing overload on parts of our network. The connections are swamping the media nodes and this is causing the 404s you may be experiencing.  We have been working on this all this evening, and are implementing  a fix.
We'll keep you posted.

Update:  Issues are back with the 1 partition. We are working to recitify the problem by moving resources around while also trying to stop the bleeding.  We understand the problems this is causing many of you and we're working hard to smooth things out.

Thank you for your patience.

Things are back to operating normally.  We are checking the logs for any strange behavior.  There were some misbehaving ips, but it is not yet clear if this was the cause of the network overload.  We will continue to analyze what happened.
Sorry for the interruption to those affected users.

Something is causing a branch of our distribution network to overload.  We are looking into this now, and will report back soon.

We apologize for the interruption in service this evening. While testing new features, we interfered with the main database. We have taken steps to ensure we can test these features in the future with no impact on service.

Systems are solid and back to normal, with heavy traffic catching due to the outage that affected a portion of Libsyn users.  Unfortunately a portion of statistical data was lost for a short increment of time.  This situation has been remedied and the tech team is hard at work ensuring your feeds are always working, regardless of the technical challenge.

We had a rough spell yesterday morning (Tuesday, April 13th) from around 7:30 until 9:30am.  Many files on the network were coming up as "Not Found." 

This was a regrettable side effect of a new measure we've been trying to fit into place over the past few weeks. The new measure is meant to give improperly formed links a better chance at finding their destination file.

While the implementation of this measure has been rough and presented us with a number of unexpected issues, the end result will be a more fail-safe and balanced network. 

Our apologies for the intermittent troubles.  We'll keep you posted here on the progress / any ongoing issues with the media delivery network.

Over the course of the next 24 hours, we will be migrating the delivery nodes of the libsyn system. The upgrade will allow for better performance and ensure more accurate statistics for those folks who inadvertently link directly to the edge nodes instead of http://media.libsyn.com/media/username/file. That is still the proper way to link to your media, but sometimes the improper links get out there.. maybe someone who likes a show copies the url and posts it somewhere else. We've seen an increasing number of these types of occurrences, and this will fix that. We've been trying different options for a while now, and have finally come up with a solution that we are confident with. You wouldn't believe the weird things that happen with some file types. Did you know if you pass a query string to a .m4a file, quicktime won't play it in a web browser? Ya, we found that out the hard way. So, as always, please bring to our immediate attention any delivery issues that you see over the weekend. We want to make sure the ship is tight before the monday morning rush.

We are still experiencing 404's on newly posted files. File propagation has not yet fully recovered from this weekend's issues. We are recommending that folks upload media, but hold out publishing to the rss feeds and blogs until the file has been confirmed to have reached the delivery network. You can do this in the "publish" tab by selecting "save draft", and then confirm that the file has propagated by checking http://media.libsyn.com/media/username/filename. We are sorry for the inconvenience and will keep you posted.

We regretfully had some issues regarding our system upgrades this evening and are working to recover. As a result, data delivery will be slowed while we restore connectivity to a large part of our network. We will keep you posted. We apologize for the inconvenience.

---

Update: (10AM EST)

The system has been restored to approx. 80% full capacity. We anticipate full recovery sometime early this afternoon. Historically, Monday's are intense traffic days, and that extra added attention will slow down the recovery a little bit.

Thank you for hanging in there with us.

---

Update: (8:30PM EST)

Of course we have to have a record day, today of all days, and it has caused the end-end repair of the distribution network to take a little longer. Currently, somewhere between 1/2 and 3/4 our users should be back up to 100%. Others will still be experiencing some slow downloads, occasional timeouts, and a few 404's from time to time while the network is healing. Specifically, new uploads are taking longer than usual to get out to the whole system. If you use and outside feed to link to your libsyn media, it may help a little if you upload, then hold back on releasing the file into your feed until it has made it around a little bit. (Especially you big shows) A good check is to hit http://media.libsyn.com/media/username/filename in a webbrowser, and if the node your directed to is not media0, then it should be ok. The higher the number the better (>media20 or >media30).

Hello Libsynners,

If you've been following this blog, you'll know that since the summer we've been taking steps to add redundancy and robustness to our media delivery network.

By robustness we mean the number of downloads the network can sustain at any given time (we currently, by our most conservative estimates, serve around 3.5 million files each week). We have been and contiinue to beef up our infrastructure as the demand for your programs increases.

By redundancy we mean that should any single part of our system fail, your files will continue to be served. A few weeks ago we made a big move towards this redundancy, seperating our media delivery network from the main libsyn site.

Note that links to your media should be updated to point to http://media.libsyn.com/media/your-username/file-name to provide your listeners with the best / fastest performance possible. Old links (http://libsyn.com/media/your-username/file-name) will still work, but will result in a slightly slower connection to your file).

In addition to that seperation of media delivery from the main site, we've also added a failover for the media delivery, so incidents like last night, where a faulty power supply resulted in ~1 hour of slow connections and timeouts, will be much less likely.

Please let us know if you have any questions about the new linking standard, or any of the recent changes to the libsyn network. Thanks as always for your feedback and support

- Your Libsyn Support Team

We are currently resolving some media node issues which is causing slow responses for media requests and timeouts. We will have it back shortly.

UPDATE:

A faulty power supply caused a disruption in part of our delivery system. We redirected around the problem while the power supply was being replaced. Now things are back in full. We will be implementing a redundancy here as well as better monitoring tools to prevent this in the future.

We apologize for the inconvenience.

Wow, you guys are getting popular! Our servers having been smoking since coming back from the holidays.. must be all those iPods everyone got for christmas. We are bringing more hardware online to handle the increased load. The biggest problem currently is slow starts for media files. We are going to try some things to fix that as best we can while the new servers are being built and installed. Thank you everyone for your patience.

Around 3pm eastern today, we encountered some propagation issues for newly uploaded files that led to some system instability. We are currently working on the problem. Some users that have recently uploaded files may encounter some 404s as the files make it to the distribution network. We will keep you posted. We apologize for the inconvenience.

Things are functioning much better. We'll let you know once its all clear.

We are temporarily disabling the promo-swapping feature. We are working with Kiptronic to resolve a bug and get it back online asap. The truncated file downloads some of you have been experiencing, we believe, may be related. Libsyn and Kiptronic are working hard to ensure smooth, seamless promo swapping operation. Thank you.

We have brought several more media nodes online tonight- both current media and archived data. Hopefully this will bring the overall speed of the delivery up all around the system. There are some speed issues with the blog pages that we have been noticing, and are working on making that better. We think there are caching things we can do in addition to rewriting some db stuff.