We’ve have to take FTP offline to do some unexpected maintenance – we will update this post when it’s back up.
UPDATE – 10:40AM EST – FTP is back online and quickcast functionality should be working again.
We’re getting reports of assets not loading for some users and we’ve been able to replicate this issue ourselves. It does not appear to be affecting all users so it might just be an issue with a particular node on the content delivery network. We are escalating the issue with them and will report back when we know more.
UPDATE – August 4th 2011 – 12:00AM EST – Our content delivery network provider has tracked this down to being a single node in their Washington, DC area cluster that’s causing the problem. We expect this issue to be resolved shortly and will update this post once we recieve confirmation from our provider that the issue is fixed.
UPDATE – August 4th 2011 – We received notice at approx 12:30AM EST this morning that this issue has been resolved.
UPDATE – August 4th 2011 – 9:25AM EST – We’re seeing different errors with reports of users getting 401 unauthorized errors when trying to load assets. We’re escalating this with our provider immediately and we apologize for the trouble.
UPDATE – August 4th 2011 – 4:45PM EST – Assets on blog pages and external sites should be working again – we have failed them over to our backup system while the content delivery network gets their act together. Unfortunately assets in RSS feeds are still not loading – we may fail those over to the backup systems soon as well. We sincerely apologize for these troubles today and will plan in the future to fail-over assets to backup systems much faster.
Note: having assets failed over to the backup system means newly uploaded assets may take a while before they begin to load properly.
UPDATE – August 4th 2011 – 7:30PM EST – All assets should now be serving correctly – we apologize for the problem and now have systems and procedures in place to prevent this sort of issue from occurring again.
This morning while performing regularly scheduled updates on our servers there was a brief content delivery outage. This was an unintended outage and our developers worked immediately to resolve the issue. All issues have been resolved and we apologize for any inconvenience caused.
We are currently experiencing an extremely high and unusual load on our web-engines (the systems that generate feeds and blogs) – we have put the system into maintenance mode while we investigate.
UPDATE – 7PM EST – We’ve been able to mitigate the bad incoming traffic and have turned the logins back on. All systems are back to normal now.
We’ll be doing some system maintenance at 8AM EST on Monday. We expect this maintenance to be completed quickly but we are setting a 30 minute time window for it. During this time logins and FTP will be disabled, while media downloads, feeds and blog pages should be unaffected. We will update this post on Monday morning when we start the maintenance and then afterwards when it’s completed.
UPDATE [850 Eastern]: Update went smoothly, everything is up and running as expected. If you notice anything unusual, please contact support.
We are currently experiencing a delay receiving raw logs from our CDN, which has stopped stats processing. Check back for an update as soon as we find out when log delivery will be restored.
UPDATE: [1730 Eastern] CDN technicians have been looking into the problem since early this morning. We are still waiting to hear back from them about log delivery being restored.
UPDATE: [1820 Eastern] Our CDN has informed us that the issue was related to a hardware failure. They have restored logs from yesterday and today from an unaffected backup system and we are in the process of downloading these logs. Processing should commence within the next hour. Over the next day, stats will be backfilling and will be caught up as soon as possible. We appreciate your patience and apologize for the inconvenience. If there are any changes, we will update this post.
UPDATE: [5/6/11 – 9AM EST] Stats processing caught up overnight and is functioning normally again.
Today at 2PM EST we’ll be putting the system into maintenance mode for roughly 10 – 15 minutes. During that time we’ll be beginning some changes to our database infrastructure and automated failover processes that will prevent the issue that happened last night from occuring again.
2:15 PM EST – Today’s maintenance was completed successfully and all services are running normally again.
10:20PM EST -We’re currently experiencing some database problems which are affecting all our services. We’re working on it as fast as we can and will report back when things are up and running again.
UPDATE 10:30PM EST – media downloads are working again
UPDATE 10:45PM EST – all services have been fully restored – we apologize for the outage and will be looking at options to avoid this type of issue from occuring again in the future.
Libsyn has worked with Apple to identify how we could get our user’s show artwork updated more reliably. As a result of working with Apple we believe that we have identified a way that can get iTunes to refresh most of the artwork that was “stuck” or outdated. Many of the shows that were originally reported as having stuck artwork are now showing up correctly in iTunes. If your show still appears stuck, please be patient. iTunes servers take some time to refresh the images that they cache and these changes are still trickling out. Your artwork should refresh shortly, so keep checking back regularly.
It is finally Spring here and we are about to enjoy a Spring Break weekend!
Support for our services will continue to be available throughout the weekend, however our support team will be working on a reduced schedule. All priority issues will be addressed first but please allow for longer response times on requests. This allows our team to spend a little quality time with our respective families. Rest assured none of us ever stray too far away from the job, someone is always watching the gauges. As usual any system issues will be posted on the Support Blog.
We hope you all have a safe and happy Spring Break weekend!