[9:30 AM EST] – This morning we discovered that one of our stats processing jobs became stuck ~9 hours ago and failed to restart correctly. We have restarted the process and confirmed that stats processing is running normally again. We expect the stats processing to catch up over the next 4-5 hours. We’ll post an update when the processing is fully caught up again.
[4:45 PM EST] – Stats processing is now fully caught up and continues to work normally.
One of the network providers our data center uses is having issues on their network within the UK and elsewhere in Europe this morning. We’ve had our data center disable that provider for the time being. This should resolve any connectivity issues users were having. If you are still having troubles connecting to our services please email support.
It has come to our attention that there are issues and latency for subscribers to any RSS feed (Libsyn and non-Libsyn hosted) in the iTunes store. This can cause delays in accessing content for new subscribers as well as obtaining new posts from existing subscribers.
We have reached out to iTunes and are currently waiting a response. We will update this post with new information as it comes across.
UPDATE 10/4/2016 9:13am: iTunes is aware of the issue and are working to resolve the latency issues. The issue is still ongoing and intermittent.
UPDATE 10/5/2016 12:06pm: iTunes has continued to investigate intermittent delays, however, the issue seems to have alleviated at this time. If you continue to experience issues subscribing to a show, please contact support@libsyn.com.
[8AM EST] – Our FTP server is currently experiencing connectivity issues and will be unavailable for most users. We’re working with our provider where the FTP server is hosted and will post an update when we hear back from them.
[11AM EST] – The network switch connected to our cabinet failed and has been replaced, the FTP server is back up and running normally again now.
[11:30AM EST] – We’re doing some maintenance on our stats system today. Stats processing is currently paused we’ll update this post as our work progresses.
[2 PM EST] – Stats processing is running again now, we’ll update this post once it’s fully caught up.
[4 PM EST] – Stats processing is fully caught up at this time. Today’s maintenance brought additional hardware online to our stats processing to improve performance and support our continued growth. Thanks for your patience.
[11AM EST] – We’re seeing uploads be slow and sometimes fail to one of our cloud storage providers. We’ve opened a ticket with them and they are aware of the problem and have engineers working on it. You’ll see this as the ‘processing’ step of uploads takes longer than normal and may fail. We’ll update this post when we get an update from our provider.
[12:45PM EST] – The cloud storage provider is still working on the issue but at this time they haven’t provided any ETA on when it will be resolved.
[4PM EST] – Since our primary cloud storage provider is still having issues, we’ve removed them from our upload process at this time and our relying on our secondary cloud storage partner. We’ve verified that uploads are functioning normally again with this change in place.
[10AM EST] – We’re conducting maintenance on our stats system today. The purpose of this is to add additional hardware to our stats cluster, this will improve the performance of both processing and display of stats. Stats processing is currently paused and the display of stats in the dashboard may become unavailable at times. We’ll update this post throughout the day with updates on our progress.
[3PM EST] – The maintenance is progressing and is nearly completed at this time. We’ll post another update when stats processing has resumed.
[4PM EST] – Stats processing has resumed now and is functioning normally. We expect stats to catch up sometime late night or early tomorrow morning.
[7:30 AM EST] – We’re doing some maintenance on our stats systems today, during this time stats processing is paused. We’ll update this post as the work progresses.
[10 AM EST] – Stats processing has been re-started at this time, we’ll update this post when stats processing is fully caught up.
[2:30 PM EST] – At this time stats processing is fully caught up and continues to run normally.
[12:50 PM] – We’re experiencing issues uploading to one of our cloud storage partners. This will result in uploads failing at the ‘processing’ step. We’ve opened a ticket with them and will update this post when we know more.
[1:15 PM] – We believe the issue has been resolved, we’ll continue to monitor the situation.
9:25AM EST – We’re aware of and are investigating issues with episodes and content not being properly published and released. We’ll post an update when we know more.
[UPDATE 9:45AM EST] – We have episodes releasing again now, there is quite a backlog though so it may take a little while to catch up.
[UPDATE 10:20AM EST] – We’re about halfway through the backlog of episodes at this point.
[UPDATE 10:35AM EST] – We’ve now finished processing and releasing the backlog of episodes. The releasing processing is running normally again at this time.
