There appears to be a delay in new episode content being entered into the iTunes store. We have been in contact with iTunes and expect episode updates will be operating smoothly again soon. This will only affect new content being entered into the iTunes store itself, content in feeds will continue to be syndicated out to subscribers immediately.
Stats processing was halted sometime yesterday evening due to an error, we’ve restarted processing and will keep an eye on the system to see if the error occurs again. At this point the error appears to stem from invalid data in a single log file that wasn’t properly stripped out. No data has been lost and we expect processing to catch up over the next several hours.
UPDATE 2:30 PM EST – Stats processing is fully caught up now.
Stats processing is currently behind by approximately 12 hours this is due to an error that caused processing to be halted. We’ve restarted the stats processing and expect it to be fully caught up in the next few hours.
We’re currently aware of an issue that has caused stats processing to be delayed. No data has been lost we’re experiencing a delay in processing of the log files. We’ll update this post when processing has been restarted.
UPDATE October 8th, 2012 – 1:30 PM EST – Stats processing has been restarted and we expect it to be caught up in about 5 hours.
There’s one last configuration change we need to make to the network that we missed doing earlier. This should only take about 2 minutes. We’re starting this immediately and will update when it’s completed.
UPDATE 12:00PM EST We’ve completed this last step in the maintenance, thanks for your patience.
We’ll be putting the system into maintenance mode at 11:45AM EST today to perform emergency work on the database. Asset and Media delivery will continue to function as normal but access to the Libsyn Dashboard will be disabled. We apologize for the short notice we expect the maintenance to last 15-20 minutes. We’ll update this post when the work begins and when it’s finished.
UPDATE 11:55 AM – The emergency work on the database has begun, we’ll update this post when it’s finished.
UPDATE 12:30 PM – We’ve completed the maintenance and have put the platform back online.
Stats processing is currently paused, we suffered a hardware failure in one of the stats servers and are in the process of failing over to a backup machine. We expect stats processing to resume at approximately 6PM EST today. We’ll update this post if that changes.
UPDATE – 2:30 PM EST - Good news getting stats processing running on the backup machine went much quicker than expected. We’re now processing the back log of data and expect to be caught up by the end of the day.
Our helpdesk system suffered a hardware failure earlier today causing it to be down for a large portion of the day, we apologize for the inconvience. If you did e-mail in a ticket during the downtime you should get the auto-response e-mail in a few minutes as the ticket system begins to download the backlog of incoming ticket e-mails.
The helpdesk and knowledgebase systems are currently backup and functioning normally again.
April 9th – 3:35PM EST - We received an e-mail from our CDN provider stating that they are investigating a possible issue with the storage device that our account is assigned to. To avoid any issues we’ve temporarily disabled uploads. We will update this post as we get new information and will re-enable uploads as soon as we can.
UPDATE – 4:10 PM EST – It appears that our storage on the CDN has come back up. We will confirm that the issue is resolved and then will re-enable uploads.
UPDATE – 4:25 PM EST – We are still waiting for confirmation from our CDN provider that the issue is fixed but we have put through several test uploads and they have all worked correctly so we have turned uploads back on.
March 20, 2012 – 9:45AM EST -You may have noticed that your stats haven’t updated since the 19th and you may notice that the “last updated” time stamp on the stats keeps changing and going backwards. Last week we became aware of a possible situation where we were missing logs from our CDN providers UK location. We began working with them and they were able to retrieve this missing data from their systems. Yesterday afternoon they gave us the missing data and we began processing it. We ran into a few issues processing the data and that set the processing back by quite a few hours. The missing data covers the following dates: March 8th – 15th. You can expect to see your numbers across those dates increase depending on how many of your downloads comes from the UK.
UPDATE – 10:00AM EST – At this point the backlog of the missing data is finished processing and we are now working through the remaining logs from the 19th and starting on the logs from today the 20th.
