Stats processing is currently behind due to an error that caused processing to be halted. We’ve restarted the stats processing and expect it to be fully caught up in the next few hours.
Stats processing is currently stopped, this appears to stem from the database issues we had last night. We’re investigating and hope to have processing up and running again soon. We’ll update this post with details as we get them.
UPDATE – 3:10PM EST – We have rectified the problem that caused stats processing to stop. We’ve begun processing through the backlog which will take approximately 8 hours or so. We’ll update this post once processing is completely caught up.
UPDATE – 8:15PM EST – Stats processing is now fully caught up and has finished going through the backlog from earlier today.
At approximately 11PM tonight we received alerts from our monitoring systems and quickly discovered it to be an issue with our main database cluster. A fail-over had been triggered and while this did cause a brief interruption in services everything was still running smoothly. During the next hour we put the Libsyn platform into maintenance mode for approximately two separate periods of 10 minutes each. This was done to allow us to switch back to the main database cluster after ensuring that it was functioning properly. As of 12:05 AM EST we are running on our main database cluster again and all systems are functioning normally.
We received notification today from our datacenter provider that they will be performing a software upgrade on their core routing equipment at 5AM EST tomorrow Sunday June 16th. They are telling us to expect at most a 5-10 minute outage. Our admin staff will be online during this time and will be in contact with the datacenter and kept up-to-date as the work is done. We’ll update this post when the work begins and when it is completed.
UPDATE 11:30PM EST – June 15th:
We just received noticed that the scheduled network maintenance planned for tomorrow morning has been postponed and is now scheduled for next Sunday June 23rd
Stats processing is currently running about 9 hours behind due to an issue overnight that caused processing to stop. We have resumed the stats processing and will update this post when it is completely caught up.
UPDATE 2PM EST – At this time stats processing is completely caught up and is continuing to process normally.
Some listeners may notice a ‘504’ error when attempting to access content. As of this time, it appears to be happening only on rare occasions. We are aware of our issue and have contacted our provider where the issue is stemming. We hope to have this resolved as quickly as possible.
We will update this notice once the issue has been considered resolved.
UPDATE 10PM EST – At this time we believe the issue has resolved, it’s been several hours since we’ve seen any reports of the 504 timeout occurring. If you do still see this error please open a support ticket by e-mailing email@example.com.
There appears to be a delay in new episode content being entered into the iTunes store. We have been in contact with iTunes and expect episode updates will be operating smoothly again soon. This will only affect new content being entered into the iTunes store itself, content in feeds will continue to be syndicated out to subscribers immediately.
Stats processing was halted sometime yesterday evening due to an error, we’ve restarted processing and will keep an eye on the system to see if the error occurs again. At this point the error appears to stem from invalid data in a single log file that wasn’t properly stripped out. No data has been lost and we expect processing to catch up over the next several hours.
UPDATE 2:30 PM EST – Stats processing is fully caught up now.