Between 3:30am and 10:30am on 4/24, requests for media files were intermittently resulting in "404 File Not Found" errors.  The outage was caused by human error / a mis-configured startup sequence on one of our media redirectors.  We apologize for the trouble.  The problem was discovered around 10am and resolved by 10:30.

As promised yesterday, we have pushed out a new version of stats interface. Responding to many user's feedback, we have added:

• episode columns sorting
• CSV downloads of entire episode grid
• access to apache/XML logs

In addition, we have changed some behind the scenes stuff to improve performance and stability. Please leave us some feedback on the forums

Libsyn stats v.1.6 has been out for about a month now. The feedback we have gotten so far has been very positive and many suggestions we received are either in the works or planned for the near future. If you haven't already, please feel free to jump into the discussion on the forums. We are continuing to improve and make fixes to the current system, and will roll out a new release over the weekend.

The development of our next generation stats system has been very exciting. We really can't wait to show off everything stats 2.0 will do! While we progress to that stage, there are some more measures we need to take to improve performance and stability. The first is to purge some old XML and Apache logs. If you are currently using the apache or xml logging feature, any log file older then January 2008 will be purged from the system on 4/18/2008. Following that date, the first of the month will mark the rollover period whereby only the oldest month will be purged. (eg. on 5/1/2008, February logs will be removed,etc, etc..). The download links for the logs will be moved to the 1.6 stats interface and the 1.5.x interface will be officially phased out. The access to historical data back to September 2006 will still be available through the stats interface- nothing is being removed from there. The vast majority of libsyn users will not be affected by this data retention policy.

Finally, while we are repairing the issues being reported of recent episodes not being picked up, the stats display may appear to be a bit erratic. We are restoring to a backup from earlier in the week and then reprocessing the weeks worth of data. We anticipate everything to be back to normal late this evening into the early morning hours.

Thanks to everyone for their feedback and suggestions in helping us build a better stats experience for all users.

---

update 4/11 12:30pm

The prep work for the brief rollback we were planning for early this morning took longer than expected.  We will be performing the rollback this afternoon and will post back here when it is complete.

---

update 4/11 6:20pm

The rollback is complete.  Stats should be current and up to date.  We've recovered the missing data from over the last few days (thought there may be slight amounts of irrecoverable data from last weekend).  Stay tuned also for some requested features we'll be pushing out in short order.

We have been tracking down the bug(s) which is causing the last episodes published by some users.  We will keep you published as we learn more.

---

update: we have temporarily stopped the stats engines- meaning no new updates of any files will occur for the time being. We are working on a plan to recover the data for those users affected over the last several days worth of data.

We do not want to jump the gun on pointing the finger for what happened, it would seem that the recent interruptions with Amazon EC2- which we take advantage of for our stats processing- has caused some unexpected behavior. This would also account for why a swath of users across only some of the partitions were affected.

---

Update (4/8/2007) 9:00PM: Stats processing resumed earlier this evening. Everything appears to be caught up, and for the most part, missing downloads have been restored. It is possible that a subset of new files got missed as we could not tell the exact time in which each EC2 engine got disrupted. We took a time from 1200 GMT on friday as our roll back point. There were measures taken to mitigate double counting of downloads for files which were tracking fine over that time period, but it is also possible that some double counts occured.

We have repaired this mishap as best we can and are moving forward with the work on the next generation stats system which will handle these types of issues in a much better manner. I apologize for the inconvenience this has caused.

We are currently experiencing some database issues which is causing problems connecting and uploading new media. We are working to restore service as quickly as possible

update: We will continue to monitor the load on the databases. The disruption was intermittent over the course of a couple hours in the evening on Monday. The team reacted quickly, resolved the issue, restoring the service to those experiencing problems logging in, navigating the website, and uploading media. If you uploaded any files between 9:30 - 11:00 PM EST and are either not seeing them show up in your account or are experiencing 404 file not found errors, you may need to try uploading again.