This is an update on what is being done to improve the stats -

Over the holidays we will be changing where the statistics are collected.  Previously the libsyn service was hosted on multiple cdns; one that was managed in house and then two to three other cdn providers.  The statistics were gathered by the redirect media nodes, that would accept requests, refuse bad requests, redirect based on db references to where particular media files are hosted, and collect statistical data.  This was not an efficient architecture decision as the redirect nodes were handling too many tasks.  While all the stats data was being collected,  the system was also introducing  noise into our logs that would ultimately choke up the stats parsing engine.  This would cause the the stats displays to report improperly or fail to report at all (spinning beach ball).  We would then have to go in and remove the noise by hand.  This took a significant amount of time for the repairs and for the re-running of data.  We would then have to wait as the stats engine caught up to date.  The reasons we managed multiple cdns were to keep our costs down for our users, and to allow us to scale our system from the in-house distribution platform launched in '04 to one based on industrial solutions.

What has changed over the last couple of months is that we have migrated our network to a unified cdn partner.  Now all media is served from one system.  The media is no longer shuffled to different cdns based on demand (an inefficient overhead), or moved to slower servers when archived.  Our download performance has been greatly improved for all media.  We have also begun migrating our stats data storage from sql based databases to a higher grade platform.

We are now migrating the collection of the stats data off of the redirect nodes.  We will be collecting the statistics directly from our cdn partner.  This data output is much cleaner and we believe this will improve the performance of the stats crunching engines greatly.  We also have been working on a more streamlined reporting tool for the stats tab of your libsyn account.  We have to take it one step at a time; first the consolidation of media which is now complete, then the change of collection method to data directly from the cdn.  This should bring our performance to an industry best state.  When this is completed we can finally overhaul our stats interface based on our users input, and the experiences we have gained from Wizzard Media's advertising initiatives.

We are very positive about the plan we have in place and we will give you updates as they progress.  So much of the work has been behind the scenes architecture changes, and we realize our users cannot see any immediate benefits.  But we assure you we are committed to making libsyn the best publishing platform for podcasters.

We are happy to say that in early 2008 you will see the fruition of all the work that has been done to fortify the stats system.

We have discovered a break in the stats engine which occurred on friday, 11/30, which have caused updates to backup for most users over the weekend. The systems are being restarted and should pickup the updates over the course of today and into tomorrow.

There is also the standing issue of missing data dating back over the last several months. That is a longer process to fix, please be patient while we work on that as well.

We apologize for the inconvenience.

After spending the better part of the day yesterday trying to understand what was going on with the delivery network (for anyone who didn't notice, requests for newly uploaded files were failing with 503 and 504 errors), we were able to trace the problem to one of our service providers who had rolled out some new, less than stable code for our account.

Normally we would let you all know if we were planning an upgrade or anything that might interrupt service, but this particular upgrade happened without notification for us, so we couldn't notify you.  We apologize for the inconvenience  the problems caused, and you can be sure we are making it very clear that something like this cannot happen again.  The availability of your files is vital to both your livelihood and ours.

We finally were able to get the new code rolled back, which almost immediately fixed the problem, around 12:30am this morning.

If you find you are having problems with any of your files, please open a ticket with support, and make sure you include the URL of the file in question.

Thanks


 

We are currently experiencing issues with our file delivery CDN. You may experience intermittent issues as we are working with the data provider to fix the issue.

More updates to follow.

update: We have had been working all day between our origin data center and our CDN to resolve the issue. The problem continues to plague new files uploaded, and older files that may not have been downloaded in a while. We are looking to put some stop-gap fixes in place while the issues with the CDN get resolved.

We apologize for the inconvenience this issue has caused.