Just wanted to update everyone on how the rollback process is going. We're happy to report that about 85% of the reprocessing is complete, and our testing is showing that issues with missing data are being resolved as this process progressess. We are also very excited to report that the new stats UI is working, tested, and ready to roll as soon as the re-processing is complete.

Our goal to get the new data to all of you this week looks like it's going to slip into next week. There was a lot of data to run through and it's taking longer then we had hoped. The good news is that the new UI, combined with some new hardware to back it, is a lot faster and more responsive than the previous UI. So not only will your data be easier to understand, but it should load much faster as well.

We will keep everyone updated as things progress.

Hi everyone.  We are happy to report that the system wide stats rollback / recovery is moving right along.  We expect the analysis and testing to be complete next week.  At that point we'll release the updated numbers and UI updates into your dashboard.  Thanks as always for hanging in there.

Over the past few weeks we've received a number of reports of users replacing existing files, but the new versions of those files never becoming available.  We tracked down the root of this problem (related to a recent configuration change with our CDN provider) and got it fixed.  Now, when you update an existing file, the new version should be available for download immediately, and the old version should no longer get served.

We are currently investigating an error with recently uploaded media returning an XML error file instead of the media. The problem seems to be with our backend service provider.

We are working with the providers to resolve the issue as quickly as possible.

Update: problem seems to be very widespread and affecting other customers of the amazon S3 service: details from Amazon Forum

The "origin" server for the libsyn CDN service is held on S3. It has historically been very very solid, and I'm sure will have the service back up very quickly. As of 9:30 EST they have not provided an ETA.

---

Update: Files being uploaded will not be available for download righ away as a result of this issue. Once everything is resolved on amazon's end, we will reload all the recent uploaded files.

Thank you for your patience.

---

Update: As of 10:30 EST, the CDN is now available again, and all files should now be available. If you notice problems with any media files, please contact support@libsyn.com.

 
Dear Users,

We have an update on the status of libsyn.com stats! We know stats accuracy is an ongoing issue and that libsyn users have had to exhibit a lot of patience and swallow a lot of frustration as we've worked through the problems over the years. As we said in a previous post, we have assembled a solid roadmap towards achieving stability and accuracy, and providing industry-leading statistical reports that tell you producers and network administrators exactly what you need to know about the consumption trends of your content.

As promised, we took the first step of identifying and fixing the issues with the current libsyn.com stats (mainly, some episodes published after July 2007 not showing up in the system, or suddenly stop reporting downloads). We sifted through the code that processes the raw data into data we can use in the reports we provide to you, and believe we have identified the root of the problem, as well as the fix for it. We've set up a test environment, applied this fix, and are now in the process of re-processing the raw data from Summer 2007 forward and verifying that this "fixed" data is accurate. Depending on how that verification goes, the reprocessing could be finished in as soon as 7 - 10 days.

While fixing the existing system is an important step, we are really excited about the development that is getting underway for a longer term solution. As we've said in the past, the current iteration of the stats application was built to be scalable, but it just has TOO many moving parts and missing pieces (for example the ability to speedily re-process data, or easily rollback and re-process from an arbitrary date). We are working on a plan that will provide scalability, flexibility for us in terms of administrating the application, and will give us the ability to continue to store historical data indefinitely (a wall that the current system was bound to come up against sooner or later). This is the "data warehouse" model we discussed in a previous post, and is a key component to our long-term plans for the libsynPRO enterprise service. The good news for all of our independent (i.e. libsyn.com) users is that the "libsynPRO" stats engine is also going to be the libsynINDIE stats engine and we are fast tracking the development and integration into both systems. It's still a little too early to give an accurate deliverable date, but we are shooting for early second quarter of this year.

Finally, as a bonus we are working vigorously to implement an updated stats UI into the current libsyn.com dashboard. We were originally waiting for libsyn.com 2.0 to be done and in place before we started introducing new features, but the long-loading XML+Flash interface is pretty long in the tooth, and we figured out a way to shoehorn in the new UI into the old system, so we figured what the heck. We've beaten you guys up so much over the past year regarding stats performance and stability, the least we can do is SHOW you all the data we have been collecting but were unable to deliver in a meaningful way for so long.

So, that's where things stand. We will keep you posted as the reprocessing progresses.

Love, Libsyn

PS - you'll notice that the libsyn.com stats updates are now happening less frequently than normal. We've had to tone-down the update cycle for those during the re-processing, since both the live system and the system running in the background work off of some of the same data