Some users may have noticed that stats haven't updated since: 2009-09-14 21:30:41GMT This is due to a hardware issue.
We are working on getting this issue resolved and will have stats up and running again as soon as possible.
There should be no loss of data, its only the machine that delivers the data that has been paused.
Please contact support@libsyn.com if you have any questions.
Thanks


----------
Update: 11:30pm EST
Stats were restored around 5pm EST today. Please let us know if you run into any lingering issues. Thanks

We have received reports that some Libsyn users are seeing an error message when accessing their accounts' 'Stats' display. There was a hardware problem with the Indie stats machine and our developers are taking care of it immediately. In the meantime, the Stats display may be slow or nonresponsive. We apologize for any inconvenience this may have caused. Please email support@libsyn.com if you have any questions.

A few users began to report low-than-normal stats yesterday afternoon (May 6th). We did some investigating into this issue and realized that there were some hardware issues causing stats to not get relayed to the display.
We've remedied these issues and are in the process of rolling back to before the hardware failed and will be restoring these stats to affected users over the next day.
During this time period stats may appear to not be working, when in reality they are just a day or so behind and will be catching up to the moment that the rollback is complete.
Sorry for any frustration.
Please email support@libsyn.com if you have any questions.
Thanks

Our development team has temporarily disabled the Stats display while our system updates data across all users. This process should be completed within a few hours. There has been no loss in collected data-- please email support@libsyn.com if you have any questions in the meantime. We apologize for any inconvenience this may have caused.

---

2/25/09 8:30pm est
Stats are continuing to catch up, but its taking a little longer than we expected.  They should continue to catch up through the night and be back to normal by the morning. Please email support@libsyn.com with any other questions.

---

2/26/09 9:00am EST
The stats display has been restored after collected data was updated across the Libsyn system.  Please email support@libsyn.com if you have any questions.

We experienced an issue with our stats system over the weekend, and our developers temporarily paused the stats engine while they fixed the problem. All stats data continued to be collected during that time. The stats engine was then brought back on-line. Most Libsyn users' accounts now have updated, current stats data. However, a small segment of our users who are grouped on one partition of our system will notice that their accounts are displaying older stats data.

---

Update: 2/18/09 9:00AM EST

The stats issue has been resolved for all Libsyn users.  Stats are processing again on all partitions-- the stats engine is now displaying updated data.  We apologize for any inconvenience this may have caused.  Please email support@libsyn.com if you have any other questions.

Some users reported that their stats display was slow to update earlier today-- we resolved an issue with our stats engine that was causing the problem.  The stats engine has been restarted, and updated stats totals will be displaying shortly.  If you notice any issues, or if you have any questions, please email support@libsyn.com.

Update 10/24/08 11:00am EST We have completed performing maintenance on our Stats system after experiencing a hardware issue earlier this week. Stats have been restored across the Libsyn system and are now displaying for all users. We were able to use our secondary sources of data to update the system, which resolved the issue. Please contact Libsyn Support (support@libsyn.com) if you have any other questions.

---

Due to some hardware issues that occurred earlier in this week, we are currently running some maintenance on our stats system. Stats will appear to be frozen, but in actuality are rolling back about a week. Users will see stats progress forward through the night until they are full caught up later tonight.
No data appears to have been lost.
Please let us know if you have any questions.
Thanks for your patience.

Update (6:00pm EST) - Our data center technicians have replaced a faulty disk and are in the process of rebuilding a RAID array. Stats data is continuing to be collected across the Libsyn system, and we expect full functionality to be restored by tomorrow morning at the earliest. We will continue to provide updates on the situation.

---

Update (10/03/08 1:00pm EST) - A faulty disk on the indie Stats log parser has been replaced. The Stats parser has begun to process data collected across the Libsyn system. Updated stats data will be available shortly.

The Libsyn Indie Stats updates are currently paused while backend maintenance is being performed on the hardware running this system. While numbers will not appear to change, all downloads are still being captured and no data is being lost. We will keep everyone posted on an ETA for the system to be back up. Once the Stats service has been restored, it may take several hours for the most recent Stats numbers to display while the Libsyn system updates user data.

---

Update (7:00pm EST) - The database update revealed hardware problems with one of the disks in the RAID array. The datacenter has not provided us with an exact ETA. Fortunately, the data that was processed up to that point was not affected, nor was data collection. The hardware is being serviced on-site and we expect to start processing in the very near future.
We apologize for the inconvenience and appreciate your patience. Our statistics system has made some huge strides in the past year and we expect to see even bigger improvements coming soon. One of the safeguards recently put in place is the ability to retain data in an event like this one. Again, no current numbers have been impacted and, as soon as service is restored, the processor will catch up quickly.

The stats page was temporarily unavailable this morning due to an internal issue.  There has been no data loss and all new stats were collected during the interruption. The datacenter team has restored the service to all users.  Please contact Libsyn Support if you have any questions.

The stats view is temporarily unavailable this morning due to server issues. There has been no data loss and all new stats are still being collected. The datacenter team is working as quickly as possible to restore the service to all users.  

---

UPDATE: The stats display issue has been resolved. Stats are now displaying for all users. Please contact Libsyn Support at support@libsyn.com if you have any other questions.

The stats view may be unavailable from time to time this morning while the backend database is being worked on. There has been no data loss and all new stats are still be collected. The datacenter team is working as quickly as possible to restore the service to all users.

---

Update (6:05pm EST): The server repair is taking longer then expected, the team is working through the night to repair the service as quickly as possible. The download data is still being collected and no data has been lost as a result of this downtime.

---

Update (11:15 pm EST): A replacement server has been brought online and about 60% of the data has been copied over. We are attempting to bring up a backup image of the database from yesterday to allow users access to all stats leading up to this mornings failure while the remainder of the data is copying. If you login and see that the last updated date is behind, rest assured that it's still copying, or processing and will catch up very soon.

As promised yesterday, we have pushed out a new version of stats interface. Responding to many user's feedback, we have added:

• episode columns sorting
• CSV downloads of entire episode grid
• access to apache/XML logs

In addition, we have changed some behind the scenes stuff to improve performance and stability. Please leave us some feedback on the forums

Libsyn stats v.1.6 has been out for about a month now. The feedback we have gotten so far has been very positive and many suggestions we received are either in the works or planned for the near future. If you haven't already, please feel free to jump into the discussion on the forums. We are continuing to improve and make fixes to the current system, and will roll out a new release over the weekend.

The development of our next generation stats system has been very exciting. We really can't wait to show off everything stats 2.0 will do! While we progress to that stage, there are some more measures we need to take to improve performance and stability. The first is to purge some old XML and Apache logs. If you are currently using the apache or xml logging feature, any log file older then January 2008 will be purged from the system on 4/18/2008. Following that date, the first of the month will mark the rollover period whereby only the oldest month will be purged. (eg. on 5/1/2008, February logs will be removed,etc, etc..). The download links for the logs will be moved to the 1.6 stats interface and the 1.5.x interface will be officially phased out. The access to historical data back to September 2006 will still be available through the stats interface- nothing is being removed from there. The vast majority of libsyn users will not be affected by this data retention policy.

Finally, while we are repairing the issues being reported of recent episodes not being picked up, the stats display may appear to be a bit erratic. We are restoring to a backup from earlier in the week and then reprocessing the weeks worth of data. We anticipate everything to be back to normal late this evening into the early morning hours.

Thanks to everyone for their feedback and suggestions in helping us build a better stats experience for all users.

---

update 4/11 12:30pm

The prep work for the brief rollback we were planning for early this morning took longer than expected.  We will be performing the rollback this afternoon and will post back here when it is complete.

---

update 4/11 6:20pm

The rollback is complete.  Stats should be current and up to date.  We've recovered the missing data from over the last few days (thought there may be slight amounts of irrecoverable data from last weekend).  Stay tuned also for some requested features we'll be pushing out in short order.

We have been tracking down the bug(s) which is causing the last episodes published by some users.  We will keep you published as we learn more.

---

update: we have temporarily stopped the stats engines- meaning no new updates of any files will occur for the time being. We are working on a plan to recover the data for those users affected over the last several days worth of data.

We do not want to jump the gun on pointing the finger for what happened, it would seem that the recent interruptions with Amazon EC2- which we take advantage of for our stats processing- has caused some unexpected behavior. This would also account for why a swath of users across only some of the partitions were affected.

---

Update (4/8/2007) 9:00PM: Stats processing resumed earlier this evening. Everything appears to be caught up, and for the most part, missing downloads have been restored. It is possible that a subset of new files got missed as we could not tell the exact time in which each EC2 engine got disrupted. We took a time from 1200 GMT on friday as our roll back point. There were measures taken to mitigate double counting of downloads for files which were tracking fine over that time period, but it is also possible that some double counts occured.

We have repaired this mishap as best we can and are moving forward with the work on the next generation stats system which will handle these types of issues in a much better manner. I apologize for the inconvenience this has caused.

We launched v.1.6 of the libsyn stats service yesterday. Here's a 7 minute demonstration video to see what they are all about. Please excuse the fact that I have a cold.. and am not, in fact, the EVP of Stats.

Issues addressed in this release:

1. lack of updates for new files
2. low numbers for past 9 months
3. speed to load stats UI
4. accuracy of data
5. user agents list (zune, appleTV, iPhone, miro, etc added)
6. real trending data over time.. per month/week/day downloads of episodes or entire show
7. general confusion from v.1.5.2

The "old stats" 1.5.2 will continue to be updated while "new stats" 1.6 continues to catch up for some users.  By the end of this month, we intend to stop supporting the old stats with any new updates and only the new stats will be getting new data.

There are currently 2 partitions which contain many popular users and therefore taking longer to re-process that last 9 months worth of data.  We've held off an extra couple weeks from releasing this version while we waited for all users to catch up.  Our feelings now are why make the 90% suffer for the 10% that are behind.  If you are unlucky enough to be in that 10%, please be patient while your data catches up.

For the most recent issues some users have been experiencing this week regarding slow or no updates, those engines are working to catch up.  You should see v.1.5.2 numbers updating over the weekend.

Just wanted to update everyone on how the rollback process is going. We're happy to report that about 85% of the reprocessing is complete, and our testing is showing that issues with missing data are being resolved as this process progressess. We are also very excited to report that the new stats UI is working, tested, and ready to roll as soon as the re-processing is complete.

Our goal to get the new data to all of you this week looks like it's going to slip into next week. There was a lot of data to run through and it's taking longer then we had hoped. The good news is that the new UI, combined with some new hardware to back it, is a lot faster and more responsive than the previous UI. So not only will your data be easier to understand, but it should load much faster as well.

We will keep everyone updated as things progress.

Hi everyone.  We are happy to report that the system wide stats rollback / recovery is moving right along.  We expect the analysis and testing to be complete next week.  At that point we'll release the updated numbers and UI updates into your dashboard.  Thanks as always for hanging in there.

 
Dear Users,

We have an update on the status of libsyn.com stats! We know stats accuracy is an ongoing issue and that libsyn users have had to exhibit a lot of patience and swallow a lot of frustration as we've worked through the problems over the years. As we said in a previous post, we have assembled a solid roadmap towards achieving stability and accuracy, and providing industry-leading statistical reports that tell you producers and network administrators exactly what you need to know about the consumption trends of your content.

As promised, we took the first step of identifying and fixing the issues with the current libsyn.com stats (mainly, some episodes published after July 2007 not showing up in the system, or suddenly stop reporting downloads). We sifted through the code that processes the raw data into data we can use in the reports we provide to you, and believe we have identified the root of the problem, as well as the fix for it. We've set up a test environment, applied this fix, and are now in the process of re-processing the raw data from Summer 2007 forward and verifying that this "fixed" data is accurate. Depending on how that verification goes, the reprocessing could be finished in as soon as 7 - 10 days.

While fixing the existing system is an important step, we are really excited about the development that is getting underway for a longer term solution. As we've said in the past, the current iteration of the stats application was built to be scalable, but it just has TOO many moving parts and missing pieces (for example the ability to speedily re-process data, or easily rollback and re-process from an arbitrary date). We are working on a plan that will provide scalability, flexibility for us in terms of administrating the application, and will give us the ability to continue to store historical data indefinitely (a wall that the current system was bound to come up against sooner or later). This is the "data warehouse" model we discussed in a previous post, and is a key component to our long-term plans for the libsynPRO enterprise service. The good news for all of our independent (i.e. libsyn.com) users is that the "libsynPRO" stats engine is also going to be the libsynINDIE stats engine and we are fast tracking the development and integration into both systems. It's still a little too early to give an accurate deliverable date, but we are shooting for early second quarter of this year.

Finally, as a bonus we are working vigorously to implement an updated stats UI into the current libsyn.com dashboard. We were originally waiting for libsyn.com 2.0 to be done and in place before we started introducing new features, but the long-loading XML+Flash interface is pretty long in the tooth, and we figured out a way to shoehorn in the new UI into the old system, so we figured what the heck. We've beaten you guys up so much over the past year regarding stats performance and stability, the least we can do is SHOW you all the data we have been collecting but were unable to deliver in a meaningful way for so long.

So, that's where things stand. We will keep you posted as the reprocessing progresses.

Love, Libsyn

PS - you'll notice that the libsyn.com stats updates are now happening less frequently than normal. We've had to tone-down the update cycle for those during the re-processing, since both the live system and the system running in the background work off of some of the same data

This is an update on what is being done to improve the stats -

Over the holidays we will be changing where the statistics are collected.  Previously the libsyn service was hosted on multiple cdns; one that was managed in house and then two to three other cdn providers.  The statistics were gathered by the redirect media nodes, that would accept requests, refuse bad requests, redirect based on db references to where particular media files are hosted, and collect statistical data.  This was not an efficient architecture decision as the redirect nodes were handling too many tasks.  While all the stats data was being collected,  the system was also introducing  noise into our logs that would ultimately choke up the stats parsing engine.  This would cause the the stats displays to report improperly or fail to report at all (spinning beach ball).  We would then have to go in and remove the noise by hand.  This took a significant amount of time for the repairs and for the re-running of data.  We would then have to wait as the stats engine caught up to date.  The reasons we managed multiple cdns were to keep our costs down for our users, and to allow us to scale our system from the in-house distribution platform launched in '04 to one based on industrial solutions.

What has changed over the last couple of months is that we have migrated our network to a unified cdn partner.  Now all media is served from one system.  The media is no longer shuffled to different cdns based on demand (an inefficient overhead), or moved to slower servers when archived.  Our download performance has been greatly improved for all media.  We have also begun migrating our stats data storage from sql based databases to a higher grade platform.

We are now migrating the collection of the stats data off of the redirect nodes.  We will be collecting the statistics directly from our cdn partner.  This data output is much cleaner and we believe this will improve the performance of the stats crunching engines greatly.  We also have been working on a more streamlined reporting tool for the stats tab of your libsyn account.  We have to take it one step at a time; first the consolidation of media which is now complete, then the change of collection method to data directly from the cdn.  This should bring our performance to an industry best state.  When this is completed we can finally overhaul our stats interface based on our users input, and the experiences we have gained from Wizzard Media's advertising initiatives.

We are very positive about the plan we have in place and we will give you updates as they progress.  So much of the work has been behind the scenes architecture changes, and we realize our users cannot see any immediate benefits.  But we assure you we are committed to making libsyn the best publishing platform for podcasters.

We are happy to say that in early 2008 you will see the fruition of all the work that has been done to fortify the stats system.

We have discovered a break in the stats engine which occurred on friday, 11/30, which have caused updates to backup for most users over the weekend. The systems are being restarted and should pickup the updates over the course of today and into tomorrow.

There is also the standing issue of missing data dating back over the last several months. That is a longer process to fix, please be patient while we work on that as well.

We apologize for the inconvenience.

We have been working to restart the log aggregator which has fallen out of sync. It will take several hours, but all data from yesterday and today will eventually catch up. Thank you for your patience as we maintain the current system while continue to rebuild the libsyn stats engine from the ground up.

We have been working to keep the stats engine updating regularly over the past week while we diagnose some hardware and configuration issues. There were some disk issues that we cleared up this afternoon and as of this evening we believe all queues should be caught up.

While we believe all the issues have been resolved regarding the queue of log files to process, over the next several days updates may slow down again from time to time.

The slow updates, or "stalled stats" issue is a high priority concern that the team is working on improving. Thank you for your continued patience and support while we rebuild our stats system from the ground up.

This article if not for the faint of heart. It gets a little long, a little technical, but it is not the type of thing we could throw into bullet points and be confident we've covered everything we wanted to convey. So, if you are totally happy with the way our service is working for you and don't have a lot of interest in stats, go about your business and just know we are working on making them much better and if you should ever need them, they'll be there for you.

For the stats connoisseurs, please read this forum post.

There were a set of logs that were being blocked from access by the log aggregator so they weren't getting parsed into the stats system... these were from 8/31 to present. We just pushed them all in today, so users with low hits may see that this resolves the problem... and users in general will see a spike in their hits today, especially for archived files.

We are investigating some more reports of lower-then-expected download numbers. There have been some adjustments made and we need a couple complete days of data to see if the graphs smooth out.

We will keep you posted as we learn more about what's going on and who may be affected.

Update from the stats investigation department: We have discovered a bug in the file filter which strains out extraneous files from the counters and it was causing valid hits to get skipped. We are rolling back a few days worth of data and rerunning right now. Expect it to recover pretty quickly and accurate numbers to follow.

Will update when we have an ETA

---

[update 8/31/2007 7:04am]

The rollback is complete.  Thanks to everyone who sent in detailed reports of inconsistencies they were seeing.  That kind of information was VERY helpful in getting to the bottom of the problem.

Finally!  After a long wait, I'm happy to announce the stats are caught up and once again running in near-real-time (there may be a 5-10 minute lag between when a file is downloaded and when it appears in your stats display).

Once again, if you would like to report any anomalies to the support team, please be sure to include specifics like the file names of affected files, your user name, and try to describe exactly which numbers seem to be off-kilter.  Writing us to say, "my stats look weird can you check it out?" only slows things down as our support staff will then need to try and decipher exactly what you are referring to.  We appreciate your cooperation on this.

Another update: based on the rate things are going, we expect the system to take about another 36 hours to become fully current.  That puts us at around 7pm Eastern, Tuesday evening.  Hold tight everyone, we're almost there.

---

[updated 8/15 7:28am ]

As some have noticed, our 7pm Tuesday night ETA was a little optimistic.  As we speak the whole system is catching up and is currently processing numbers for August 7th.   The speed of this catchup process is dependent on a number of of factors, so rather than giving another estimate which may or may not be right, I'll say that all pistons are firing, the stats engines are chewing through all of the raw data, and we'll post here again when we are completely up to date. 

In the meantime, you can check out your numbers for everything up until the 7th by looking under the stats tab of your libsyn dashboard.  Again: there will be inflated numbers for July 23rd and for some of you, July 24th.  These spikes represent downloads of your show that were fed into the system "late" so they were counted as having happened on the 23rd/24th when in fact they happened in the days and weeks previous to that. In total this was a few hundred thousand hits that were added after the fact, and this will affect some users more than others.

If you feel that something about your numbers isn't right, and you send our support team a report of it, please help us out by being as specific as possible.  Include your user name, the file name(s) affected, and the specific numbers that seem to be wrong.  This helps us greatly to figure out what, if anything, is going on.

---

[update 8/21 12:26pm]

Sorry folks, been neglecting updates on the blog.  There is a running thread on the state of the stats system here on the forums.

The latest:




 

The stats rollback / restoration was delayed slightly as Pittsburgh got hit was some severe thunderstorms which caused damage and electricity outages throughout the city.  The process is once again underway.  Stay tuned...

---

[ update: 1:15pm ]

Things are looking good, everyone.  It's taking a little longer than we expected to get this thing going because we found a couple of "gotchas" that would have caused the numbers to look erratic (hits being bunched into days they didn't belong in, mostly).  Though all the numbers would have been counted, we figured at this point, we should smooth out those wrinkles so that when your stats come back, they accurately represent your audience's downloading behavior.

I know we say it a lot, but thanks again for hanging in there with us.

[update 1:50pm]

Because this is a rollback, we are literally rolling the clocks back on the engine that reports data to your stats display. Effectively, what you see right now is the stats as they were mid-day on 7/23. Shortly, the system will start moving forward in time as it reinserts the missing data. Since the clocks have been set back to a couple of weeks ago, recent entries in the stats folder may have seemed to disappear. They should reappear as we climb back through time from July 23rd to the present.
There may also be a slight spike on July 23rd as we're inserting some data that never made it to the the entries from beforehand.
Thanks for your patience.

[update 11:50pm]

We're facing some more challenges.  After watching carefully as the collection service runs through ten full days of data, we are seeing what we believe to be inconsistencies.  Further investigation is ongoing.  There is a good chance we will be resetting / re-starting the rollback process more than once before the weekend is through.  We're trying our hardest to have fresh, accurate, up to date numbers for you all when the week begins.

Starting in just a few minutes, we are going to begin the rollback / restore of the stats system. Effectively, the system is going to be reset to July 23rd and will re-crunch all of the numbers moving forward.  Once this process begins, we'll take some measurements and update this post with the estimated time when we'll be all caught up again.

Hi Everyone- I thought I'd take a moment to give a quick update on the status of the stats system:

We are fairly confident we have wiped out all of the pesky bugs that were wreaking such havoc the past few weeks.  To be absolutely sure there are no gotchas we may have missed, we've opted to extend the testing into Thursday. 

We will update again with specifics on the rollback / restore when that process begins.

[ Originally posted at http://forum.libsyn.com/viewtopic.php?t=7743 ]

Friends,

I know the latest stats issues are causing a bit of an uproar, so I am pulling my head out from under the hood for a minute to explain what's going on.

We have discovered some bugs in the new collection service we put into place around the 23rd of July. (The collection service is the piece of the stats system that feeds the raw hits into a queue to be analyzed by the 'stats-engine'). We've been tracking these issues for a few weeks, and are fully aware that there is a serious problem with the numbers you are seeing. Some people are reporting super high spikes, others are seeing numbers that are way too low. Some see a combination of both, and still others are not getting stats data on some files at all.

We are currently peeling back the layers to find and repair the bugs that have caused these problems. Admittedly, we were probably a bit hasty in pushing the new collection service into production so fast. The old service, however, was on its last leg and could no longer scale to the level of demand being placed on it, so we felt like we had to make the leap. It's probably safe to say that we didn't quite stick that landing.

The good news is, we have isolated and repaired a number of obscure bugs. We have also narrowed the field on a major issue that is causing the collection service to intermittently crash (which can cause you to see low points, followed by extremely high points in your daily numbers).

Once we have routed out the last of these bugs, we have plans to COMPLETELY RESTORE the past data from ~7/23 to present. That is to say, we are going to roll the system back in time to the 23rd, and re-run the numbers going forward. This will eliminate the erratic reporting you've witnessed in the past weeks, and should restore the normal trends that you're used to seeing for your shows.

We know this is extremely frustrating for many of you, It is for us as well. While we've come a long way in the past few years, we are still learning to manage and scale a system of this size. So if you can find it in you, we'd really appreciate your patience and support as we work this one out.

Thank you, sincerely,

Marty
 

Unfortunately we have to report again that some users are seeing lower then normal download numbers over the past several days. We have confirmed that all hits are being logged, however we have yet to recreate the problem exactly. Looking in aggregate at our entire network, the overall average does not appear to be drastically lower then normal, however on an individual basis, we have definitely confirmed there are downloads being logged however not making it into the stats system.

This is not a simple problem- like a piece of hardware breaking that can be easily fixed. It is a bug deep in the new stats aggregator code. The team is working to resolve the issue as quickly as possible, however it may take several days or longer to determine the exact problem, roll back the system and restore the missed downloads.

We appreciate everyone's patience and understanding while we work through this. There is a light at the end of tunnel as we continue to grow our development team and build out newer and better tools and engines to publish, serve and track media.

Just wanted to drop a line and say that we're not ignoring the stats issues. We're sorry for all the frustration its causing and we are still looking into it. The predominant issues still seem to be on either end of the spectrum: low numbers, or numbers that are too high. It seems that the high numbers are showing up mostly in the web downloads section. We're continuing to investigate and will post more soon. Thanks for your patience.

We identified and fixed the bug that was causing the stats system to be unable to differentiate between podcatcher and web downloads.  We have decided to leave the mis-categorization of these numbers over the past few days as-is.  To be very clear: all downloads should have been counted, however downloads by podcatchers like iTunes, Doppler, Juice, etc will not be counted as podcacther (syndicated) hits.  Instead they appear as "web" downloads, but the total numbers should be accurate.

Anyone who suspects otherwise, please let us know.  If you do send in a report of inaccurate stats, please be as detailed as possible, showing the trend for specific numbers you have seen in the past, versus the numbers outside that trend that you are seeing on a particular episode.

Also, please remember that there will likely be an expected low-point in your numbers for the days leading up to 7/23, and then a spike in your numbers from 7/23 - 7/26. 

Thank you friends,

Libsyn

One of the main challenges we've faced with the growth of the Libsyn system has been feeding the stats that are collected into the stats system for analysis.  Some time has been dedicated in past weeks to developing a system that will make this a more reliable process.  We are now putting that system into place.  As we do, users will notice that their stats numbers appear to be frozen, or lower than normal.  This is a temporary situation and your statistics will begin updating again by the end of the day today.  Thanks for your patience as we work to continue to improve upon the Libsyn system.

---

Update (7/25, 4:00pm)

We finished our rollout of the new stats collection system last night, and turned the stats engines back on.  While we expected there to be some degree of "spikiness" for yesterdays and todays numbers (since stats were stopped for a number of hours and some logs had not been fed into the system for about a week, those downloads would get counted when they were finally fed into the system), we have had some reports of extremely high numbers (and especially higher than normal volumes of  web downloads).  We are currently examining whether these are the spikes we expected, or whether something else is going on.  We will update this post with our findings.

---

Our Findings (7/26, 3:30am)

Thanks to everyone for the helpful reports of issues you've observed in the past day or so.  What we found were four main issues which are explained as follows:


1) "Stats are showing extremely high numbers (for 7/23 - 7/26)"

2) "Stats are not appearing for new episodes"

3) "Stats display shows a low number of podcatcher vs web downloads"

4) "Stats are frozen!"

Early this morning the update scripts finished and the stats engines were turned back on. Please report any individual issues to support email and we'll track them down one by one. Again, everyone may notice a slight decrease over the 3 days ( 5/29 - 6/1 ) so please don't report that.

Thank you for your patience

In order to repair the inflated stats and complete the roll back and reprocess, we need to stop the stats engine. We will continue to collect all data and everything will get counted, however the display will not update until sometime tomorrow. We have rough calculations on how long this will take, but will keep everyone updated if it looks like it's going to take longer then expected.

We appreciate your patience during this process. Thanks

Another hard day of work to correct the inflated stats. We wanted to give everyone an update on where things stand. The 321 users affected by the issue turned out to be an underestimate. We honestly don't know the exact number of users affected, so we are rolling back to an earlier backup, well before this issue began.

In short, the total requests for files downloaded on may 30th, june 1st and june 2nd will be slightly lower then normal, but the les and unique ip counts shouldn't be affected. For more technical description, keep reading.

We are currently scrubbing the request queue and reducing any duplication of IP address and request for file by the same IP and same user agent from the same libsyn node server to 1 per day for may 30th ,june 1st, and june 2nd. What that means that for those 3 days worth of data, the total requests, LES impressions, and unique IP counts will be closer to each other then normal. LES and unique IP counts should be affected the least by this procedure. This is the only way we can correct the current inflation which resulted from a combination of database lock and an automated script that wasn't properly keeping track of which files it had already processed. This is a unique scenario that hasn't happened before, and monitoring has been put in place to prevent it from happening in the future.

I know this is alot more detail then most people need and can really use, but we wanted to be clear on what we were doing. We expect this whole process to take another couple days. There's hundreds or millions of lines to process and clean up and reinsert.

Update on the fix for the inflated stats of last week. Many users may still see numbers which are much larger than normal. Files downloaded between June 1st and June 2nd are particularly inflated. Our solution for this is to roll back the database to a backup version then we will backfill the data with the correct stats numbers. This may take several days to fully complete.

Daily hits from June 3rd on are correct so the temporary false inflation over the time span mentioned will eventually we back to normal. We apologize for problems this causes but we feel it is vital to provide accurate statistical information.

We have been tracking down a number of support requests from users who are commenting on much larger then normal downloads for their latest episodes. After many hours of analysis and bug tracking, we have found the cause of the inflation that is affecting only a portion of users (321 to be exact).

We are working on the fix to correct the last couple days worth of stats loads which have been duplicated way beyond proportion, however it may take several days to completely roll back everyone's data. We apologize for the false sense of excitement some people have experienced when they saw they're latest efforts have been downloaded millions instead of thousands of times, and we are working hard to ensure everything is true and accurate. The bug that caused this glitch was (of course) an edge case and has been fixed so that this shouldn't happen again

Again, sorry for the trip up, everything going forward from yesterday on should be accurate and we'll do our best to fix the error asap. Thank you

Many users may be experiencing lower then normal downloads over the last couple days. We are working on the problem and ensuring those hits make their way back into the system as soon as possible. We don't have an exact ETA as of yet when all the data will be back in, but we have the data logged and are working to get it reinserted into the matrix (not making light of the situation, that's really what the machines are called). We are sorry for the continuous headaches this is causing.

---

Update (5/24): The program which was running overnight to catch up the missed hits is still running this morning :( Around 11:30 last night, Hoopes called me with the epiphany of what was wrong, why the hits for some users (not all) have been low or not updating over the last couple days (since around the 20th) and why our alarm system didn't alert us to the problem. It basically had to do with the way we parse the logs for one of the network providers we use, and that script was running out of memory. We're pushing more and more data and the logs are getting bigger and bigger every time this script runs (once every few minutes). I know 98% of you don't care about any of these and want to know 'when the hell are stats going to update again?!' The answer to that is hopefully by the end of the day, but there are no guarantees. Since we're inserting into the system which is also running and keeping up with all the demands, it takes a little longer then normal to insert into the databases. More updates to follow

---

Update (5/25): The backlog of stats should be all caught up now. You may see a swell in yesterday or today's numbers- those are hits collected since the 20th, but got inserted into the system over yesterday and today. Please meebo/skype/ or email us directly if you are still experiencing problems.

Hey guys, just taking a minute to update everyone on the status of stats on libsyn. The backqueue we mentioned a few days ago is about 70% complete at this point, the final partitions of users are being loaded back in now. That means there's going to be an unusual bump of hits showing up as 5/8, but are really hits dating back to the beginning of the month. The good news is, data has not been lost.. it's only going to show up a little bit lumpy in historical charts of hits over time.

A few other issues have been cleared up, such as the timing for a reconcile script that compiles some summary data. Many users were noticing discrepancies between total impression and overall distribution totals. That should be resolved now. Some people may have seen their total impressions number go down, as previously it was higher then the overall total number in the overall distribution block of the stats display. This is because stats records that you have chosen to delete were still being calculated into that total impressions number, and now that is no longer the case. Other weird issues we are dealing with on a case by case basis for individual users.

We are continuing to work hard to provide timely and accurate statistical data. The popularity of all of your shows has really kept us busy. Great job everyone.

There were a couple days worth of hits backlogged on one of our front end machine which is part of the cause of low numbers across the board the last couple days. There are other issues we are still debugging, and will post updates as we have them.

Everyone may notice their numbers for tuesday were very low. There were some process that were handing the stats system and holding up data base inserts. Those requests are being back loaded into the system now and will show up very soon.

To be honest with everyone, we've been having a hell of a time with stats over the past couple weeks. A reload of the libsynPRO stats over this past weekend blew up and caused problems for both libsynPRO and libsynINDIE users. Before that (and in still on going) there are some wayward bugs that are causing miscalculations for users that have deleted stats in the past, basically accounting for a mismatch in total impressions vs overall total distribution (a number that SHOULD be the same).

It's safe to say that with the recent growth libsyn has seen, the log processing system is in need of an overhaul. We have high confidence in our engine, but the data input piece has some flaws. New designs are being worked on as we speak. In the meantime, we are fixing as many pieces to handle the demand as possible, and working hard to keep everyone satisfied- providing as accurate and timely data as possible. As always, we understand the importance of the stats and we appreciate everyone's patients and understanding.

We are currently undergoing a deep investigation and audit of the libsyn stats system. There are a puzzling combination of issues plaguing many users- but not all! That's what's been the hardest thing to diagnose. Most random spot checks we do pulls up a user that has everything working correctly.

We have been compiling the issues people have been writing in about, posting on the forums, etc. The most commonly reported issues are:

stats display very slow to load, or otherwise spiners forever

LAA not updating, reporting 0, or otherwise just wrong

total impressions != overall total distribution

drastically higher total requests then unique or LES audience on some episodes

randomly lower then normal audience for individual episodes

 

Everything we've been seeing points to corrupted data somewhere in the pipeline.

We will keep everyone updated on our findings and appreciate your patience while we work through the issues.

---

update:  The stats display is currently taking an extremely long time to load.  In some instances it takes several minutes to bring up your data.  This is a result of some massive data loading going on in the background.  We are working to implement a caching system to speed up the retrievals of the xml right now to ease the pain.   

There was a hardware transfer that occurred early this morning which, in the process, temporarily took down some stats processing hardware by mistake. The techs at the datacenter are working to bring the hardware back online ASAP. We will keep you updating as we get an ETA from the datacenter.

UPDATE: Everything should be back to normal, thanks to our datacenter guys for being johnny-on-the-spot. Please email support@libsyn.com if anything looks greatly awry.

We have confirmed there continues to be a problem with the libsyn stats system. Across the board downloads have been reporting lower then they actually should be and are not consistent with the bandwidth or raw logs we have been collecting. We are confident all the data is being collected, there is however a bug somewhere in the calculation and reporting process which has been difficult to locate and squash. Our team is working on the problem and we will keep everyone updated on our progress.

---

Update (4/13/2007): We discovered a segment of our delivery network was not sending its logs to our stats engine to be processed.  This was part of the problem why a lot of users were seeing lower then normal numbers over the past several days.  Those logs are in the queue to be processed and will make their way back to your show.  The datastamp on those hits, however, will be for the same day so it may appear that there was a surge of traffic for that day.

There have been some "low days" many users are reporting over the last several days. All the data has been collected, and is in the queue to be processed. The end result may be, unfortunately, some lopsided days where it seems like all of a sudden your shows are getting a ton of downloads as the system catches up. The problem of out of sequence logs has been a ailment for sometime. The team is working on a solution so that this problem no longer plagues the libsyn stats system. Thank you for your patience.

We are getting reports that the stats system is not updating.  We are looking into this now, and hope to have things back up shortly.  It is important to note that we are not losing any data.  It is just the parsing system that is malfunctioning.

---

Update: 2:00pm - All stats are updating regularly again.  

One of our data providers is have some problems delivering us logs as timely as usual.  As a result, the processing of those logs is being delayed and some of you are not seeing updates in your stats display.

We have tickets open with the provider and will catch up on today's stats when we get the logs delivered.

I am sorry for the inconvenience this causes.

A portion of our stats engine has a backup that may result in some users stats appearing to be abnormally low over the last couple days. No data has been lost- it's just a little slow to get into the system. As a result, today (and maybe tomorrow) will look like huge spikes as the hits get rolled in. We apologize for the frustration.

Stats are stuck for some of our users.  We are aware of the problem and are working on it.  We will keep you updated.

After about 1,000 emails all requesting the same feature, we finally caught on.  The stats display version 1.5.1 was rolled out tonight and features (are you ready stats junkies?)  "Remove record" feature!

So finally you can go in remove errant files from your stats display.  Please be cautious when doing this- we don't want to create a new problem while we fix another ("Dear support, I accidently deleted the stats for all my shows, HELP!!!!!!!!").  Ok?  There's an "are you sure" popup that you have to confirm before the record is actually deleted, but still, pay attention to what you are doing.

Currently, when you delete a single episode's stats, it has to go in and re-parse the XML to draw that display again.  I know, it's lame, but the double buffering array thing we are doing has given me a double headache, and well, you shouldn't have to use this feature too often and I wanted to get it out into your hands sooner rather then later.

Also, a slight performance boost was given to the XML loader in this version.  Shows with several years of data still take 20+ seconds to load, but hopefully alot of you will notice snappier load times.

More great stuff to come!
 

It's been a long time coming (too long) but today we are upgrading our server software for statistics. This is primarily a behind-the-scenes upgrade intended to bring increased reliability to our statistics system, and to enable future release of highly desired features such as remove-your-own-files from stats (as in the case of an erroneous upload), the Libsyn Active Audience number, and eventually Geo-stats, among many others. With our new system we have noticed that the stats display takes a little longer to load for users with many files, and will be working to bring the display time down in that case.

We will release more updates as we update the front end display as well

We are aware of the current stats issues people are seeing in which recent files show more days worth of stats than they have been available. This is due to some changes we have made to our network to distribute more files across more server farms, but the problem will be resolved when we upgrade our stats software within the next 24-48 hours.

Hey guys, you may notice some stats irregularities this evening as we're cutting over to a new processing engine.  All steps towards a better, more reliable, more accurate stats system.

Thanks for your patience.

Updates were made to stats tonight that put the cruncher a bit behind.  Things should be caught up and restored in a couple hours.
Thanks

The new hardware for the version 3 stats system has arrived and is being configured, and the code is in its final stages before deployment.  There is no ETA but we are dedicating our time to its completion. The backup system went south again but it should be back up shortly.  We appreciate everyone who has stuck with us during the stats issues. 

We know that stats are temporarily not accessible.  We are working on the issue, and assure you that all numbers are being collected and tabulated.  The connection to the servers that deliver the XML back to the display is hosed, and we're working to get it back ASAP!

Thanks for your patience.

Hey guys, the interim stats box fell over. We are working to bring it back up. The v3 stats engine is nearing completion and will be ready to roll by the top of next week. Please bear with us, promise to have this thing licked very soon.

Stats are back online.

(1) Whilst I was at a conference, our failing stats hardware gave up the ghost for good.
(2) I debated about the value of even trying to resuscitate the old system, since we are literally days away from the V3 server.
(3) The rest of LibSyn convinced me that it was necessary, and special hardware was obtained and put in place to help us through the remaining days to V3.
(4) This hardware is now online and serving stats.

Anomalies?

Shouldn't be too many, but on some of these kinds of awkward recoveries, you may see date buckets split or duplicated, esp around the time of our collapse, which was Sept 1.

The future:

I am in the final stages of development for the V3 server. This will not initially bring new features online, but rather new speed and (most importantly) stability. It establishes a new architecture, however, which will be far more adaptable to new features. Depending on company priorities, I would like geo-location to be the first of these features.

The timeline:

Sept 7 : Stats server with core functionality in stress testing mode;
Sept 8 : V3 stats server ready for emergency failover if V2 dies again.
Sept 10 : Extended functionality complete (referers, podcatcher info, etc.)
Sept 12 : V3 stats server online.

I understand this has been a rough month for stats; and ironically it was just about a year ago that I came onboard with LibSyn because V1 stats were hitting their limits. Since that time, stats have grown 150x as large as they were then, and now the V2 system, which is actually backwards compatible to V1, has hit its limit. The V3 system is redesigned from the ground up and is expected to directly support 100,000x growth and to scale without limit. Obviously, until we see all this in production it's just more words, but fortunately, we'll see it in production quite soon.

(1) It's taking longer to get the next version of stats ready to go than previously hoped. My apologies to all of you.

(2) Our stats server has been increasingly flaky in recent weeks, which you have noticed.

(3) Today I put in a hot backup of the stats server so that we can fail over to the standby in case of further hardware issues.

The downside to this hot backup is that it means I can't reliably fix stats or delete files until the v3 server is done. It's more of a parallel system than a genuine backup.

Those of you with outstanding tickets in our support queue, I'll go through and send an explanation and projected timeline. Once we have v3, you will be able to control your own stats, which should benefit everyone.

-Bluejack

We are no longer calculating statistics on  the following file formats:  "jpg", "jpeg", "gif", "png", "html", "htm", "phtml", "xml".  This is  to counteract the inflation of audience statistics due to  the inclusion of these files in the media distribution folder, as well as to optimize the performance of our stats engine.  Images, as a general rule, should be uploaded using the "new post" form, or via FTP to your /images folder.  Any concerns about this revision should be sent to support@libsyn.com with subject: 7/11/06 Stats Revision.

We have put out a new flash display for the statistics we collect.  It's just incremental fixes and changes, mostly to help reduce the confusion and help people learn: 'How big is my audience?'.  We're not there yet, as you'll notice a large coming soon sign on a quater of the overall stats portion of the app.  However, we'll be activating those sections very soon, as well as releasing RSS stats (!!!) and tying the LES (libsyn estimated statistics) as well as new term for all you acronym lovers to chew on: LAA (libsyn active audience) into the Kiptronic podcast marketplace. 

The LES numbers most requested by potential sponsors are "Impressions" and "Audience."  We will be offering impressions (LES for all episodes) that will represent activity across your whole show: yesterday, last week, last month, and for all time.  We will also be offering active audience- our score for how many people we calculate that right now, actively listen/watch your show.  This takes into account web downloads as well as iTunes and other podcatchers.  The LAA formula we are using for that has been posted and will be discussed of the forum.

If it looks like your stats are missing newer files, here's the story: I rolled out a new version of the stats engine yesterday to support some upcoming features, but I added a bug.

This bug fails to put the right date on some files.

If you appear to be missing some files, check the bottom of your stats page: probably they have been tagged with what the unix world likes to call the "Beginning of Time" (1970).

I am working to correct this bug and should have the fix in place this afternoon at some point.

If you have sent a ticket to support, thank you! You'll get a reply. I'll be using your reports to ensure that the fix actually does correct all the possible problems from this bug.

-Bluejack

PS: For the technically minded: stats keeps a local cache snapshot of the master data for all shows; in this new version, the local cache is failing to get the correct upload time from the master database. I changed some of this code in order to more gracefully handle dates for shows that are no longer in the master database, but clearly I missed something important. In order to fix the bug, I will need to find the erroneous caching code & rebuild the cache & roll a new stats app.

A mysql error blocked connections from our stats database from 2:05:36 am to 4:47:13 pm; we will be recovering the data and restarting the stats system within the next couple of hours.

Apologies to all, and thanks to those who brought the problem to our attention.

--Bluejack

Two hours worth of file transfers were replayed into the stats db, and some attendant display errors were corrected. At this time, stats should be fully back to normal. If you see anything contraindicators, please let us know. Also: in the past, when we have had to recover from logs, "phantom" files have occasionally crept into the stats displays of some users. I have added additional checks to attempt to prevent this, but if you see anything of the sort I'll clean it up and make the recovery process smarter for next time. As always, just drop a note to our support email box with a note about anything that doesn't look right.

Yesterday we thought we had fully corrected the problem that was leading to low stats numbers for this past weekend. We were wrong. As of 7:45 pm, Wed, Jan 11, we *have* corrected the problem, and confirmed this correction, but still need to recover numbers from apache logs from the period of 8:00 monday morning until we finally got the new loadbalancer recording stats correctly. This recovery operation will take place overnight.

Here's the truth about what's going on with the stats recently: (believe it or not, i'm going to route it back to a power supply again)

Last week, when we had the power blow out in one of our important load balancer servers, we recovered and then put in a fail-over system which more of less included adding another server to do the mirror of what the load balancer does, and then split the load between the two (which was also meant the improve the performance of 'click .. .. .. .. .. . download'. So, the new box went it, all was well and good, and everything (including stats) seemed to be working fine. Well.. in the truth was, in fact, that the stats weren't making it back to the stats database from the new load balancer.

So the bad news is, about 1/2 the hits from the last few days haven't been getting counted into your numbers. The very good news is, we are now much better at recovering from the old logs and can back-fill the missing data. It still takes us a little while to reassemble the millions of file requests, so we appreciate it if you pass it along that everything will be restored, and the as of right now, we are counting correctly.

We are really sorry for the past few days, it seems when it rains it pours.. and it sucks.. Thanks for hanging in there with us.

Stats have been acting up today. Database problems have caused updates to freeze. We are currently working on the problem and no data has been lost. However, it will take time to restore the data for all the affected users. We'll let you know when things are back 100%. We apologize for the inconvenience.

we have the new hot rod running stats, and the old pinto is being retired

bluejack has been toiling away in the thrilling world of statistics aggregation and data mining and a bunch of other numbers crap

spot checking has shown the data to be identical to the old stats, so hopefully you won't even notice this switch except for the fact that stats are updated much more often

of course, you guys are the worlds best customers/people/friends/internetters, and i don't know where i was going with this sentence, but it must be because i am giddy that BJ has chipped this monkey off of my back with an ice pick

please let us know if you notice anything fishy (notify us as gently as possible please, you know my ego is fragile, and i haven't been hugged in a hot minute)


peace and chicken grease, ya heard?

love,
- hoopes

leave forum posts here:
http://libsyn.com/forum/viewtopic.php?t=980

Hey everyone.
Just like Sam Beckett We're trying to set right what once went wrong. If you feel your stats numbers are being reported in error, please respond to this forum post asap and we will look into the issue.

Good evening everyone. In the ongoing saga of accurate stats about your podcasts, our team is working hard to redesign some aspects of how we are doing things to be much more efficient. Our previous iterations were not designed to scale to 1100 users serving millions of file requests. We are building and learning as we go. It's the libsyn way. Thank you for your understanding