Wed, 13 June 2007 This was a bug in the new media delivery mechanism we recently rolled out. We've addressed the immediate problem and are taking steps to prevent similar incidents in the future. Our apologies to anyone this affected. Category: Media Delivery -- posted at: 11:08 PM Comments[3] |
Fri, 8 June 2007 Early this morning the update scripts finished and the stats engines were turned back on. Please report any individual issues to support email and we'll track them down one by one. Again, everyone may notice a slight decrease over the 3 days ( 5/29 - 6/1 ) so please don't report that. Thank you for your patience Category: Stats -- posted at: 11:38 AM Comments[0] |
Wed, 6 June 2007 In order to repair the inflated stats and complete the roll back and reprocess, we need to stop the stats engine. We will continue to collect all data and everything will get counted, however the display will not update until sometime tomorrow. We have rough calculations on how long this will take, but will keep everyone updated if it looks like it's going to take longer then expected.
We appreciate your patience during this process. Thanks Category: Stats -- posted at: 11:48 AM Comments[0] |
Tue, 5 June 2007 Another hard day of work to correct the inflated stats. We wanted to give everyone an update on where things stand. The 321 users affected by the issue turned out to be an underestimate. We honestly don't know the exact number of users affected, so we are rolling back to an earlier backup, well before this issue began. In short, the total requests for files downloaded on may 30th, june 1st and june 2nd will be slightly lower then normal, but the les and unique ip counts shouldn't be affected. For more technical description, keep reading. We are currently scrubbing the request queue and reducing any duplication of IP address and request for file by the same IP and same user agent from the same libsyn node server to 1 per day for may 30th ,june 1st, and june 2nd. What that means that for those 3 days worth of data, the total requests, LES impressions, and unique IP counts will be closer to each other then normal. LES and unique IP counts should be affected the least by this procedure. This is the only way we can correct the current inflation which resulted from a combination of database lock and an automated script that wasn't properly keeping track of which files it had already processed. This is a unique scenario that hasn't happened before, and monitoring has been put in place to prevent it from happening in the future. I know this is alot more detail then most people need and can really use, but we wanted to be clear on what we were doing. We expect this whole process to take another couple days. There's hundreds or millions of lines to process and clean up and reinsert. Category: Stats -- posted at: 6:00 PM Comments[0] |
Mon, 4 June 2007 Update on the fix for the inflated stats of last week. Many users may still see numbers which are much larger than normal. Files downloaded between June 1st and June 2nd are particularly inflated. Our solution for this is to roll back the database to a backup version then we will backfill the data with the correct stats numbers. This may take several days to fully complete.
Daily hits from June 3rd on are correct so the temporary false inflation over the time span mentioned will eventually we back to normal. We apologize for problems this causes but we feel it is vital to provide accurate statistical information. Category: Stats -- posted at: 6:04 PM Comments[0] |
Fri, 1 June 2007 We have been tracking down a number of support requests from users who are commenting on much larger then normal downloads for their latest episodes. After many hours of analysis and bug tracking, we have found the cause of the inflation that is affecting only a portion of users (321 to be exact).
We are working on the fix to correct the last couple days worth of stats loads which have been duplicated way beyond proportion, however it may take several days to completely roll back everyone's data. We apologize for the false sense of excitement some people have experienced when they saw they're latest efforts have been downloaded millions instead of thousands of times, and we are working hard to ensure everything is true and accurate. The bug that caused this glitch was (of course) an edge case and has been fixed so that this shouldn't happen again Again, sorry for the trip up, everything going forward from yesterday on should be accurate and we'll do our best to fix the error asap. Thank you Category: Stats -- posted at: 10:14 AM Comments[3] |

