Stats On Hold, Will Resume Shortly

One of the main challenges we've faced with the growth of the Libsyn system has been feeding the stats that are collected into the stats system for analysis.  Some time has been dedicated in past weeks to developing a system that will make this a more reliable process.  We are now putting that system into place.  As we do, users will notice that their stats numbers appear to be frozen, or lower than normal.  This is a temporary situation and your statistics will begin updating again by the end of the day today.  Thanks for your patience as we work to continue to improve upon the Libsyn system.

---

Update (7/25, 4:00pm)

We finished our rollout of the new stats collection system last night, and turned the stats engines back on.  While we expected there to be some degree of "spikiness" for yesterdays and todays numbers (since stats were stopped for a number of hours and some logs had not been fed into the system for about a week, those downloads would get counted when they were finally fed into the system), we have had some reports of extremely high numbers (and especially higher than normal volumes of  web downloads).  We are currently examining whether these are the spikes we expected, or whether something else is going on.  We will update this post with our findings.

---

Our Findings (7/26, 3:30am)

Thanks to everyone for the helpful reports of issues you've observed in the past day or so.  What we found were four main issues which are explained as follows:


1) "Stats are showing extremely high numbers (for 7/23 - 7/26)"

As mentioned earlier in this post, we did expect some higher than normal numbers when we resumed feeding the hits into the system.  In some cases though, these numbers were extremely high, on the order of 2 - 3x the normal numbers.  The reports we received of this were eventually attributed to a known issue with version 3.2 of the TVTonic client, where  the client was repeatedly downloading a file or files again and again.  In one instance, we discovered 6,000+ downloads of the same file over a relatively short period of time, from a single IP address using this client.  The folks at TVTonic are on it though and their latest release is not affected by this bug.

2) "Stats are not appearing for new episodes"

This was a human error where one of the log sources was not activated properly in our new collection system, so the hits were just sitting there, not being fed in.  This was resolved about an hour ago, so those of you who experienced this problem, you should start seeing your numbers shortly.

3) "Stats display shows a low number of podcatcher vs web downloads"

This issue is still under investigation.  The system is somehow failing to identify hits from known "podcatcher" clients as such.  All hits are still being counted but in some cases, downloads by iTunes, Juice, etc show up as "web downloads."

4) "Stats are frozen!"

While the new collection service is more robust than its predecessor, we have not quite worked out all the kinks yet, so certain scenarios cause it to crash.  Once re-started, it recovers perfectly, but during the time between the crash and the re-start, stats appear to be frozen.  We are putting an alarm on this service while we search for a solution to the underlying issue that might cause it to fall over.  This should minimize the time between a crash and a restart, should it happen again.