Many users may be experiencing lower then normal downloads over the last couple days. We are working on the problem and ensuring those hits make their way back into the system as soon as possible. We don't have an exact ETA as of yet when all the data will be back in, but we have the data logged and are working to get it reinserted into the matrix (not making light of the situation, that's really what the machines are called). We are sorry for the continuous headaches this is causing.

---

Update (5/24): The program which was running overnight to catch up the missed hits is still running this morning :( Around 11:30 last night, Hoopes called me with the epiphany of what was wrong, why the hits for some users (not all) have been low or not updating over the last couple days (since around the 20th) and why our alarm system didn't alert us to the problem. It basically had to do with the way we parse the logs for one of the network providers we use, and that script was running out of memory. We're pushing more and more data and the logs are getting bigger and bigger every time this script runs (once every few minutes). I know 98% of you don't care about any of these and want to know 'when the hell are stats going to update again?!' The answer to that is hopefully by the end of the day, but there are no guarantees. Since we're inserting into the system which is also running and keeping up with all the demands, it takes a little longer then normal to insert into the databases. More updates to follow

---

Update (5/25): The backlog of stats should be all caught up now. You may see a swell in yesterday or today's numbers- those are hits collected since the 20th, but got inserted into the system over yesterday and today. Please meebo/skype/ or email us directly if you are still experiencing problems.

Hey guys, just taking a minute to update everyone on the status of stats on libsyn. The backqueue we mentioned a few days ago is about 70% complete at this point, the final partitions of users are being loaded back in now. That means there's going to be an unusual bump of hits showing up as 5/8, but are really hits dating back to the beginning of the month. The good news is, data has not been lost.. it's only going to show up a little bit lumpy in historical charts of hits over time.

A few other issues have been cleared up, such as the timing for a reconcile script that compiles some summary data. Many users were noticing discrepancies between total impression and overall distribution totals. That should be resolved now. Some people may have seen their total impressions number go down, as previously it was higher then the overall total number in the overall distribution block of the stats display. This is because stats records that you have chosen to delete were still being calculated into that total impressions number, and now that is no longer the case. Other weird issues we are dealing with on a case by case basis for individual users.

We are continuing to work hard to provide timely and accurate statistical data. The popularity of all of your shows has really kept us busy. Great job everyone.

There were a couple days worth of hits backlogged on one of our front end machine which is part of the cause of low numbers across the board the last couple days. There are other issues we are still debugging, and will post updates as we have them.

Everyone may notice their numbers for tuesday were very low. There were some process that were handing the stats system and holding up data base inserts. Those requests are being back loaded into the system now and will show up very soon.

To be honest with everyone, we've been having a hell of a time with stats over the past couple weeks. A reload of the libsynPRO stats over this past weekend blew up and caused problems for both libsynPRO and libsynINDIE users. Before that (and in still on going) there are some wayward bugs that are causing miscalculations for users that have deleted stats in the past, basically accounting for a mismatch in total impressions vs overall total distribution (a number that SHOULD be the same).

It's safe to say that with the recent growth libsyn has seen, the log processing system is in need of an overhaul. We have high confidence in our engine, but the data input piece has some flaws. New designs are being worked on as we speak. In the meantime, we are fixing as many pieces to handle the demand as possible, and working hard to keep everyone satisfied- providing as accurate and timely data as possible. As always, we understand the importance of the stats and we appreciate everyone's patients and understanding.

We are currently undergoing a deep investigation and audit of the libsyn stats system. There are a puzzling combination of issues plaguing many users- but not all! That's what's been the hardest thing to diagnose. Most random spot checks we do pulls up a user that has everything working correctly.

We have been compiling the issues people have been writing in about, posting on the forums, etc. The most commonly reported issues are:

stats display very slow to load, or otherwise spiners forever

LAA not updating, reporting 0, or otherwise just wrong

total impressions != overall total distribution

drastically higher total requests then unique or LES audience on some episodes

randomly lower then normal audience for individual episodes

 

Everything we've been seeing points to corrupted data somewhere in the pipeline.

We will keep everyone updated on our findings and appreciate your patience while we work through the issues.

---

update:  The stats display is currently taking an extremely long time to load.  In some instances it takes several minutes to bring up your data.  This is a result of some massive data loading going on in the background.  We are working to implement a caching system to speed up the retrievals of the xml right now to ease the pain.