Update from the stats investigation department: We have discovered a bug in the file filter which strains out extraneous files from the counters and it was causing valid hits to get skipped. We are rolling back a few days worth of data and rerunning right now. Expect it to recover pretty quickly and accurate numbers to follow.

Will update when we have an ETA

---

[update 8/31/2007 7:04am]

The rollback is complete.  Thanks to everyone who sent in detailed reports of inconsistencies they were seeing.  That kind of information was VERY helpful in getting to the bottom of the problem.

Almost exactly 48 hours after the initial drop we're seeing almost full recovery of traffic. We will continue to monitor the graphs and traffic curves.

Finally!  After a long wait, I'm happy to announce the stats are caught up and once again running in near-real-time (there may be a 5-10 minute lag between when a file is downloaded and when it appears in your stats display).

Once again, if you would like to report any anomalies to the support team, please be sure to include specifics like the file names of affected files, your user name, and try to describe exactly which numbers seem to be off-kilter.  Writing us to say, "my stats look weird can you check it out?" only slows things down as our support staff will then need to try and decipher exactly what you are referring to.  We appreciate your cooperation on this.

I understand that you may not be getting this message, or if you are there's a chance some of your audience is still not getting your podcasts.  We will post this around to as many of the boards as we can to keep people informed of the status.

We are seeing an upturn in the MRTG usage graphs which indicates a good portion of the name servers have updated and are now resolving correctly for libsyn.com domains.  It is hard for us to give a blanket ETA as right now it's totally dependent on local ISP's and even home computer DNS caches.  It is tremendously frustrating to know what the problem is but have no way to improve the situation.

At about 2:30 EST on Thursday, Aug 23, there was a momentary DNS misconfiguration that caused some users to be unable to view the site, or in fact, resolve anything under the libsyn.com domain. The situation was recognized and resolved immediately, but due to the nature of the DNS system, some users have cached the incorrect data, either on their computer, or on the ISP DNS servers. As with everything DNS related, the caches will time out, and the correct data will be received, and we've been told the maximum time this will take is 24 hours. (It's usually much shorter, however) In the meantime, a potential remedy for end users is a reboot of their computer to hopefully flush that local cache. We apologize for any downtime.

Some listeners may be noticing slow-ish downloads when requesting a file from the archives.  This is a capacity issue that is currently being addressed.  Basically a section of our archives is nearing its capacity, so we are taking steps to expand the infrastructure that handles that piece of the archives, which will resolve the current slowness and allow much more room for future growth.

This issue is intermittent and potentially affects about 20% of archived files. We expect to have a short-term fix in place sometime this afternoon.  Long term build-out of our infrastructure is continuing.

Another update: based on the rate things are going, we expect the system to take about another 36 hours to become fully current.  That puts us at around 7pm Eastern, Tuesday evening.  Hold tight everyone, we're almost there.

---

[updated 8/15 7:28am ]

As some have noticed, our 7pm Tuesday night ETA was a little optimistic.  As we speak the whole system is catching up and is currently processing numbers for August 7th.   The speed of this catchup process is dependent on a number of of factors, so rather than giving another estimate which may or may not be right, I'll say that all pistons are firing, the stats engines are chewing through all of the raw data, and we'll post here again when we are completely up to date. 

In the meantime, you can check out your numbers for everything up until the 7th by looking under the stats tab of your libsyn dashboard.  Again: there will be inflated numbers for July 23rd and for some of you, July 24th.  These spikes represent downloads of your show that were fed into the system "late" so they were counted as having happened on the 23rd/24th when in fact they happened in the days and weeks previous to that. In total this was a few hundred thousand hits that were added after the fact, and this will affect some users more than others.

If you feel that something about your numbers isn't right, and you send our support team a report of it, please help us out by being as specific as possible.  Include your user name, the file name(s) affected, and the specific numbers that seem to be wrong.  This helps us greatly to figure out what, if anything, is going on.

---

[update 8/21 12:26pm]

Sorry folks, been neglecting updates on the blog.  There is a running thread on the state of the stats system here on the forums.

The latest:




 

The stats rollback / restoration was delayed slightly as Pittsburgh got hit was some severe thunderstorms which caused damage and electricity outages throughout the city.  The process is once again underway.  Stay tuned...

---

[ update: 1:15pm ]

Things are looking good, everyone.  It's taking a little longer than we expected to get this thing going because we found a couple of "gotchas" that would have caused the numbers to look erratic (hits being bunched into days they didn't belong in, mostly).  Though all the numbers would have been counted, we figured at this point, we should smooth out those wrinkles so that when your stats come back, they accurately represent your audience's downloading behavior.

I know we say it a lot, but thanks again for hanging in there with us.

[update 1:50pm]

Because this is a rollback, we are literally rolling the clocks back on the engine that reports data to your stats display. Effectively, what you see right now is the stats as they were mid-day on 7/23. Shortly, the system will start moving forward in time as it reinserts the missing data. Since the clocks have been set back to a couple of weeks ago, recent entries in the stats folder may have seemed to disappear. They should reappear as we climb back through time from July 23rd to the present.
There may also be a slight spike on July 23rd as we're inserting some data that never made it to the the entries from beforehand.
Thanks for your patience.

[update 11:50pm]

We're facing some more challenges.  After watching carefully as the collection service runs through ten full days of data, we are seeing what we believe to be inconsistencies.  Further investigation is ongoing.  There is a good chance we will be resetting / re-starting the rollback process more than once before the weekend is through.  We're trying our hardest to have fresh, accurate, up to date numbers for you all when the week begins.

Starting in just a few minutes, we are going to begin the rollback / restore of the stats system. Effectively, the system is going to be reset to July 23rd and will re-crunch all of the numbers moving forward.  Once this process begins, we'll take some measurements and update this post with the estimated time when we'll be all caught up again.

Hi Everyone- I thought I'd take a moment to give a quick update on the status of the stats system:

We are fairly confident we have wiped out all of the pesky bugs that were wreaking such havoc the past few weeks.  To be absolutely sure there are no gotchas we may have missed, we've opted to extend the testing into Thursday. 

We will update again with specifics on the rollback / restore when that process begins.

[ Originally posted at http://forum.libsyn.com/viewtopic.php?t=7743 ]

Friends,

I know the latest stats issues are causing a bit of an uproar, so I am pulling my head out from under the hood for a minute to explain what's going on.

We have discovered some bugs in the new collection service we put into place around the 23rd of July. (The collection service is the piece of the stats system that feeds the raw hits into a queue to be analyzed by the 'stats-engine'). We've been tracking these issues for a few weeks, and are fully aware that there is a serious problem with the numbers you are seeing. Some people are reporting super high spikes, others are seeing numbers that are way too low. Some see a combination of both, and still others are not getting stats data on some files at all.

We are currently peeling back the layers to find and repair the bugs that have caused these problems. Admittedly, we were probably a bit hasty in pushing the new collection service into production so fast. The old service, however, was on its last leg and could no longer scale to the level of demand being placed on it, so we felt like we had to make the leap. It's probably safe to say that we didn't quite stick that landing.

The good news is, we have isolated and repaired a number of obscure bugs. We have also narrowed the field on a major issue that is causing the collection service to intermittently crash (which can cause you to see low points, followed by extremely high points in your daily numbers).

Once we have routed out the last of these bugs, we have plans to COMPLETELY RESTORE the past data from ~7/23 to present. That is to say, we are going to roll the system back in time to the 23rd, and re-run the numbers going forward. This will eliminate the erratic reporting you've witnessed in the past weeks, and should restore the normal trends that you're used to seeing for your shows.

We know this is extremely frustrating for many of you, It is for us as well. While we've come a long way in the past few years, we are still learning to manage and scale a system of this size. So if you can find it in you, we'd really appreciate your patience and support as we work this one out.

Thank you, sincerely,

Marty
 

Unfortunately we have to report again that some users are seeing lower then normal download numbers over the past several days. We have confirmed that all hits are being logged, however we have yet to recreate the problem exactly. Looking in aggregate at our entire network, the overall average does not appear to be drastically lower then normal, however on an individual basis, we have definitely confirmed there are downloads being logged however not making it into the stats system.

This is not a simple problem- like a piece of hardware breaking that can be easily fixed. It is a bug deep in the new stats aggregator code. The team is working to resolve the issue as quickly as possible, however it may take several days or longer to determine the exact problem, roll back the system and restore the missed downloads.

We appreciate everyone's patience and understanding while we work through this. There is a light at the end of tunnel as we continue to grow our development team and build out newer and better tools and engines to publish, serve and track media.

Just wanted to drop a line and say that we're not ignoring the stats issues. We're sorry for all the frustration its causing and we are still looking into it. The predominant issues still seem to be on either end of the spectrum: low numbers, or numbers that are too high. It seems that the high numbers are showing up mostly in the web downloads section. We're continuing to investigate and will post more soon. Thanks for your patience.