We have been working to restart the log aggregator which has fallen out of sync. It will take several hours, but all data from yesterday and today will eventually catch up. Thank you for your patience as we maintain the current system while continue to rebuild the libsyn stats engine from the ground up.

The ability to post new topics or reply to old posts has been down for a few days, we inadvertently broke something when installing our new anti-spam measures.  We apologize for any inconvenience this may have caused our users, but rest assured that the forums are back up and in working order, with all the new anti-spam features in place.

We have also updated the blog pages with similar measures in an attempt to lessen the amount of spam users receive.  Please email us at support@libsyn.com if you have any problems regarding these new changes.

A combination of different activities caused some run away processes to back up traffic going to libsyn.com's main application servers and database.  This resulted in slow page loads while hitting libsyn blogs, rss feeds, and login sessions on libsyn.com.  Media delivery was not affected by this issue.

The system in place to alert us of these issues failed due to some mis configuration after a recent system change.  This caused a slower than usual response time from our support team.

All issues have been resolved, the safety nets have been reconfigured.  We apologize for inconvenience and are working to deliver a redesigned backend for libsyn.com which will dramatically improve the quality of service for all libsyn users.  Thank you for your patience while we rebuild libsyn.

We have been working to keep the stats engine updating regularly over the past week while we diagnose some hardware and configuration issues. There were some disk issues that we cleared up this afternoon and as of this evening we believe all queues should be caught up.

While we believe all the issues have been resolved regarding the queue of log files to process, over the next several days updates may slow down again from time to time.

The slow updates, or "stalled stats" issue is a high priority concern that the team is working on improving. Thank you for your continued patience and support while we rebuild our stats system from the ground up.

After several days of investing strange media playback behavior, we have discovered a bug in our content storage system. The details are still not completely solid for which files could be affected, but as of right now we believe it to be larger (100mb+) files potentially getting corrupted as we push media into our storage array. It is not every file.. the bug occurs intermittently making it very hard to pin down. There have been sporadic reports of archived files getting corrupted and today some reports of new media files not playing properly once downloaded.

This issue is being addressed and we are working to detect the incomplete files and replace the corrupted or cut-off files from backup. Depending on the media type, sometimes the result of this bug will be a file that is just cut-off, or in the case of certain video codecs, the movie not playing at all. Quicktime reports "invalid public atom was found in the movie" in some cases, however in iTunes the file just doesn't show up.

While we are making a sweep of the terabytes of data we have stored, if you are experiencing "weird" issues like these please let us know immediately and we will investigate your issue specifically.

---

Update 10/8: The issue, unfortunately, seems to have extended beyond large files. About 5% of the files in our archives have been cut-off. We believe this has resulted from a script which ran to repair the mime-types of these files. The mime-type problem was causing files to not "stream" or progressive play download. The "fix" seems to have done more damage then good.

We are running some scripts now to recover as many of the files from backups as we can. Unfortunately there will likely be files which will need to be re-uploaded. We will contact the users directly who this pertains to via email.

There have been extra validation checks put into place to ensure this will never happen again.

---

Update 10/9: We have scoured through the backups that we do have for some of the media and have found about 50% of the potentiallycorrupted files. Users who have been affected will receive an email with a list of the files that we have detected a file size mismatch on. You will be asked to verify those files by downloading them and if they are cut-off or corrupted, please reupload those files at your earliest convenience. Once uploaded, they will be automatically pushed into the archives and you the storage will not be counted towards your monthly quota.

Stay tuned for more updates to come.

A problem was discovered earlier today that was causing some newly uploaded files to not appear in the media tab of the libsyn dashboard.  This issue was limited to files with illegal characters or spaces in the filenames.  We fixed all instances of this that we were able to identify.  If do happen to have files that you uploaded not showing up, please let us.

For future reference, we "clean up" filenames by replacing spaces with underscores, and removing characters which are not alphanumeric (a-z, 0-9), periods  (.), hyphens (-), or underscores (_).

 

As mentioned in a previous post, we have been laying the groundwork for some time now, in preparation for a substantial upgrade to our network infrastructure.  In general we will be simplifying and consolidating the system that delivers your media to your audiences.  The switchover to the new infrastructure should be transparent from yours and your listeners points of view.

We'd like to encourage those of you who manage your own RSS feeds and/or who post links to your files directly to double check those links and make sure they follow the standard format of http://media.libsyn.com/media/username/filename

If any of your links don't look like that, there's a good chance those links will stop working in the near future.

The actual "switch" will be flipped starting late Sunday night.  We will keep you informed on the progress of things, and invite you to send any questions or concerns to support@libsyn.com

There was an issue when publishing new episodes if you had selected the option to also ping the podcast directories.  The episodes were getting published fine, however users were seeing an error message.

That problem has been fixed.  Please let us know about any more publishing-releted issues as we have made some recent changes to the publishing system.

Sorry again for the inconvenience.

Some of you may have had a hard time connecting to Libsyn FTP in the last day or so.  One of the FTP servers was not handling logins properly and had to be re-started, and you should be able to access your FTP accounts once again.

Hi everyone.  This is just a note to let you all know that we have implemented some changes to the web-uploader that we wanted you to be aware of.  The majority of the changes are on the back-end, and are part of our preparation for an upcoming network upgrade (details to be announced later in the day).  On the front-end, you should only notice some cosmetic changes, as well as a bit more info on the status of your upload as it progresses.

If you have any issues with the updated uploader, please catch us on the support chat at http://support.libsyn.com, or drop an email to support@libsyn.com.

---

Update 9/28 3:06pm

Thanks to those who alerted us that in some cases, publishing new posts was not working properly (for example if an apostraphe was included in the post title).  We've addressed this issue and it should be working as expected now.

We have multiple reports of download requests resulting in "404 File Not Found" errors.  We have verified this issue and have an open support ticket with the CDN that is the source of the problem.  We will work with them to get this fixed as quickly as possible, and apologize for the inconvenience.

---

Update: This issue was confirmed to be resolved at 8:45pm

---

Update: 9/23: The issues with the 404 on media files requests is happening again. We were told it was a corruption of our CDN's core database occurred yesterday after a server crash, however it was repaired. We are in contact with our providers and working to resolve the issue and/or temporarily route around it while the problems persist.

This article if not for the faint of heart. It gets a little long, a little technical, but it is not the type of thing we could throw into bullet points and be confident we've covered everything we wanted to convey. So, if you are totally happy with the way our service is working for you and don't have a lot of interest in stats, go about your business and just know we are working on making them much better and if you should ever need them, they'll be there for you.

For the stats connoisseurs, please read this forum post.

There were a set of logs that were being blocked from access by the log aggregator so they weren't getting parsed into the stats system... these were from 8/31 to present. We just pushed them all in today, so users with low hits may see that this resolves the problem... and users in general will see a spike in their hits today, especially for archived files.

Due to the pervasiveness of commenting spam we are removing email notifications when comments are left to your posts. Spamming is a very frustrating obstacle with blogs/commenting and unfortunately spammers often can work around whatever hurdles that may be in their way. We'll continue to explore methods of curbing their attacks, but for now we're at least going to cut back on some of the junk that may fill your inbox. Thanks

We are investigating some more reports of lower-then-expected download numbers. There have been some adjustments made and we need a couple complete days of data to see if the graphs smooth out.

We will keep you posted as we learn more about what's going on and who may be affected.

Update from the stats investigation department: We have discovered a bug in the file filter which strains out extraneous files from the counters and it was causing valid hits to get skipped. We are rolling back a few days worth of data and rerunning right now. Expect it to recover pretty quickly and accurate numbers to follow.

Will update when we have an ETA

---

[update 8/31/2007 7:04am]

The rollback is complete.  Thanks to everyone who sent in detailed reports of inconsistencies they were seeing.  That kind of information was VERY helpful in getting to the bottom of the problem.

Almost exactly 48 hours after the initial drop we're seeing almost full recovery of traffic. We will continue to monitor the graphs and traffic curves.

Finally!  After a long wait, I'm happy to announce the stats are caught up and once again running in near-real-time (there may be a 5-10 minute lag between when a file is downloaded and when it appears in your stats display).

Once again, if you would like to report any anomalies to the support team, please be sure to include specifics like the file names of affected files, your user name, and try to describe exactly which numbers seem to be off-kilter.  Writing us to say, "my stats look weird can you check it out?" only slows things down as our support staff will then need to try and decipher exactly what you are referring to.  We appreciate your cooperation on this.

I understand that you may not be getting this message, or if you are there's a chance some of your audience is still not getting your podcasts.  We will post this around to as many of the boards as we can to keep people informed of the status.

We are seeing an upturn in the MRTG usage graphs which indicates a good portion of the name servers have updated and are now resolving correctly for libsyn.com domains.  It is hard for us to give a blanket ETA as right now it's totally dependent on local ISP's and even home computer DNS caches.  It is tremendously frustrating to know what the problem is but have no way to improve the situation.

At about 2:30 EST on Thursday, Aug 23, there was a momentary DNS misconfiguration that caused some users to be unable to view the site, or in fact, resolve anything under the libsyn.com domain. The situation was recognized and resolved immediately, but due to the nature of the DNS system, some users have cached the incorrect data, either on their computer, or on the ISP DNS servers. As with everything DNS related, the caches will time out, and the correct data will be received, and we've been told the maximum time this will take is 24 hours. (It's usually much shorter, however) In the meantime, a potential remedy for end users is a reboot of their computer to hopefully flush that local cache. We apologize for any downtime.

Some listeners may be noticing slow-ish downloads when requesting a file from the archives.  This is a capacity issue that is currently being addressed.  Basically a section of our archives is nearing its capacity, so we are taking steps to expand the infrastructure that handles that piece of the archives, which will resolve the current slowness and allow much more room for future growth.

This issue is intermittent and potentially affects about 20% of archived files. We expect to have a short-term fix in place sometime this afternoon.  Long term build-out of our infrastructure is continuing.

Another update: based on the rate things are going, we expect the system to take about another 36 hours to become fully current.  That puts us at around 7pm Eastern, Tuesday evening.  Hold tight everyone, we're almost there.

---

[updated 8/15 7:28am ]

As some have noticed, our 7pm Tuesday night ETA was a little optimistic.  As we speak the whole system is catching up and is currently processing numbers for August 7th.   The speed of this catchup process is dependent on a number of of factors, so rather than giving another estimate which may or may not be right, I'll say that all pistons are firing, the stats engines are chewing through all of the raw data, and we'll post here again when we are completely up to date. 

In the meantime, you can check out your numbers for everything up until the 7th by looking under the stats tab of your libsyn dashboard.  Again: there will be inflated numbers for July 23rd and for some of you, July 24th.  These spikes represent downloads of your show that were fed into the system "late" so they were counted as having happened on the 23rd/24th when in fact they happened in the days and weeks previous to that. In total this was a few hundred thousand hits that were added after the fact, and this will affect some users more than others.

If you feel that something about your numbers isn't right, and you send our support team a report of it, please help us out by being as specific as possible.  Include your user name, the file name(s) affected, and the specific numbers that seem to be wrong.  This helps us greatly to figure out what, if anything, is going on.

---

[update 8/21 12:26pm]

Sorry folks, been neglecting updates on the blog.  There is a running thread on the state of the stats system here on the forums.

The latest:




 

The stats rollback / restoration was delayed slightly as Pittsburgh got hit was some severe thunderstorms which caused damage and electricity outages throughout the city.  The process is once again underway.  Stay tuned...

---

[ update: 1:15pm ]

Things are looking good, everyone.  It's taking a little longer than we expected to get this thing going because we found a couple of "gotchas" that would have caused the numbers to look erratic (hits being bunched into days they didn't belong in, mostly).  Though all the numbers would have been counted, we figured at this point, we should smooth out those wrinkles so that when your stats come back, they accurately represent your audience's downloading behavior.

I know we say it a lot, but thanks again for hanging in there with us.

[update 1:50pm]

Because this is a rollback, we are literally rolling the clocks back on the engine that reports data to your stats display. Effectively, what you see right now is the stats as they were mid-day on 7/23. Shortly, the system will start moving forward in time as it reinserts the missing data. Since the clocks have been set back to a couple of weeks ago, recent entries in the stats folder may have seemed to disappear. They should reappear as we climb back through time from July 23rd to the present.
There may also be a slight spike on July 23rd as we're inserting some data that never made it to the the entries from beforehand.
Thanks for your patience.

[update 11:50pm]

We're facing some more challenges.  After watching carefully as the collection service runs through ten full days of data, we are seeing what we believe to be inconsistencies.  Further investigation is ongoing.  There is a good chance we will be resetting / re-starting the rollback process more than once before the weekend is through.  We're trying our hardest to have fresh, accurate, up to date numbers for you all when the week begins.

Starting in just a few minutes, we are going to begin the rollback / restore of the stats system. Effectively, the system is going to be reset to July 23rd and will re-crunch all of the numbers moving forward.  Once this process begins, we'll take some measurements and update this post with the estimated time when we'll be all caught up again.

Hi Everyone- I thought I'd take a moment to give a quick update on the status of the stats system:

We are fairly confident we have wiped out all of the pesky bugs that were wreaking such havoc the past few weeks.  To be absolutely sure there are no gotchas we may have missed, we've opted to extend the testing into Thursday. 

We will update again with specifics on the rollback / restore when that process begins.

[ Originally posted at http://forum.libsyn.com/viewtopic.php?t=7743 ]

Friends,

I know the latest stats issues are causing a bit of an uproar, so I am pulling my head out from under the hood for a minute to explain what's going on.

We have discovered some bugs in the new collection service we put into place around the 23rd of July. (The collection service is the piece of the stats system that feeds the raw hits into a queue to be analyzed by the 'stats-engine'). We've been tracking these issues for a few weeks, and are fully aware that there is a serious problem with the numbers you are seeing. Some people are reporting super high spikes, others are seeing numbers that are way too low. Some see a combination of both, and still others are not getting stats data on some files at all.

We are currently peeling back the layers to find and repair the bugs that have caused these problems. Admittedly, we were probably a bit hasty in pushing the new collection service into production so fast. The old service, however, was on its last leg and could no longer scale to the level of demand being placed on it, so we felt like we had to make the leap. It's probably safe to say that we didn't quite stick that landing.

The good news is, we have isolated and repaired a number of obscure bugs. We have also narrowed the field on a major issue that is causing the collection service to intermittently crash (which can cause you to see low points, followed by extremely high points in your daily numbers).

Once we have routed out the last of these bugs, we have plans to COMPLETELY RESTORE the past data from ~7/23 to present. That is to say, we are going to roll the system back in time to the 23rd, and re-run the numbers going forward. This will eliminate the erratic reporting you've witnessed in the past weeks, and should restore the normal trends that you're used to seeing for your shows.

We know this is extremely frustrating for many of you, It is for us as well. While we've come a long way in the past few years, we are still learning to manage and scale a system of this size. So if you can find it in you, we'd really appreciate your patience and support as we work this one out.

Thank you, sincerely,

Marty
 

Unfortunately we have to report again that some users are seeing lower then normal download numbers over the past several days. We have confirmed that all hits are being logged, however we have yet to recreate the problem exactly. Looking in aggregate at our entire network, the overall average does not appear to be drastically lower then normal, however on an individual basis, we have definitely confirmed there are downloads being logged however not making it into the stats system.

This is not a simple problem- like a piece of hardware breaking that can be easily fixed. It is a bug deep in the new stats aggregator code. The team is working to resolve the issue as quickly as possible, however it may take several days or longer to determine the exact problem, roll back the system and restore the missed downloads.

We appreciate everyone's patience and understanding while we work through this. There is a light at the end of tunnel as we continue to grow our development team and build out newer and better tools and engines to publish, serve and track media.

Just wanted to drop a line and say that we're not ignoring the stats issues. We're sorry for all the frustration its causing and we are still looking into it. The predominant issues still seem to be on either end of the spectrum: low numbers, or numbers that are too high. It seems that the high numbers are showing up mostly in the web downloads section. We're continuing to investigate and will post more soon. Thanks for your patience.

The archival process has fallen behind and is now catching up.  Files that were due to archive on July 25th or later are affected by this issue.  The archiver is cranking through them.  If this is affecting your ability to upload new episodes, please get in touch with our support team and they will help you free up the space that you need.

We identified and fixed the bug that was causing the stats system to be unable to differentiate between podcatcher and web downloads.  We have decided to leave the mis-categorization of these numbers over the past few days as-is.  To be very clear: all downloads should have been counted, however downloads by podcatchers like iTunes, Doppler, Juice, etc will not be counted as podcacther (syndicated) hits.  Instead they appear as "web" downloads, but the total numbers should be accurate.

Anyone who suspects otherwise, please let us know.  If you do send in a report of inaccurate stats, please be as detailed as possible, showing the trend for specific numbers you have seen in the past, versus the numbers outside that trend that you are seeing on a particular episode.

Also, please remember that there will likely be an expected low-point in your numbers for the days leading up to 7/23, and then a spike in your numbers from 7/23 - 7/26. 

Thank you friends,

Libsyn

One of the main challenges we've faced with the growth of the Libsyn system has been feeding the stats that are collected into the stats system for analysis.  Some time has been dedicated in past weeks to developing a system that will make this a more reliable process.  We are now putting that system into place.  As we do, users will notice that their stats numbers appear to be frozen, or lower than normal.  This is a temporary situation and your statistics will begin updating again by the end of the day today.  Thanks for your patience as we work to continue to improve upon the Libsyn system.

---

Update (7/25, 4:00pm)

We finished our rollout of the new stats collection system last night, and turned the stats engines back on.  While we expected there to be some degree of "spikiness" for yesterdays and todays numbers (since stats were stopped for a number of hours and some logs had not been fed into the system for about a week, those downloads would get counted when they were finally fed into the system), we have had some reports of extremely high numbers (and especially higher than normal volumes of  web downloads).  We are currently examining whether these are the spikes we expected, or whether something else is going on.  We will update this post with our findings.

---

Our Findings (7/26, 3:30am)

Thanks to everyone for the helpful reports of issues you've observed in the past day or so.  What we found were four main issues which are explained as follows:


1) "Stats are showing extremely high numbers (for 7/23 - 7/26)"

2) "Stats are not appearing for new episodes"

3) "Stats display shows a low number of podcatcher vs web downloads"

4) "Stats are frozen!"

Got word from the good folks at Apple:

This was a bug in the new media delivery mechanism we recently rolled out.  We've addressed the immediate problem and are taking steps to prevent similar incidents in the future.  Our apologies to anyone this affected.

Early this morning the update scripts finished and the stats engines were turned back on. Please report any individual issues to support email and we'll track them down one by one. Again, everyone may notice a slight decrease over the 3 days ( 5/29 - 6/1 ) so please don't report that.

Thank you for your patience

In order to repair the inflated stats and complete the roll back and reprocess, we need to stop the stats engine. We will continue to collect all data and everything will get counted, however the display will not update until sometime tomorrow. We have rough calculations on how long this will take, but will keep everyone updated if it looks like it's going to take longer then expected.

We appreciate your patience during this process. Thanks

Another hard day of work to correct the inflated stats. We wanted to give everyone an update on where things stand. The 321 users affected by the issue turned out to be an underestimate. We honestly don't know the exact number of users affected, so we are rolling back to an earlier backup, well before this issue began.

In short, the total requests for files downloaded on may 30th, june 1st and june 2nd will be slightly lower then normal, but the les and unique ip counts shouldn't be affected. For more technical description, keep reading.

We are currently scrubbing the request queue and reducing any duplication of IP address and request for file by the same IP and same user agent from the same libsyn node server to 1 per day for may 30th ,june 1st, and june 2nd. What that means that for those 3 days worth of data, the total requests, LES impressions, and unique IP counts will be closer to each other then normal. LES and unique IP counts should be affected the least by this procedure. This is the only way we can correct the current inflation which resulted from a combination of database lock and an automated script that wasn't properly keeping track of which files it had already processed. This is a unique scenario that hasn't happened before, and monitoring has been put in place to prevent it from happening in the future.

I know this is alot more detail then most people need and can really use, but we wanted to be clear on what we were doing. We expect this whole process to take another couple days. There's hundreds or millions of lines to process and clean up and reinsert.

Update on the fix for the inflated stats of last week. Many users may still see numbers which are much larger than normal. Files downloaded between June 1st and June 2nd are particularly inflated. Our solution for this is to roll back the database to a backup version then we will backfill the data with the correct stats numbers. This may take several days to fully complete.

Daily hits from June 3rd on are correct so the temporary false inflation over the time span mentioned will eventually we back to normal. We apologize for problems this causes but we feel it is vital to provide accurate statistical information.

We have been tracking down a number of support requests from users who are commenting on much larger then normal downloads for their latest episodes. After many hours of analysis and bug tracking, we have found the cause of the inflation that is affecting only a portion of users (321 to be exact).

We are working on the fix to correct the last couple days worth of stats loads which have been duplicated way beyond proportion, however it may take several days to completely roll back everyone's data. We apologize for the false sense of excitement some people have experienced when they saw they're latest efforts have been downloaded millions instead of thousands of times, and we are working hard to ensure everything is true and accurate. The bug that caused this glitch was (of course) an edge case and has been fixed so that this shouldn't happen again

Again, sorry for the trip up, everything going forward from yesterday on should be accurate and we'll do our best to fix the error asap. Thank you

Many users may be experiencing lower then normal downloads over the last couple days. We are working on the problem and ensuring those hits make their way back into the system as soon as possible. We don't have an exact ETA as of yet when all the data will be back in, but we have the data logged and are working to get it reinserted into the matrix (not making light of the situation, that's really what the machines are called). We are sorry for the continuous headaches this is causing.

---

Update (5/24): The program which was running overnight to catch up the missed hits is still running this morning :( Around 11:30 last night, Hoopes called me with the epiphany of what was wrong, why the hits for some users (not all) have been low or not updating over the last couple days (since around the 20th) and why our alarm system didn't alert us to the problem. It basically had to do with the way we parse the logs for one of the network providers we use, and that script was running out of memory. We're pushing more and more data and the logs are getting bigger and bigger every time this script runs (once every few minutes). I know 98% of you don't care about any of these and want to know 'when the hell are stats going to update again?!' The answer to that is hopefully by the end of the day, but there are no guarantees. Since we're inserting into the system which is also running and keeping up with all the demands, it takes a little longer then normal to insert into the databases. More updates to follow

---

Update (5/25): The backlog of stats should be all caught up now. You may see a swell in yesterday or today's numbers- those are hits collected since the 20th, but got inserted into the system over yesterday and today. Please meebo/skype/ or email us directly if you are still experiencing problems.

Hey guys, just taking a minute to update everyone on the status of stats on libsyn. The backqueue we mentioned a few days ago is about 70% complete at this point, the final partitions of users are being loaded back in now. That means there's going to be an unusual bump of hits showing up as 5/8, but are really hits dating back to the beginning of the month. The good news is, data has not been lost.. it's only going to show up a little bit lumpy in historical charts of hits over time.

A few other issues have been cleared up, such as the timing for a reconcile script that compiles some summary data. Many users were noticing discrepancies between total impression and overall distribution totals. That should be resolved now. Some people may have seen their total impressions number go down, as previously it was higher then the overall total number in the overall distribution block of the stats display. This is because stats records that you have chosen to delete were still being calculated into that total impressions number, and now that is no longer the case. Other weird issues we are dealing with on a case by case basis for individual users.

We are continuing to work hard to provide timely and accurate statistical data. The popularity of all of your shows has really kept us busy. Great job everyone.

There were a couple days worth of hits backlogged on one of our front end machine which is part of the cause of low numbers across the board the last couple days. There are other issues we are still debugging, and will post updates as we have them.

Everyone may notice their numbers for tuesday were very low. There were some process that were handing the stats system and holding up data base inserts. Those requests are being back loaded into the system now and will show up very soon.

To be honest with everyone, we've been having a hell of a time with stats over the past couple weeks. A reload of the libsynPRO stats over this past weekend blew up and caused problems for both libsynPRO and libsynINDIE users. Before that (and in still on going) there are some wayward bugs that are causing miscalculations for users that have deleted stats in the past, basically accounting for a mismatch in total impressions vs overall total distribution (a number that SHOULD be the same).

It's safe to say that with the recent growth libsyn has seen, the log processing system is in need of an overhaul. We have high confidence in our engine, but the data input piece has some flaws. New designs are being worked on as we speak. In the meantime, we are fixing as many pieces to handle the demand as possible, and working hard to keep everyone satisfied- providing as accurate and timely data as possible. As always, we understand the importance of the stats and we appreciate everyone's patients and understanding.

We are currently undergoing a deep investigation and audit of the libsyn stats system. There are a puzzling combination of issues plaguing many users- but not all! That's what's been the hardest thing to diagnose. Most random spot checks we do pulls up a user that has everything working correctly.

We have been compiling the issues people have been writing in about, posting on the forums, etc. The most commonly reported issues are:

stats display very slow to load, or otherwise spiners forever

LAA not updating, reporting 0, or otherwise just wrong

total impressions != overall total distribution

drastically higher total requests then unique or LES audience on some episodes

randomly lower then normal audience for individual episodes

 

Everything we've been seeing points to corrupted data somewhere in the pipeline.

We will keep everyone updated on our findings and appreciate your patience while we work through the issues.

---

update:  The stats display is currently taking an extremely long time to load.  In some instances it takes several minutes to bring up your data.  This is a result of some massive data loading going on in the background.  We are working to implement a caching system to speed up the retrievals of the xml right now to ease the pain.   

One of our datacenters is experiencing some downtime. Media delivery has been adjusted to compensate, but there may be intermittent download problems.
This will be remedied as soon as possible, and this post will be updated with further information.
Thank you for your patience.

There was a hardware transfer that occurred early this morning which, in the process, temporarily took down some stats processing hardware by mistake. The techs at the datacenter are working to bring the hardware back online ASAP. We will keep you updating as we get an ETA from the datacenter.

UPDATE: Everything should be back to normal, thanks to our datacenter guys for being johnny-on-the-spot. Please email support@libsyn.com if anything looks greatly awry.

Users who have enabled the logging option for their stats will notice that the logs are not updating.  We are performing some maintenance on this log-writing system to ensure continuity of service and near-real-time performance.  We will have updates back online by tomorow morning.
  Thanks for understanding.

---

Update (4/17/2007 5:50am EDT) - Stats logs are back online and updating regularly. 

We have confirmed there continues to be a problem with the libsyn stats system. Across the board downloads have been reporting lower then they actually should be and are not consistent with the bandwidth or raw logs we have been collecting. We are confident all the data is being collected, there is however a bug somewhere in the calculation and reporting process which has been difficult to locate and squash. Our team is working on the problem and we will keep everyone updated on our progress.

---

Update (4/13/2007): We discovered a segment of our delivery network was not sending its logs to our stats engine to be processed.  This was part of the problem why a lot of users were seeing lower then normal numbers over the past several days.  Those logs are in the queue to be processed and will make their way back to your show.  The datastamp on those hits, however, will be for the same day so it may appear that there was a surge of traffic for that day.

There were intermittent DNS and database conflicts yesterday which have resulted in a few different issues for some users. First off, if you couldn't login yesterday or had trouble accessing some parts of the site or uploading, that problem is resolved. If you were able to upload new media, however the database was not updated, you may have problems today where files do not download, or there are no stats being recorded, etc. We have attempted to sweep through and reconcile the missing database records, however we may have missed some and you should contact us via meebo, or support@libsyn and we'll look into issues on an individual basis. Finally, if you had written into support, but did not get a response yesterday, our email support system was being affected by the same issues, and we are working through the backlog now. We apologize for the headache this may have caused.

There have been some "low days" many users are reporting over the last several days. All the data has been collected, and is in the queue to be processed. The end result may be, unfortunately, some lopsided days where it seems like all of a sudden your shows are getting a ton of downloads as the system catches up. The problem of out of sequence logs has been a ailment for sometime. The team is working on a solution so that this problem no longer plagues the libsyn stats system. Thank you for your patience.

There's been several issues over the past 24 hours which have all now been resolved. Stats:

The stats system has been playing catchup for the last day. All data is being collected, it's in the queue to be processed, and you should be seeing timely updates once again when everything is caught up. In the meantime, it may appear your numbers are not increasing and new files may not yet appear in the stats display

Download timeouts

There was an issue with one of our data providers which was causing some users, and especially archived files, to be slow to start and time out resulting in no downloads. The fail-over mechanism that should have recovered the issue has been fixed as well as the problem with the provider itself.

Archiving

The archiving issue which has been causing files to not move to the archives has been resolved.

Kiptronic insertions

Libsyn users who are using Kiptronic were experiencing issues with insertions not working. That has been resolved.

We are sorry the status of these issues were not published earlier- the team has been working hard to fix all the problems. Please feel free to contact us through meebo or skype (libsynsupport).

We are battling some database issues right now.  Logins are currently being affected by the issue. We will keep you posted on the status.

---

Update: (12:30) The database issues have been repaired.  In an effort to reduce database load, we have temporarily taken down the libsyn directory.  The code will be revamped and brought back online quicker and better then ever very soon.

Many users may have been experiencing problems uploading and/or connecting to FTP (same issue) this morning.

The issue has been resolved.  Sorry for the interruption.

We are getting reports that the stats system is not updating.  We are looking into this now, and hope to have things back up shortly.  It is important to note that we are not losing any data.  It is just the parsing system that is malfunctioning.

---

Update: 2:00pm - All stats are updating regularly again.  

We are experiencing a partial network outage that will likely result in some of you or your listeners being unable to connect to libsyn.com.  We have our datacenter techs on the scene working on the issue and will post an update when we have more info.

---

Update: 5:11pm

We are back up at full capacity.  In reviewing our network graphs, it appears as though some but not all of the Libsyn traffic was hindered.  This "brown out" lasted for around 20 miuntes.  For those of you who are geek-savvy: 2 backbone connections at our primary data center lost connectivity.  We worked with the techs there to manually route around the problem and get back to full capacity as quickly as possible.

We apologize for any inconvenience this may have caused.

We are still cleaning up from some issues last night, one lingering one is some flakeyness with FTP this morning.

Please be sure when you are connecting via ftp to use the host name: ftp.libsyn.com and also make sure your connection type is set to passive.

We are working hard to get things back to normal with FTP as well as some problems people are having modifying or viewing their blog pages with proper templates.  All issues will be resolved ASAP.

Sorry for the inconvenience.

---

Update:  There was some hardware issues that was/is causing the blogpage template issues.  While the techs in the datacenter work on the issue, we can repair templates on an individual basis.   If you don't modify your blogtemplate during this outtage, you shouldn't notice any problems.  If you do, and your page is broken, let us know via meebo chat and we'll fix you up.

---

Update: The issue with the blog templates should be back to normal now, however the FTP uploading issue has rolled into a system-wide uploading problem which we are now fighting.  Please hold off attempting to upload anything new for the time being while we work this out.

Thank you

---

Update:  It's been a rough day, uploading should be working ok now.  There were some file propagation issues, but everything should be synced up now.

Many users may have experienced trouble uploading new media or accessing their blog pages this evening as a segment of the network is being worked on.  The problem should be taken care of shortly and everything will be back to normal.

The issue has caused us to push back the scheduled maintenance of the database a little, but we should still make it in the 10-2am EST window.

We will be performing some routine maintenance on our main database server this evening, February 20th, between 10pm and 2am.  This process should be completely transparent, however there may be a momentary blip in service as we switch from our main database server to the backup server.  This maintenance will not affect media delivery and should have minimal impact on your RSS feeds and Blog Pages.

We will post an update when the work is complete. 

---

Update [3:41am] - Well, it took longer than expected, and as some of you may have noticed, the backup DB made for a considerably slower experience if you were trying to publish.  The database maintenance is complete, and the slowdown is over.  Thanks so much for your patience.

We are experiencing some unusual traffic spikes which are causing some issues with downloads this morning.  We are working hard on the issue and hope to have a resolution very soon.

---

Update: We are back up to full capacity after the team spent the last couple hours fighting off a DDOS attack.  We are very sorry for the interruption this has caused.  It took us a little bit of time to discover the exact problem and then to combat it.  We will be watching everything very closely to ensure we've covered it all.

One of our data providers is have some problems delivering us logs as timely as usual.  As a result, the processing of those logs is being delayed and some of you are not seeing updates in your stats display.

We have tickets open with the provider and will catch up on today's stats when we get the logs delivered.

I am sorry for the inconvenience this causes.

Some of you have been requesting this feature, so we took some time to put it together for you.  At the bottom of the Stats tab you'll now see a link to a CSV file, which contains a stats summary for all of your episodes stats are being tracked for.

Click this link and you'll be prompted to download a CSV file, which can be imported into your favorite spreadsheet application and analyzed to your heart's content.

We have more coming down the pipe in terms of the way you view and make sense of your stats, but we thought that this would be a good followup to the recent engine update and interface tweaks.

Please, let us hear any feedback you have.

The fields in the CSV are as follows:

FILE NAME - the name of the file these numbers are for
PUB DATE - the date the episode for that file was published
ALL DL SYND - all syndicated (aggregator) downloads
ALL DL DIR - all direct (non-aggregator) downloads
UNIQUE DL SYND - unique IP downloads (aggregator)
UNIQUE DL DIR - unique IP downloads (non-aggregator)
LES DL SYND - L.E.S. calculated downloads (aggregator)
LES DL DIR - L.E.S. calculated downloads (non-aggregator)

A portion of our stats engine has a backup that may result in some users stats appearing to be abnormally low over the last couple days. No data has been lost- it's just a little slow to get into the system. As a result, today (and maybe tomorrow) will look like huge spikes as the hits get rolled in. We apologize for the frustration.

For some of the same users affected last night by the hardware outage, there appears to be some residual issues regarding incomplete uploads propagating around the network.  We are in the process of ensuring integrity across the entire delivery network and running some scripts by hand to ensure the full copies of all files are making it out to all the corners of the globe.

We understand the seriousness of this issue- delivering partial files is almost as bad-if not worse- then not being able to deliver at all.  This has our full attention and the team is working hard to get this done quickly.

A portion of our users may be experiencing problems uploading new media (podcast files as well as images) to libsyn right now.  The issue is hardware related and our team is working hard on the issue.

We appreciate your patience and apologize for the inconvenience.

---

Update: The issue has been resolved and all users should be back to normal.

If you find yourself having trouble logging into libsyn.com, please try closing your browser and re-opening it, and/or empty your browser cache, and try to login again.

 We've been tweaking some things on the backend that may make this necessary for some users.  Sorry for any disturbance-

---

Update: We recognize that there is still an issue for many people.  We are working to have it repaired ASAP.

---

Update: We found what we believe to be the final couple issues making it difficult for some users to log in.  Please go through the process to clear your cache one more time and try.  If you cannot login, please email support@libsyn and we'll work each person's issue individually.
Thanks for your patience

Many people are experiencing an issue with their latest episodes not showing up in the iTunes podcast directory- however people subscribed to their RSS feeds are getting them.  It's an issue Apple has addressed and hopes to have fixed as early as tomorrow.

We'll keep you posted if we learn any more.

---

Update:  Got news today from Apple that everything is fully functional again.

There are a few issues this morning which are causing requests for media files to timeout.  The team has been working the issue for a few hours and are putting in some new hardware to handle the post-christmas load.  There are also some databases that need to be cleaned out.

We'll update when things are back to normal.

Stats are stuck for some of our users.  We are aware of the problem and are working on it.  We will keep you updated.