Libsyn Support Blog

Database Maintenance Tonight

Thu, 14 Aug 2008 21:52:00 GMT

We will be performing some maintenance on the database tonight beginning around 11pm eastern time. There may be a few short interruptions of service and we suggest users avoid logging into the system between 11pm and midnight.

Stats Issue Resolved

Tue, 5 Aug 2008 13:06:00 GMT

The stats view is temporarily unavailable this morning due to server issues. There has been no data loss and all new stats are still being collected. The datacenter team is working as quickly as possible to restore the service to all users.

UPDATE: The stats display issue has been resolved. Stats are now displaying for all users. Please contact Libsyn Support at support@libsyn.com if you have any other questions.

Server Issues

Tue, 22 Jul 2008 18:29:00 GMT

We have experienced problems with one of our main Libsyn.com servers early this morning. Our back-up measures have taken effect but there have been momentary disruptions in service while we resolve the issue. Media delivery and user stats have not been affected.

New Uploads Issue Resolved

Sun, 20 Jul 2008 17:57:00 GMT

We are currently experiencing issues with one of our main service providers. Please be advised that new uploads will not become available for download until these issues have been resolved.

--------------------------------------------------------------------------------------

Update 7/21 9:00am
The file serving issue with one of our service providers has been resolved and uploaded files are available for download. Please contact Libsyn Support (support@libsyn.com) if you have any questions.

Stats display unavailable

Mon, 7 Jul 2008 13:44:00 GMT

The stats view may be unavailable from time to time this morning while the backend database is being worked on. There has been no data loss and all new stats are still be collected. The datacenter team is working as quickly as possible to restore the service to all users.

Update (6:05pm EST): The server repair is taking longer then expected, the team is working through the night to repair the service as quickly as possible. The download data is still being collected and no data has been lost as a result of this downtime.

Update (11:15 pm EST): A replacement server has been brought online and about 60% of the data has been copied over. We are attempting to bring up a backup image of the database from yesterday to allow users access to all stats leading up to this mornings failure while the remainder of the data is copying. If you login and see that the last updated date is behind, rest assured that it's still copying, or processing and will catch up very soon.

"Experiencing unusual high traffic" + libsyn.com stats slow loads

Tue, 10 Jun 2008 04:10:00 GMT

There are 2 issues to report on. First, there have been intermittent database errors this evening resulting in this error message for users attempting to login to libsyn.com and to visitors to libsyn blog pages and rss feeds. The engineering team continues to monitor the servers closely and take necessary measures to prevent these errors from occurring again. Media delivery, stats tracking, ad serving, and libsynPRO where not affected by these database errors.

Second, there were some upgrades made today to the libsyn.com stats display system to improve compatibility with Firefox 3.0. This code change caused some temporary issues for some users loading their stats. There was no interruption in stats collection or processing during this time- only an unusually slow load time.

Both issues have been resolved.

Database replaced

Thu, 29 May 2008 22:13:00 GMT

We had to react to an emergency database issue this afternoon which took down the website for a couple hours. There were no interruptions to media downloading, stats processing, and no data was compromised or lost. Logins to libsyn.com were unavailable, and access to blog pages and RSS feeds was intermittent.

All systems have been restored and are now fully operational.

Scheduled downtime Tuesday, 5/13/2008

Mon, 12 May 2008 23:14:00 GMT

On Tuesday, May 13th between 3:00pm - 4:00pm EST web and FTP login will be disabled while we perform database maintenance. During this period, media downloads, RSS feeds, blog pages, and statistics tracking will not be affected. Logging in, uploading, and add comments to blog posts will disabled during this period of time.

FTP uploading

Sat, 3 May 2008 14:15:00 GMT

We are investigating FTP upload issues reported this morning. Will update when there is more information

update: we are monitoring the FTP servers closely as they have been acting a little finicky all day. All is up and operational as of 8:30pm EST. FYI, web uploading has been fine throughout this and for most cases should be a suitable workaround for most cases if you should have trouble getting in through FTP.

Media Delivery Hiccups

Thu, 24 Apr 2008 15:11:00 GMT

Between 3:30am and 10:30am on 4/24, requests for media files were intermittently resulting in "404 File Not Found" errors. The outage was caused by human error / a mis-configured startup sequence on one of our media redirectors. We apologize for the trouble. The problem was discovered around 10am and resolved by 10:30.

Stats v.1.6.1 released

Sun, 13 Apr 2008 01:10:00 GMT

As promised yesterday, we have pushed out a new version of stats interface. Responding to many user's feedback, we have added:

episode columns sorting
CSV downloads of entire episode grid
access to apache/XML logs

In addition, we have changed some behind the scenes stuff to improve performance and stability. Please leave us some feedback on the forums

libsyn stats 1.6 state of the union

Thu, 10 Apr 2008 21:16:00 GMT

Libsyn stats v.1.6 has been out for about a month now. The feedback we have gotten so far has been very positive and many suggestions we received are either in the works or planned for the near future. If you haven't already, please feel free to jump into the discussion on the forums. We are continuing to improve and make fixes to the current system, and will roll out a new release over the weekend.

The development of our next generation stats system has been very exciting. We really can't wait to show off everything stats 2.0 will do! While we progress to that stage, there are some more measures we need to take to improve performance and stability. The first is to purge some old XML and Apache logs. If you are currently using the apache or xml logging feature, any log file older then January 2008 will be purged from the system on 4/18/2008. Following that date, the first of the month will mark the rollover period whereby only the oldest month will be purged. (eg. on 5/1/2008, February logs will be removed,etc, etc..). The download links for the logs will be moved to the 1.6 stats interface and the 1.5.x interface will be officially phased out. The access to historical data back to September 2006 will still be available through the stats interface- nothing is being removed from there. The vast majority of libsyn users will not be affected by this data retention policy.

Finally, while we are repairing the issues being reported of recent episodes not being picked up, the stats display may appear to be a bit erratic. We are restoring to a backup from earlier in the week and then reprocessing the weeks worth of data. We anticipate everything to be back to normal late this evening into the early morning hours.

Thanks to everyone for their feedback and suggestions in helping us build a better stats experience for all users.

update 4/11 12:30pm

The prep work for the brief rollback we were planning for early this morning took longer than expected. We will be performing the rollback this afternoon and will post back here when it is complete.

update 4/11 6:20pm

The rollback is complete. Stats should be current and up to date. We've recovered the missing data from over the last few days (thought there may be slight amounts of irrecoverable data from last weekend). Stay tuned also for some requested features we'll be pushing out in short order.

Latest episodes not showing up in stats

Mon, 7 Apr 2008 18:02:00 GMT

We have been tracking down the bug(s) which is causing the last episodes published by some users. We will keep you published as we learn more.

update: we have temporarily stopped the stats engines- meaning no new updates of any files will occur for the time being. We are working on a plan to recover the data for those users affected over the last several days worth of data.

We do not want to jump the gun on pointing the finger for what happened, it would seem that the recent interruptions with Amazon EC2- which we take advantage of for our stats processing- has caused some unexpected behavior. This would also account for why a swath of users across only some of the partitions were affected.

Update (4/8/2007) 9:00PM: Stats processing resumed earlier this evening. Everything appears to be caught up, and for the most part, missing downloads have been restored. It is possible that a subset of new files got missed as we could not tell the exact time in which each EC2 engine got disrupted. We took a time from 1200 GMT on friday as our roll back point. There were measures taken to mitigate double counting of downloads for files which were tracking fine over that time period, but it is also possible that some double counts occured.

We have repaired this mishap as best we can and are moving forward with the work on the next generation stats system which will handle these types of issues in a much better manner. I apologize for the inconvenience this has caused.

errors logging in / uploading

Tue, 1 Apr 2008 02:34:00 GMT

We are currently experiencing some database issues which is causing problems connecting and uploading new media. We are working to restore service as quickly as possible

update: We will continue to monitor the load on the databases. The disruption was intermittent over the course of a couple hours in the evening on Monday. The team reacted quickly, resolved the issue, restoring the service to those experiencing problems logging in, navigating the website, and uploading media. If you uploaded any files between 9:30 - 11:00 PM EST and are either not seeing them show up in your account or are experiencing 404 file not found errors, you may need to try uploading again.

Laymen's terms explanation of 3/27/08 downtime

Fri, 28 Mar 2008 14:51:00 GMT

To you straight from the mouths of our datacenter team:

"I will try to explain it as easy as possible. at 18:45EST 3-27-2008; The PFC3B line card in the Primary Supervisor of Core1 went into a LOOP; This caused an issue that did not allow Core1 to process Layer3 traffic across the MSFC3. Layer2 however was not affected in this matter which allowed Core2 to continue running normally and pass Layer2 traffic Across the Core1 Layer2 Back-bone. HSRP which is used as a counter-measure; still showed Core1 as functional even though it was experiencing a Loop.

at 19:00EST; a CISCO SMART-NET Tech has been dispatched with additional hardware (SUP, MSFC3, PFC3) in case a defective board was found.

At 19:35EST, Core1 was taken off-line in preparation for Cisco's Arrival.

Upon arrival at about 20:45EST; a debug was generated and CISCO has re-certified the Module. We are waiting on Final Explanation from CISCO as to what caused the PFC3 daughter board to act in the way that it has.

We have taken numerous Physical counter-measures to make sure that this will not happen again. Some of these counter-measures include modification to the HSRP algorythm as well as physical scripting.

Lastly; This incident only affected clients that the HSRP was Active on the Device. We have placed more sophisticated HSRP counter-measures."

Laymen's explanation: there was a hardware failure. Our side of the blame line is not having enough replication of our DNS records in another datacenter. That is being fixed today.

Sorry again for the downtime.

Cogent / Telia

Thu, 20 Mar 2008 20:08:00 GMT

As you may have already heard (and as reported in this article), two major network carriers (Cogent and Telia) have "stopped talking to each other." While Libsyn.com does push a decent amount of traffic through Cogent (which could mean problems for users on the Telia network requesting that content), we are connected to the net through multiple large providers. The word from our main data center is that most, if not all, of the requests for libsyn content from users on the Telia network should get sent through alternate (non-cogent) channels.

If you or someone you know is having problems getting content from libsyn, please find out the IP address of the affected user (try http://whatismyip.com) and send it to Libsyn suport letting them know you're having issues.

Thanks!

stats v.1.6 video demo

Tue, 11 Mar 2008 20:12:00 GMT

We launched v.1.6 of the libsyn stats service yesterday. Here's a 7 minute demonstration video to see what they are all about. Please excuse the fact that I have a cold.. and am not, in fact, the EVP of Stats.

Stats v.1.6 dropping on monday (3/10/2008)

Fri, 7 Mar 2008 08:09:00 GMT

Issues addressed in this release:

lack of updates for new files
low numbers for past 9 months
speed to load stats UI
accuracy of data
user agents list (zune, appleTV, iPhone, miro, etc added)
real trending data over time.. per month/week/day downloads of episodes or entire show
general confusion from v.1.5.2

The "old stats" 1.5.2 will continue to be updated while "new stats" 1.6 continues to catch up for some users. By the end of this month, we intend to stop supporting the old stats with any new updates and only the new stats will be getting new data.

There are currently 2 partitions which contain many popular users and therefore taking longer to re-process that last 9 months worth of data. We've held off an extra couple weeks from releasing this version while we waited for all users to catch up. Our feelings now are why make the 90% suffer for the 10% that are behind. If you are unlucky enough to be in that 10%, please be patient while your data catches up.

For the most recent issues some users have been experiencing this week regarding slow or no updates, those engines are working to catch up. You should see v.1.5.2 numbers updating over the weekend.

Update on stats reprocessing

Fri, 29 Feb 2008 03:43:00 GMT

Just wanted to update everyone on how the rollback process is going. We're happy to report that about 85% of the reprocessing is complete, and our testing is showing that issues with missing data are being resolved as this process progressess. We are also very excited to report that the new stats UI is working, tested, and ready to roll as soon as the re-processing is complete.

Our goal to get the new data to all of you this week looks like it's going to slip into next week. There was a lot of data to run through and it's taking longer then we had hoped. The good news is that the new UI, combined with some new hardware to back it, is a lot faster and more responsive than the previous UI. So not only will your data be easier to understand, but it should load much faster as well.

We will keep everyone updated as things progress.

Stats Update 2/19/2008

Tue, 19 Feb 2008 20:22:00 GMT

Hi everyone. We are happy to report that the system wide stats rollback / recovery is moving right along. We expect the analysis and testing to be complete next week. At that point we'll release the updated numbers and UI updates into your dashboard. Thanks as always for hanging in there.

Replacing Existing Files

Tue, 19 Feb 2008 18:49:00 GMT

Over the past few weeks we've received a number of reports of users replacing existing files, but the new versions of those files never becoming available. We tracked down the root of this problem (related to a recent configuration change with our CDN provider) and got it fixed. Now, when you update an existing file, the new version should be available for download immediately, and the old version should no longer get served.

Service Unavailable / Internal error messages

Fri, 15 Feb 2008 14:13:00 GMT

We are currently investigating an error with recently uploaded media returning an XML error file instead of the media. The problem seems to be with our backend service provider.

We are working with the providers to resolve the issue as quickly as possible.

Update: problem seems to be very widespread and affecting other customers of the amazon S3 service: details from Amazon Forum

The "origin" server for the libsyn CDN service is held on S3. It has historically been very very solid, and I'm sure will have the service back up very quickly. As of 9:30 EST they have not provided an ETA.

Update: Files being uploaded will not be available for download righ away as a result of this issue. Once everything is resolved on amazon's end, we will reload all the recent uploaded files.

Thank you for your patience.

Update: As of 10:30 EST, the CDN is now available again, and all files should now be available. If you notice problems with any media files, please contact support@libsyn.com.

Stats Update 2/11/2008

Mon, 11 Feb 2008 06:22:00 GMT

Dear Users,

We have an update on the status of libsyn.com stats! We know stats accuracy is an ongoing issue and that libsyn users have had to exhibit a lot of patience and swallow a lot of frustration as we've worked through the problems over the years. As we said in a previous post, we have assembled a solid roadmap towards achieving stability and accuracy, and providing industry-leading statistical reports that tell you producers and network administrators exactly what you need to know about the consumption trends of your content.

As promised, we took the first step of identifying and fixing the issues with the current libsyn.com stats (mainly, some episodes published after July 2007 not showing up in the system, or suddenly stop reporting downloads). We sifted through the code that processes the raw data into data we can use in the reports we provide to you, and believe we have identified the root of the problem, as well as the fix for it. We've set up a test environment, applied this fix, and are now in the process of re-processing the raw data from Summer 2007 forward and verifying that this "fixed" data is accurate. Depending on how that verification goes, the reprocessing could be finished in as soon as 7 - 10 days.

While fixing the existing system is an important step, we are really excited about the development that is getting underway for a longer term solution. As we've said in the past, the current iteration of the stats application was built to be scalable, but it just has TOO many moving parts and missing pieces (for example the ability to speedily re-process data, or easily rollback and re-process from an arbitrary date). We are working on a plan that will provide scalability, flexibility for us in terms of administrating the application, and will give us the ability to continue to store historical data indefinitely (a wall that the current system was bound to come up against sooner or later). This is the "data warehouse" model we discussed in a previous post, and is a key component to our long-term plans for the libsynPRO enterprise service. The good news for all of our independent (i.e. libsyn.com) users is that the "libsynPRO" stats engine is also going to be the libsynINDIE stats engine and we are fast tracking the development and integration into both systems. It's still a little too early to give an accurate deliverable date, but we are shooting for early second quarter of this year.

Finally, as a bonus we are working vigorously to implement an updated stats UI into the current libsyn.com dashboard. We were originally waiting for libsyn.com 2.0 to be done and in place before we started introducing new features, but the long-loading XML+Flash interface is pretty long in the tooth, and we figured out a way to shoehorn in the new UI into the old system, so we figured what the heck. We've beaten you guys up so much over the past year regarding stats performance and stability, the least we can do is SHOW you all the data we have been collecting but were unable to deliver in a meaningful way for so long.

So, that's where things stand. We will keep you posted as the reprocessing progresses.

Love, Libsyn

PS - you'll notice that the libsyn.com stats updates are now happening less frequently than normal. We've had to tone-down the update cycle for those during the re-processing, since both the live system and the system running in the background work off of some of the same data

Some URLs Not Redirecting Properly

Tue, 22 Jan 2008 01:58:00 GMT

From about 8:00 to 8:40pm on tonight, some requests for files in the network were not being redirected properly, and in some cases, the wrong file was delivered. We were able to fix the problem as soon as it came to our attention. We apologize for any trouble this may have caused you or your listeners.

Weekend Issues

Mon, 14 Jan 2008 19:32:00 GMT

Hi Everyone. First off: our apologies for this post not being made sooner. We were short-staffed over the weekend, and the most important thing (letting you all know something was up) got overlooked.

There were two issues this weekend affecting some, but not all of Libsyn publishers:

1) Saturday Night / Early Sunday morning around 2:30am Eastern, the datacenter we colocate a bulk of libsyn servers in was performing some routing maintenance on their core routers. We were told to expect a more or less seamless experience, but as it worked out, there was an oversight on their end which resulted in some people not being able to connect (download files, upload files, or check RSS feeds). Alarms started going off on our end around 4am, and we worked with them though the night / into the day on Sunday repairing the problem.

2) During this same time one of the machines that houses a section of users' home directories had a disk that was becoming full. With all of our focus on the other problem, this disk filled up completely, meaning new files didn't get saved properly.

Both of these issues could result in the "Access Denied" XML error that many of you are seeing. This basically means the file was not found on the CDN. Please let our support team know if you're getting this error and, if your file made it to your home directory, then we can force it out to the CDN. If the file isn't in your home directory, re-uploading it will fix the problem.

We're sorry for the trouble this has caused anyone, and are committed to getting any broken files fixed as fast as possible.

Note: We are also having some issues currently with FTP. It looks like a disk went bad on one of our FTP servers. If at all possible, please use Web upload until we update this post saying everything is fixed. When it rains, it pours. Thanks for hanging in there.

Update: The FTP issue has been resolved, so feel free to use the FTP method for upload again.

_________________________
If you are having trouble with access errors please include username AND filename when emailing support@libsyn.com. Thank You!

Stats Report - 12/13/07

Thu, 13 Dec 2007 20:06:00 GMT

This is an update on what is being done to improve the stats -

Over the holidays we will be changing where the statistics are collected. Previously the libsyn service was hosted on multiple cdns; one that was managed in house and then two to three other cdn providers. The statistics were gathered by the redirect media nodes, that would accept requests, refuse bad requests, redirect based on db references to where particular media files are hosted, and collect statistical data. This was not an efficient architecture decision as the redirect nodes were handling too many tasks. While all the stats data was being collected, the system was also introducing noise into our logs that would ultimately choke up the stats parsing engine. This would cause the the stats displays to report improperly or fail to report at all (spinning beach ball). We would then have to go in and remove the noise by hand. This took a significant amount of time for the repairs and for the re-running of data. We would then have to wait as the stats engine caught up to date. The reasons we managed multiple cdns were to keep our costs down for our users, and to allow us to scale our system from the in-house distribution platform launched in '04 to one based on industrial solutions.

What has changed over the last couple of months is that we have migrated our network to a unified cdn partner. Now all media is served from one system. The media is no longer shuffled to different cdns based on demand (an inefficient overhead), or moved to slower servers when archived. Our download performance has been greatly improved for all media. We have also begun migrating our stats data storage from sql based databases to a higher grade platform.

We are now migrating the collection of the stats data off of the redirect nodes. We will be collecting the statistics directly from our cdn partner. This data output is much cleaner and we believe this will improve the performance of the stats crunching engines greatly. We also have been working on a more streamlined reporting tool for the stats tab of your libsyn account. We have to take it one step at a time; first the consolidation of media which is now complete, then the change of collection method to data directly from the cdn. This should bring our performance to an industry best state. When this is completed we can finally overhaul our stats interface based on our users input, and the experiences we have gained from Wizzard Media's advertising initiatives.

We are very positive about the plan we have in place and we will give you updates as they progress. So much of the work has been behind the scenes architecture changes, and we realize our users cannot see any immediate benefits. But we assure you we are committed to making libsyn the best publishing platform for podcasters.

We are happy to say that in early 2008 you will see the fruition of all the work that has been done to fortify the stats system.

Stats not updating

Mon, 3 Dec 2007 17:22:00 GMT

We have discovered a break in the stats engine which occurred on friday, 11/30, which have caused updates to backup for most users over the weekend. The systems are being restarted and should pickup the updates over the course of today and into tomorrow.

There is also the standing issue of missing data dating back over the last several months. That is a longer process to fix, please be patient while we work on that as well.

We apologize for the inconvenience.

File Delivery Errors: Resolution

Sun, 2 Dec 2007 00:19:00 GMT

After spending the better part of the day yesterday trying to understand what was going on with the delivery network (for anyone who didn't notice, requests for newly uploaded files were failing with 503 and 504 errors), we were able to trace the problem to one of our service providers who had rolled out some new, less than stable code for our account.

Normally we would let you all know if we were planning an upgrade or anything that might interrupt service, but this particular upgrade happened without notification for us, so we couldn't notify you. We apologize for the inconvenience the problems caused, and you can be sure we are making it very clear that something like this cannot happen again. The availability of your files is vital to both your livelihood and ours.

We finally were able to get the new code rolled back, which almost immediately fixed the problem, around 12:30am this morning.

If you find you are having problems with any of your files, please open a ticket with support, and make sure you include the URL of the file in question.

Thanks

Error delivering files

Fri, 30 Nov 2007 16:39:00 GMT

We are currently experiencing issues with our file delivery CDN. You may experience intermittent issues as we are working with the data provider to fix the issue.

More updates to follow.

update: We have had been working all day between our origin data center and our CDN to resolve the issue. The problem continues to plague new files uploaded, and older files that may not have been downloaded in a while. We are looking to put some stop-gap fixes in place while the issues with the CDN get resolved.

We apologize for the inconvenience this issue has caused.

Picking up stats data

Mon, 19 Nov 2007 20:35:00 GMT

We have been working to restart the log aggregator which has fallen out of sync. It will take several hours, but all data from yesterday and today will eventually catch up. Thank you for your patience as we maintain the current system while continue to rebuild the libsyn stats engine from the ground up.

Forum posting bug and Blog Page update

Wed, 14 Nov 2007 16:17:00 GMT

The ability to post new topics or reply to old posts has been down for a few days, we inadvertently broke something when installing our new anti-spam measures. We apologize for any inconvenience this may have caused our users, but rest assured that the forums are back up and in working order, with all the new anti-spam features in place.

We have also updated the blog pages with similar measures in an attempt to lessen the amount of spam users receive. Please email us at support@libsyn.com if you have any problems regarding these new changes.

login to libsyn.com and RSS feeds timeouts

Wed, 7 Nov 2007 16:34:00 GMT

A combination of different activities caused some run away processes to back up traffic going to libsyn.com's main application servers and database. This resulted in slow page loads while hitting libsyn blogs, rss feeds, and login sessions on libsyn.com. Media delivery was not affected by this issue.

The system in place to alert us of these issues failed due to some mis configuration after a recent system change. This caused a slower than usual response time from our support team.

All issues have been resolved, the safety nets have been reconfigured. We apologize for inconvenience and are working to deliver a redesigned backend for libsyn.com which will dramatically improve the quality of service for all libsyn users. Thank you for your patience while we rebuild libsyn.

slow updating stats

Tue, 16 Oct 2007 14:25:00 GMT

We have been working to keep the stats engine updating regularly over the past week while we diagnose some hardware and configuration issues. There were some disk issues that we cleared up this afternoon and as of this evening we believe all queues should be caught up.

While we believe all the issues have been resolved regarding the queue of log files to process, over the next several days updates may slow down again from time to time.

The slow updates, or "stalled stats" issue is a high priority concern that the team is working on improving. Thank you for your continued patience and support while we rebuild our stats system from the ground up.

Possible corruption of large media files

Sat, 6 Oct 2007 22:52:00 GMT

After several days of investing strange media playback behavior, we have discovered a bug in our content storage system. The details are still not completely solid for which files could be affected, but as of right now we believe it to be larger (100mb+) files potentially getting corrupted as we push media into our storage array. It is not every file.. the bug occurs intermittently making it very hard to pin down. There have been sporadic reports of archived files getting corrupted and today some reports of new media files not playing properly once downloaded.

This issue is being addressed and we are working to detect the incomplete files and replace the corrupted or cut-off files from backup. Depending on the media type, sometimes the result of this bug will be a file that is just cut-off, or in the case of certain video codecs, the movie not playing at all. Quicktime reports "invalid public atom was found in the movie" in some cases, however in iTunes the file just doesn't show up.

While we are making a sweep of the terabytes of data we have stored, if you are experiencing "weird" issues like these please let us know immediately and we will investigate your issue specifically.

Update 10/8: The issue, unfortunately, seems to have extended beyond large files. About 5% of the files in our archives have been cut-off. We believe this has resulted from a script which ran to repair the mime-types of these files. The mime-type problem was causing files to not "stream" or progressive play download. The "fix" seems to have done more damage then good.

We are running some scripts now to recover as many of the files from backups as we can. Unfortunately there will likely be files which will need to be re-uploaded. We will contact the users directly who this pertains to via email.

There have been extra validation checks put into place to ensure this will never happen again.

Update 10/9: We have scoured through the backups that we do have for some of the media and have found about 50% of the potentiallycorrupted files. Users who have been affected will receive an email with a list of the files that we have detected a file size mismatch on. You will be asked to verify those files by downloading them and if they are cut-off or corrupted, please reupload those files at your earliest convenience. Once uploaded, they will be automatically pushed into the archives and you the storage will not be counted towards your monthly quota.

Stay tuned for more updates to come.

Files Not Showing Up in Media Tab?

Tue, 2 Oct 2007 05:13:00 GMT

A problem was discovered earlier today that was causing some newly uploaded files to not appear in the media tab of the libsyn dashboard. This issue was limited to files with illegal characters or spaces in the filenames. We fixed all instances of this that we were able to identify. If do happen to have files that you uploaded not showing up, please let us.

For future reference, we "clean up" filenames by replacing spaces with underscores, and removing characters which are not alphanumeric (a-z, 0-9), periods (.), hyphens (-), or underscores (_).

Network Upgrade Announcement

Sat, 29 Sep 2007 20:07:00 GMT

As mentioned in a previous post, we have been laying the groundwork for some time now, in preparation for a substantial upgrade to our network infrastructure. In general we will be simplifying and consolidating the system that delivers your media to your audiences. The switchover to the new infrastructure should be transparent from yours and your listeners points of view.

We'd like to encourage those of you who manage your own RSS feeds and/or who post links to your files directly to double check those links and make sure they follow the standard format of http://media.libsyn.com/media/username/filename

If any of your links don't look like that, there's a good chance those links will stop working in the near future.

The actual "switch" will be flipped starting late Sunday night. We will keep you informed on the progress of things, and invite you to send any questions or concerns to support@libsyn.com

errors publishing when pinging directories

Sat, 29 Sep 2007 04:29:00 GMT

There was an issue when publishing new episodes if you had selected the option to also ping the podcast directories. The episodes were getting published fine, however users were seeing an error message.

That problem has been fixed. Please let us know about any more publishing-releted issues as we have made some recent changes to the publishing system.

Sorry again for the inconvenience.

FTP

Fri, 28 Sep 2007 19:30:00 GMT

Some of you may have had a hard time connecting to Libsyn FTP in the last day or so. One of the FTP servers was not handling logins properly and had to be re-started, and you should be able to access your FTP accounts once again.

Web Uploader Updates

Fri, 28 Sep 2007 09:19:00 GMT

Hi everyone. This is just a note to let you all know that we have implemented some changes to the web-uploader that we wanted you to be aware of. The majority of the changes are on the back-end, and are part of our preparation for an upcoming network upgrade (details to be announced later in the day). On the front-end, you should only notice some cosmetic changes, as well as a bit more info on the status of your upload as it progresses.

If you have any issues with the updated uploader, please catch us on the support chat at http://support.libsyn.com, or drop an email to support@libsyn.com.

Update 9/28 3:06pm

Thanks to those who alerted us that in some cases, publishing new posts was not working properly (for example if an apostraphe was included in the post title). We've addressed this issue and it should be working as expected now.

Some Files 404ing

Sat, 22 Sep 2007 23:30:00 GMT

We have multiple reports of download requests resulting in "404 File Not Found" errors. We have verified this issue and have an open support ticket with the CDN that is the source of the problem. We will work with them to get this fixed as quickly as possible, and apologize for the inconvenience.

Update: This issue was confirmed to be resolved at 8:45pm

Update: 9/23: The issues with the 404 on media files requests is happening again. We were told it was a corruption of our CDN's core database occurred yesterday after a server crash, however it was repaired. We are in contact with our providers and working to resolve the issue and/or temporarily route around it while the problems persist.

What's going on with stats

Tue, 18 Sep 2007 15:33:00 GMT

This article if not for the faint of heart. It gets a little long, a little technical, but it is not the type of thing we could throw into bullet points and be confident we've covered everything we wanted to convey. So, if you are totally happy with the way our service is working for you and don't have a lot of interest in stats, go about your business and just know we are working on making them much better and if you should ever need them, they'll be there for you.

For the stats connoisseurs, please read this forum post.

Stats Spike

Tue, 4 Sep 2007 19:22:00 GMT

There were a set of logs that were being blocked from access by the log aggregator so they weren't getting parsed into the stats system... these were from 8/31 to present. We just pushed them all in today, so users with low hits may see that this resolves the problem... and users in general will see a spike in their hits today, especially for archived files.

Removing Email Comment Notifications

Sat, 1 Sep 2007 02:50:00 GMT

Due to the pervasiveness of commenting spam we are removing email notifications when comments are left to your posts. Spamming is a very frustrating obstacle with blogs/commenting and unfortunately spammers often can work around whatever hurdles that may be in their way. We'll continue to explore methods of curbing their attacks, but for now we're at least going to cut back on some of the junk that may fill your inbox. Thanks

Low stats numbers continued

Sat, 1 Sep 2007 02:20:00 GMT

We are investigating some more reports of lower-then-expected download numbers. There have been some adjustments made and we need a couple complete days of data to see if the graphs smooth out.

We will keep you posted as we learn more about what's going on and who may be affected.

Low stats numbers

Thu, 30 Aug 2007 03:49:00 GMT

Update from the stats investigation department: We have discovered a bug in the file filter which strains out extraneous files from the counters and it was causing valid hits to get skipped. We are rolling back a few days worth of data and rerunning right now. Expect it to recover pretty quickly and accurate numbers to follow.

Will update when we have an ETA

[update 8/31/2007 7:04am]

The rollback is complete. Thanks to everyone who sent in detailed reports of inconsistencies they were seeing. That kind of information was VERY helpful in getting to the bottom of the problem.

DNS Update

Sat, 25 Aug 2007 19:45:00 GMT

Almost exactly 48 hours after the initial drop we're seeing almost full recovery of traffic. We will continue to monitor the graphs and traffic curves.

Stats are Back

Fri, 24 Aug 2007 15:43:00 GMT

Finally! After a long wait, I'm happy to announce the stats are caught up and once again running in near-real-time (there may be a 5-10 minute lag between when a file is downloaded and when it appears in your stats display).

Once again, if you would like to report any anomalies to the support team, please be sure to include specifics like the file names of affected files, your user name, and try to describe exactly which numbers seem to be off-kilter. Writing us to say, "my stats look weird can you check it out?" only slows things down as our support staff will then need to try and decipher exactly what you are referring to. We appreciate your cooperation on this.

Update on DNS issues

Fri, 24 Aug 2007 12:47:00 GMT

I understand that you may not be getting this message, or if you are there's a chance some of your audience is still not getting your podcasts. We will post this around to as many of the boards as we can to keep people informed of the status.

We are seeing an upturn in the MRTG usage graphs which indicates a good portion of the name servers have updated and are now resolving correctly for libsyn.com domains. It is hard for us to give a blanket ETA as right now it's totally dependent on local ISP's and even home computer DNS caches. It is tremendously frustrating to know what the problem is but have no way to improve the situation.

Momentary DNS Misconfiguration

Thu, 23 Aug 2007 19:09:00 GMT

At about 2:30 EST on Thursday, Aug 23, there was a momentary DNS misconfiguration that caused some users to be unable to view the site, or in fact, resolve anything under the libsyn.com domain. The situation was recognized and resolved immediately, but due to the nature of the DNS system, some users have cached the incorrect data, either on their computer, or on the ISP DNS servers. As with everything DNS related, the caches will time out, and the correct data will be received, and we've been told the maximum time this will take is 24 hours. (It's usually much shorter, however) In the meantime, a potential remedy for end users is a reboot of their computer to hopefully flush that local cache. We apologize for any downtime.

Archive Expansion

Tue, 21 Aug 2007 16:17:00 GMT

Some listeners may be noticing slow-ish downloads when requesting a file from the archives. This is a capacity issue that is currently being addressed. Basically a section of our archives is nearing its capacity, so we are taking steps to expand the infrastructure that handles that piece of the archives, which will resolve the current slowness and allow much more room for future growth.

This issue is intermittent and potentially affects about 20% of archived files. We expect to have a short-term fix in place sometime this afternoon. Long term build-out of our infrastructure is continuing.

Stats ETA

Mon, 13 Aug 2007 10:42:00 GMT

Another update: based on the rate things are going, we expect the system to take about another 36 hours to become fully current. That puts us at around 7pm Eastern, Tuesday evening. Hold tight everyone, we're almost there.

[updated 8/15 7:28am ]

As some have noticed, our 7pm Tuesday night ETA was a little optimistic. As we speak the whole system is catching up and is currently processing numbers for August 7th. The speed of this catchup process is dependent on a number of of factors, so rather than giving another estimate which may or may not be right, I'll say that all pistons are firing, the stats engines are chewing through all of the raw data, and we'll post here again when we are completely up to date.

In the meantime, you can check out your numbers for everything up until the 7th by looking under the stats tab of your libsyn dashboard. Again: there will be inflated numbers for July 23rd and for some of you, July 24th. These spikes represent downloads of your show that were fed into the system "late" so they were counted as having happened on the 23rd/24th when in fact they happened in the days and weeks previous to that. In total this was a few hundred thousand hits that were added after the fact, and this will affect some users more than others.

If you feel that something about your numbers isn't right, and you send our support team a report of it, please help us out by being as specific as possible. Include your user name, the file name(s) affected, and the specific numbers that seem to be wrong. This helps us greatly to figure out what, if anything, is going on.

[update 8/21 12:26pm]

Sorry folks, been neglecting updates on the blog. There is a running thread on the state of the stats system here on the forums.

The latest:

Hi folks. Your daily update: the rollback is still catching up. We've made less progress than I had hoped in the last 24 hours and I'm sorry to say it's my fault. I "tripped over the cord" so to speak while trying to tweak things to speed up the process, which resulted in a number of hours without forward movement yesterday. I apologize for that, I was really trying to gain us a day or so in this catchup process. The misstep was corrected and we started plugging ahead again early last evening.

We're now working on 8/18 logs. I will keep my hands off the thing and let it run it's course. In response to the questions about files not showing up in stats, you are correct, if we haven't processed the logs yet for the period in time when downloads for your newest files started, then your files won't show up in the stats display.

For those who are interested, throughout this process we have also identified some bottlenecks that could be resolved by a different hardware configuration. While it would not buy us much time to move to new hardware at the moment, we are putting together a plan to upgrade the hardware the new collection service runs on, which will keep things nice and snappy during normal operation, and will help make a rollback such as the current one (god forbid we ever have to do this again) less excruciatingly slow.

Rollback Update...

Fri, 10 Aug 2007 10:32:00 GMT

The stats rollback / restoration was delayed slightly as Pittsburgh got hit was some severe thunderstorms which caused damage and electricity outages throughout the city. The process is once again underway. Stay tuned...

[ update: 1:15pm ]

Things are looking good, everyone. It's taking a little longer than we expected to get this thing going because we found a couple of "gotchas" that would have caused the numbers to look erratic (hits being bunched into days they didn't belong in, mostly). Though all the numbers would have been counted, we figured at this point, we should smooth out those wrinkles so that when your stats come back, they accurately represent your audience's downloading behavior.

I know we say it a lot, but thanks again for hanging in there with us.

[update 1:50pm]

Because this is a rollback, we are literally rolling the clocks back on the engine that reports data to your stats display. Effectively, what you see right now is the stats as they were mid-day on 7/23. Shortly, the system will start moving forward in time as it reinserts the missing data. Since the clocks have been set back to a couple of weeks ago, recent entries in the stats folder may have seemed to disappear. They should reappear as we climb back through time from July 23rd to the present.
There may also be a slight spike on July 23rd as we're inserting some data that never made it to the the entries from beforehand.
Thanks for your patience.

[update 11:50pm]

We're facing some more challenges. After watching carefully as the collection service runs through ten full days of data, we are seeing what we believe to be inconsistencies. Further investigation is ongoing. There is a good chance we will be resetting / re-starting the rollback process more than once before the weekend is through. We're trying our hardest to have fresh, accurate, up to date numbers for you all when the week begins.

Stats Rollback Underway

Thu, 9 Aug 2007 17:11:00 GMT

Starting in just a few minutes, we are going to begin the rollback / restore of the stats system. Effectively, the system is going to be reset to July 23rd and will re-crunch all of the numbers moving forward. Once this process begins, we'll take some measurements and update this post with the estimated time when we'll be all caught up again.

Quick Update

Thu, 9 Aug 2007 07:32:00 GMT

Hi Everyone- I thought I'd take a moment to give a quick update on the status of the stats system:

We are fairly confident we have wiped out all of the pesky bugs that were wreaking such havoc the past few weeks. To be absolutely sure there are no gotchas we may have missed, we've opted to extend the testing into Thursday.

We will update again with specifics on the rollback / restore when that process begins.

Are Your Stats Bugging Out? Read This:

Tue, 7 Aug 2007 01:05:00 GMT

[ Originally posted at http://forum.libsyn.com/viewtopic.php?t=7743 ]

Friends,

I know the latest stats issues are causing a bit of an uproar, so I am pulling my head out from under the hood for a minute to explain what's going on.

We have discovered some bugs in the new collection service we put into place around the 23rd of July. (The collection service is the piece of the stats system that feeds the raw hits into a queue to be analyzed by the 'stats-engine'). We've been tracking these issues for a few weeks, and are fully aware that there is a serious problem with the numbers you are seeing. Some people are reporting super high spikes, others are seeing numbers that are way too low. Some see a combination of both, and still others are not getting stats data on some files at all.

We are currently peeling back the layers to find and repair the bugs that have caused these problems. Admittedly, we were probably a bit hasty in pushing the new collection service into production so fast. The old service, however, was on its last leg and could no longer scale to the level of demand being placed on it, so we felt like we had to make the leap. It's probably safe to say that we didn't quite stick that landing.

The good news is, we have isolated and repaired a number of obscure bugs. We have also narrowed the field on a major issue that is causing the collection service to intermittently crash (which can cause you to see low points, followed by extremely high points in your daily numbers).

Once we have routed out the last of these bugs, we have plans to COMPLETELY RESTORE the past data from ~7/23 to present. That is to say, we are going to roll the system back in time to the 23rd, and re-run the numbers going forward. This will eliminate the erratic reporting you've witnessed in the past weeks, and should restore the normal trends that you're used to seeing for your shows.

We know this is extremely frustrating for many of you, It is for us as well. While we've come a long way in the past few years, we are still learning to manage and scale a system of this size. So if you can find it in you, we'd really appreciate your patience and support as we work this one out.

Thank you, sincerely,

Marty

Low stats reporting

Wed, 1 Aug 2007 20:21:00 GMT

Unfortunately we have to report again that some users are seeing lower then normal download numbers over the past several days. We have confirmed that all hits are being logged, however we have yet to recreate the problem exactly. Looking in aggregate at our entire network, the overall average does not appear to be drastically lower then normal, however on an individual basis, we have definitely confirmed there are downloads being logged however not making it into the stats system.

This is not a simple problem- like a piece of hardware breaking that can be easily fixed. It is a bug deep in the new stats aggregator code. The team is working to resolve the issue as quickly as possible, however it may take several days or longer to determine the exact problem, roll back the system and restore the missed downloads.

We appreciate everyone's patience and understanding while we work through this. There is a light at the end of tunnel as we continue to grow our development team and build out newer and better tools and engines to publish, serve and track media.

Stats Issues Cont...

Tue, 31 Jul 2007 18:18:00 GMT

Just wanted to drop a line and say that we're not ignoring the stats issues. We're sorry for all the frustration its causing and we are still looking into it. The predominant issues still seem to be on either end of the spectrum: low numbers, or numbers that are too high. It seems that the high numbers are showing up mostly in the web downloads section. We're continuing to investigate and will post more soon. Thanks for your patience.

Archiver is Running Behind

Tue, 31 Jul 2007 03:14:00 GMT

The archival process has fallen behind and is now catching up. Files that were due to archive on July 25th or later are affected by this issue. The archiver is cranking through them. If this is affecting your ability to upload new episodes, please get in touch with our support team and they will help you free up the space that you need.

Stats: Podcatcher vs Web Downloads

Fri, 27 Jul 2007 04:55:00 GMT

We identified and fixed the bug that was causing the stats system to be unable to differentiate between podcatcher and web downloads. We have decided to leave the mis-categorization of these numbers over the past few days as-is. To be very clear: all downloads should have been counted, however downloads by podcatchers like iTunes, Doppler, Juice, etc will not be counted as podcacther (syndicated) hits. Instead they appear as "web" downloads, but the total numbers should be accurate.

Anyone who suspects otherwise, please let us know. If you do send in a report of inaccurate stats, please be as detailed as possible, showing the trend for specific numbers you have seen in the past, versus the numbers outside that trend that you are seeing on a particular episode.

Also, please remember that there will likely be an expected low-point in your numbers for the days leading up to 7/23, and then a spike in your numbers from 7/23 - 7/26.

Thank you friends,

Libsyn

Stats On Hold, Will Resume Shortly

Tue, 24 Jul 2007 09:27:00 GMT

One of the main challenges we've faced with the growth of the Libsyn system has been feeding the stats that are collected into the stats system for analysis. Some time has been dedicated in past weeks to developing a system that will make this a more reliable process. We are now putting that system into place. As we do, users will notice that their stats numbers appear to be frozen, or lower than normal. This is a temporary situation and your statistics will begin updating again by the end of the day today. Thanks for your patience as we work to continue to improve upon the Libsyn system.

Update (7/25, 4:00pm)

We finished our rollout of the new stats collection system last night, and turned the stats engines back on. While we expected there to be some degree of "spikiness" for yesterdays and todays numbers (since stats were stopped for a number of hours and some logs had not been fed into the system for about a week, those downloads would get counted when they were finally fed into the system), we have had some reports of extremely high numbers (and especially higher than normal volumes of web downloads). We are currently examining whether these are the spikes we expected, or whether something else is going on. We will update this post with our findings.

Our Findings (7/26, 3:30am)

Thanks to everyone for the helpful reports of issues you've observed in the past day or so. What we found were four main issues which are explained as follows:

1) "Stats are showing extremely high numbers (for 7/23 - 7/26)"

As mentioned earlier in this post, we did expect some higher than normal numbers when we resumed feeding the hits into the system. In some cases though, these numbers were extremely high, on the order of 2 - 3x the normal numbers. The reports we received of this were eventually attributed to a known issue with version 3.2 of the TVTonic client, where the client was repeatedly downloading a file or files again and again. In one instance, we discovered 6,000+ downloads of the same file over a relatively short period of time, from a single IP address using this client. The folks at TVTonic are on it though and their latest release is not affected by this bug.

2) "Stats are not appearing for new episodes"

This was a human error where one of the log sources was not activated properly in our new collection system, so the hits were just sitting there, not being fed in. This was resolved about an hour ago, so those of you who experienced this problem, you should start seeing your numbers shortly.

3) "Stats display shows a low number of podcatcher vs web downloads"

This issue is still under investigation. The system is somehow failing to identify hits from known "podcatcher" clients as such. All hits are still being counted but in some cases, downloads by iTunes, Juice, etc show up as "web downloads."

4) "Stats are frozen!"

While the new collection service is more robust than its predecessor, we have not quite worked out all the kinks yet, so certain scenarios cause it to crash. Once re-started, it recovers perfectly, but during the time between the crash and the re-start, stats appear to be frozen. We are putting an alarm on this service while we search for a solution to the underlying issue that might cause it to fall over. This should minimize the time between a crash and a restart, should it happen again.

iTunes Submission Troubles

Wed, 4 Jul 2007 00:16:00 GMT

Got word from the good folks at Apple:

Podcast submission is down right now. See this forum post:

http://discussions.apple.com/ann.jspa?annID=383

When it disappears, the process will be fixed. No ETA but our engineers are working on it now.

Sorry for the inconvenience.

"Why am I seeing this weird MySQL error when I try to download a file?"

Thu, 14 Jun 2007 03:08:00 GMT

This was a bug in the new media delivery mechanism we recently rolled out. We've addressed the immediate problem and are taking steps to prevent similar incidents in the future. Our apologies to anyone this affected.

Stats back updating

Fri, 8 Jun 2007 15:38:00 GMT

Early this morning the update scripts finished and the stats engines were turned back on. Please report any individual issues to support email and we'll track them down one by one. Again, everyone may notice a slight decrease over the 3 days ( 5/29 - 6/1 ) so please don't report that.

Thank you for your patience

No stats update for next 24 hours

Wed, 6 Jun 2007 15:48:00 GMT

In order to repair the inflated stats and complete the roll back and reprocess, we need to stop the stats engine. We will continue to collect all data and everything will get counted, however the display will not update until sometime tomorrow. We have rough calculations on how long this will take, but will keep everyone updated if it looks like it's going to take longer then expected.

We appreciate your patience during this process. Thanks

Stats fix update

Tue, 5 Jun 2007 22:00:00 GMT

Another hard day of work to correct the inflated stats. We wanted to give everyone an update on where things stand. The 321 users affected by the issue turned out to be an underestimate. We honestly don't know the exact number of users affected, so we are rolling back to an earlier backup, well before this issue began.

In short, the total requests for files downloaded on may 30th, june 1st and june 2nd will be slightly lower then normal, but the les and unique ip counts shouldn't be affected. For more technical description, keep reading.

We are currently scrubbing the request queue and reducing any duplication of IP address and request for file by the same IP and same user agent from the same libsyn node server to 1 per day for may 30th ,june 1st, and june 2nd. What that means that for those 3 days worth of data, the total requests, LES impressions, and unique IP counts will be closer to each other then normal. LES and unique IP counts should be affected the least by this procedure. This is the only way we can correct the current inflation which resulted from a combination of database lock and an automated script that wasn't properly keeping track of which files it had already processed. This is a unique scenario that hasn't happened before, and monitoring has been put in place to prevent it from happening in the future.

I know this is alot more detail then most people need and can really use, but we wanted to be clear on what we were doing. We expect this whole process to take another couple days. There's hundreds or millions of lines to process and clean up and reinsert.

Stats inflation continued

Mon, 4 Jun 2007 22:04:00 GMT

Update on the fix for the inflated stats of last week. Many users may still see numbers which are much larger than normal. Files downloaded between June 1st and June 2nd are particularly inflated. Our solution for this is to roll back the database to a backup version then we will backfill the data with the correct stats numbers. This may take several days to fully complete.

Daily hits from June 3rd on are correct so the temporary false inflation over the time span mentioned will eventually we back to normal. We apologize for problems this causes but we feel it is vital to provide accurate statistical information.

Inflated stats for some users

Fri, 1 Jun 2007 14:14:00 GMT

We have been tracking down a number of support requests from users who are commenting on much larger then normal downloads for their latest episodes. After many hours of analysis and bug tracking, we have found the cause of the inflation that is affecting only a portion of users (321 to be exact).

We are working on the fix to correct the last couple days worth of stats loads which have been duplicated way beyond proportion, however it may take several days to completely roll back everyone's data. We apologize for the false sense of excitement some people have experienced when they saw they're latest efforts have been downloaded millions instead of thousands of times, and we are working hard to ensure everything is true and accurate. The bug that caused this glitch was (of course) an edge case and has been fixed so that this shouldn't happen again

Again, sorry for the trip up, everything going forward from yesterday on should be accurate and we'll do our best to fix the error asap. Thank you

Stats catching up

Wed, 23 May 2007 19:53:00 GMT

Many users may be experiencing lower then normal downloads over the last couple days. We are working on the problem and ensuring those hits make their way back into the system as soon as possible. We don't have an exact ETA as of yet when all the data will be back in, but we have the data logged and are working to get it reinserted into the matrix (not making light of the situation, that's really what the machines are called). We are sorry for the continuous headaches this is causing.

Update (5/24): The program which was running overnight to catch up the missed hits is still running this morning :( Around 11:30 last night, Hoopes called me with the epiphany of what was wrong, why the hits for some users (not all) have been low or not updating over the last couple days (since around the 20th) and why our alarm system didn't alert us to the problem. It basically had to do with the way we parse the logs for one of the network providers we use, and that script was running out of memory. We're pushing more and more data and the logs are getting bigger and bigger every time this script runs (once every few minutes). I know 98% of you don't care about any of these and want to know 'when the hell are stats going to update again?!' The answer to that is hopefully by the end of the day, but there are no guarantees. Since we're inserting into the system which is also running and keeping up with all the demands, it takes a little longer then normal to insert into the databases. More updates to follow

Update (5/25): The backlog of stats should be all caught up now. You may see a swell in yesterday or today's numbers- those are hits collected since the 20th, but got inserted into the system over yesterday and today. Please meebo/skype/ or email us directly if you are still experiencing problems.

Stats recovery almost complete

Fri, 11 May 2007 17:46:00 GMT

Hey guys, just taking a minute to update everyone on the status of stats on libsyn. The backqueue we mentioned a few days ago is about 70% complete at this point, the final partitions of users are being loaded back in now. That means there's going to be an unusual bump of hits showing up as 5/8, but are really hits dating back to the beginning of the month. The good news is, data has not been lost.. it's only going to show up a little bit lumpy in historical charts of hits over time.

A few other issues have been cleared up, such as the timing for a reconcile script that compiles some summary data. Many users were noticing discrepancies between total impression and overall distribution totals. That should be resolved now. Some people may have seen their total impressions number go down, as previously it was higher then the overall total number in the overall distribution block of the stats display. This is because stats records that you have chosen to delete were still being calculated into that total impressions number, and now that is no longer the case. Other weird issues we are dealing with on a case by case basis for individual users.

We are continuing to work hard to provide timely and accurate statistical data. The popularity of all of your shows has really kept us busy. Great job everyone.

Stats hits in backqueue

Tue, 8 May 2007 15:35:00 GMT

There were a couple days worth of hits backlogged on one of our front end machine which is part of the cause of low numbers across the board the last couple days. There are other issues we are still debugging, and will post updates as we have them.

stats very low for today

Wed, 2 May 2007 04:36:00 GMT

Everyone may notice their numbers for tuesday were very low. There were some process that were handing the stats system and holding up data base inserts. Those requests are being back loaded into the system now and will show up very soon.

To be honest with everyone, we've been having a hell of a time with stats over the past couple weeks. A reload of the libsynPRO stats over this past weekend blew up and caused problems for both libsynPRO and libsynINDIE users. Before that (and in still on going) there are some wayward bugs that are causing miscalculations for users that have deleted stats in the past, basically accounting for a mismatch in total impressions vs overall total distribution (a number that SHOULD be the same).

It's safe to say that with the recent growth libsyn has seen, the log processing system is in need of an overhaul. We have high confidence in our engine, but the data input piece has some flaws. New designs are being worked on as we speak. In the meantime, we are fixing as many pieces to handle the demand as possible, and working hard to keep everyone satisfied- providing as accurate and timely data as possible. As always, we understand the importance of the stats and we appreciate everyone's patients and understanding.

Stats

Mon, 30 Apr 2007 16:15:00 GMT

We are currently undergoing a deep investigation and audit of the libsyn stats system. There are a puzzling combination of issues plaguing many users- but not all! That's what's been the hardest thing to diagnose. Most random spot checks we do pulls up a user that has everything working correctly.

We have been compiling the issues people have been writing in about, posting on the forums, etc. The most commonly reported issues are:

stats display very slow to load, or otherwise spiners forever

LAA not updating, reporting 0, or otherwise just wrong

total impressions != overall total distribution

drastically higher total requests then unique or LES audience on some episodes

randomly lower then normal audience for individual episodes

Everything we've been seeing points to corrupted data somewhere in the pipeline.

We will keep everyone updated on our findings and appreciate your patience while we work through the issues.

update: The stats display is currently taking an extremely long time to load. In some instances it takes several minutes to bring up your data. This is a result of some massive data loading going on in the background. We are working to implement a caching system to speed up the retrievals of the xml right now to ease the pain.

Partial Network Issues

Sun, 29 Apr 2007 19:57:00 GMT

One of our datacenters is experiencing some downtime. Media delivery has been adjusted to compensate, but there may be intermittent download problems.
This will be remedied as soon as possible, and this post will be updated with further information.
Thank you for your patience.

Stats down

Fri, 20 Apr 2007 16:44:00 GMT

There was a hardware transfer that occurred early this morning which, in the process, temporarily took down some stats processing hardware by mistake. The techs at the datacenter are working to bring the hardware back online ASAP. We will keep you updating as we get an ETA from the datacenter.

UPDATE: Everything should be back to normal, thanks to our datacenter guys for being johnny-on-the-spot. Please email support@libsyn.com if anything looks greatly awry.

Stats Logs Frozen - Will Resume Updating Shortly

Mon, 16 Apr 2007 20:34:00 GMT

Users who have enabled the logging option for their stats will notice that the logs are not updating. We are performing some maintenance on this log-writing system to ensure continuity of service and near-real-time performance. We will have updates back online by tomorow morning.
Thanks for understanding.

Update (4/17/2007 5:50am EDT) - Stats logs are back online and updating regularly.

Stats reporting continues to be a problem

Thu, 12 Apr 2007 16:44:00 GMT

We have confirmed there continues to be a problem with the libsyn stats system. Across the board downloads have been reporting lower then they actually should be and are not consistent with the bandwidth or raw logs we have been collecting. We are confident all the data is being collected, there is however a bug somewhere in the calculation and reporting process which has been difficult to locate and squash. Our team is working on the problem and we will keep everyone updated on our progress.

Update (4/13/2007): We discovered a segment of our delivery network was not sending its logs to our stats engine to be processed. This was part of the problem why a lot of users were seeing lower then normal numbers over the past several days. Those logs are in the queue to be processed and will make their way back to your show. The datastamp on those hits, however, will be for the same day so it may appear that there was a surge of traffic for that day.

Database issues resolved

Tue, 10 Apr 2007 15:13:00 GMT

There were intermittent DNS and database conflicts yesterday which have resulted in a few different issues for some users. First off, if you couldn't login yesterday or had trouble accessing some parts of the site or uploading, that problem is resolved. If you were able to upload new media, however the database was not updated, you may have problems today where files do not download, or there are no stats being recorded, etc. We have attempted to sweep through and reconcile the missing database records, however we may have missed some and you should contact us via meebo, or support@libsyn and we'll look into issues on an individual basis. Finally, if you had written into support, but did not get a response yesterday, our email support system was being affected by the same issues, and we are working through the backlog now. We apologize for the headache this may have caused.

Stats updates coming slowly

Thu, 5 Apr 2007 19:38:00 GMT

There have been some "low days" many users are reporting over the last several days. All the data has been collected, and is in the queue to be processed. The end result may be, unfortunately, some lopsided days where it seems like all of a sudden your shows are getting a ton of downloads as the system catches up. The problem of out of sequence logs has been a ailment for sometime. The team is working on a solution so that this problem no longer plagues the libsyn stats system. Thank you for your patience.

Status on issues over the last day

Fri, 23 Mar 2007 17:39:00 GMT

There's been several issues over the past 24 hours which have all now been resolved. Stats:

The stats system has been playing catchup for the last day. All data is being collected, it's in the queue to be processed, and you should be seeing timely updates once again when everything is caught up. In the meantime, it may appear your numbers are not increasing and new files may not yet appear in the stats display

Download timeouts

There was an issue with one of our data providers which was causing some users, and especially archived files, to be slow to start and time out resulting in no downloads. The fail-over mechanism that should have recovered the issue has been fixed as well as the problem with the provider itself.

Archiving

The archiving issue which has been causing files to not move to the archives has been resolved.

Kiptronic insertions

Libsyn users who are using Kiptronic were experiencing issues with insertions not working. That has been resolved.

We are sorry the status of these issues were not published earlier- the team has been working hard to fix all the problems. Please feel free to contact us through meebo or skype (libsynsupport).

Login problems

Thu, 8 Mar 2007 16:09:00 GMT

We are battling some database issues right now. Logins are currently being affected by the issue. We will keep you posted on the status.

Update: (12:30) The database issues have been repaired. In an effort to reduce database load, we have temporarily taken down the libsyn directory. The code will be revamped and brought back online quicker and better then ever very soon.

Temporary Upload Problems

Fri, 2 Mar 2007 17:43:00 GMT

Many users may have been experiencing problems uploading and/or connecting to FTP (same issue) this morning.

The issue has been resolved. Sorry for the interruption.

Stats Not Updating

Mon, 26 Feb 2007 18:04:00 GMT

We are getting reports that the stats system is not updating. We are looking into this now, and hope to have things back up shortly. It is important to note that we are not losing any data. It is just the parsing system that is malfunctioning.

Update: 2:00pm - All stats are updating regularly again.

Partial Network Outage

Sun, 25 Feb 2007 21:40:00 GMT

We are experiencing a partial network outage that will likely result in some of you or your listeners being unable to connect to libsyn.com. We have our datacenter techs on the scene working on the issue and will post an update when we have more info.

Update: 5:11pm

We are back up at full capacity. In reviewing our network graphs, it appears as though some but not all of the Libsyn traffic was hindered. This "brown out" lasted for around 20 miuntes. For those of you who are geek-savvy: 2 backbone connections at our primary data center lost connectivity. We worked with the techs there to manually route around the problem and get back to full capacity as quickly as possible.

We apologize for any inconvenience this may have caused.

FTP / Blogpage issues

Wed, 21 Feb 2007 16:09:00 GMT

We are still cleaning up from some issues last night, one lingering one is some flakeyness with FTP this morning.

Please be sure when you are connecting via ftp to use the host name: ftp.libsyn.com and also make sure your connection type is set to passive.

We are working hard to get things back to normal with FTP as well as some problems people are having modifying or viewing their blog pages with proper templates. All issues will be resolved ASAP.

Sorry for the inconvenience.

Update: There was some hardware issues that was/is causing the blogpage template issues. While the techs in the datacenter work on the issue, we can repair templates on an individual basis. If you don't modify your blogtemplate during this outtage, you shouldn't notice any problems. If you do, and your page is broken, let us know via meebo chat and we'll fix you up.

Update: The issue with the blog templates should be back to normal now, however the FTP uploading issue has rolled into a system-wide uploading problem which we are now fighting. Please hold off attempting to upload anything new for the time being while we work this out.

Thank you

Update: It's been a rough day, uploading should be working ok now. There were some file propagation issues, but everything should be synced up now.

Portion of users affected by network problems

Wed, 21 Feb 2007 03:24:00 GMT

Many users may have experienced trouble uploading new media or accessing their blog pages this evening as a segment of the network is being worked on. The problem should be taken care of shortly and everything will be back to normal.

The issue has caused us to push back the scheduled maintenance of the database a little, but we should still make it in the 10-2am EST window.

Scheduled Maintenance for 2/20/2007

Tue, 20 Feb 2007 13:34:00 GMT

We will be performing some routine maintenance on our main database server this evening, February 20th, between 10pm and 2am. This process should be completely transparent, however there may be a momentary blip in service as we switch from our main database server to the backup server. This maintenance will not affect media delivery and should have minimal impact on your RSS feeds and Blog Pages.

We will post an update when the work is complete.

Update [3:41am] - Well, it took longer than expected, and as some of you may have noticed, the backup DB made for a considerably slower experience if you were trying to publish. The database maintenance is complete, and the slowdown is over. Thanks so much for your patience.

Traffic Spikes causing slow downloads/ timeouts

Fri, 16 Feb 2007 16:00:00 GMT

We are experiencing some unusual traffic spikes which are causing some issues with downloads this morning. We are working hard on the issue and hope to have a resolution very soon.

Update: We are back up to full capacity after the team spent the last couple hours fighting off a DDOS attack. We are very sorry for the interruption this has caused. It took us a little bit of time to discover the exact problem and then to combat it. We will be watching everything very closely to ensure we've covered it all.

Stats issue for several hundred users

Thu, 25 Jan 2007 22:48:00 GMT

One of our data providers is have some problems delivering us logs as timely as usual. As a result, the processing of those logs is being delayed and some of you are not seeing updates in your stats display.

We have tickets open with the provider and will catch up on today's stats when we get the logs delivered.

I am sorry for the inconvenience this causes.

Stats Feature, By Request

Mon, 22 Jan 2007 05:44:00 GMT

Some of you have been requesting this feature, so we took some time to put it together for you. At the bottom of the Stats tab you'll now see a link to a CSV file, which contains a stats summary for all of your episodes stats are being tracked for.

Click this link and you'll be prompted to download a CSV file, which can be imported into your favorite spreadsheet application and analyzed to your heart's content.

We have more coming down the pipe in terms of the way you view and make sense of your stats, but we thought that this would be a good followup to the recent engine update and interface tweaks.

Please, let us hear any feedback you have.

The fields in the CSV are as follows:

FILE NAME - the name of the file these numbers are for
PUB DATE - the date the episode for that file was published
ALL DL SYND - all syndicated (aggregator) downloads
ALL DL DIR - all direct (non-aggregator) downloads
UNIQUE DL SYND - unique IP downloads (aggregator)
UNIQUE DL DIR - unique IP downloads (non-aggregator)
LES DL SYND - L.E.S. calculated downloads (aggregator)
LES DL DIR - L.E.S. calculated downloads (non-aggregator)

Stats delayed

Wed, 17 Jan 2007 16:06:00 GMT

A portion of our stats engine has a backup that may result in some users stats appearing to be abnormally low over the last couple days. No data has been lost- it's just a little slow to get into the system. As a result, today (and maybe tomorrow) will look like huge spikes as the hits get rolled in. We apologize for the frustration.

Synchronization issues as a part of yesterday's hardware issue

Fri, 5 Jan 2007 05:40:00 GMT

For some of the same users affected last night by the hardware outage, there appears to be some residual issues regarding incomplete uploads propagating around the network. We are in the process of ensuring integrity across the entire delivery network and running some scripts by hand to ensure the full copies of all files are making it out to all the corners of the globe.

We understand the seriousness of this issue- delivering partial files is almost as bad-if not worse- then not being able to deliver at all. This has our full attention and the team is working hard to get this done quickly.

Uploading issues for some users

Thu, 4 Jan 2007 03:51:00 GMT

A portion of our users may be experiencing problems uploading new media (podcast files as well as images) to libsyn right now. The issue is hardware related and our team is working hard on the issue.

We appreciate your patience and apologize for the inconvenience.

Update: The issue has been resolved and all users should be back to normal.

trouble logging in?

Thu, 28 Dec 2006 19:00:00 GMT

If you find yourself having trouble logging into libsyn.com, please try closing your browser and re-opening it, and/or empty your browser cache, and try to login again.

We've been tweaking some things on the backend that may make this necessary for some users. Sorry for any disturbance-

Update: We recognize that there is still an issue for many people. We are working to have it repaired ASAP.

Update: We found what we believe to be the final couple issues making it difficult for some users to log in. Please go through the process to clear your cache one more time and try. If you cannot login, please email support@libsyn and we'll work each person's issue individually.
Thanks for your patience

Episodes not showing up in iTunes

Wed, 27 Dec 2006 22:19:00 GMT

Many people are experiencing an issue with their latest episodes not showing up in the iTunes podcast directory- however people subscribed to their RSS feeds are getting them. It's an issue Apple has addressed and hopes to have fixed as early as tomorrow.

We'll keep you posted if we learn any more.

Update: Got news today from Apple that everything is fully functional again.

Slow connections and timeouts

Wed, 27 Dec 2006 15:22:00 GMT

There are a few issues this morning which are causing requests for media files to timeout. The team has been working the issue for a few hours and are putting in some new hardware to handle the post-christmas load. There are also some databases that need to be cleaned out.

We'll update when things are back to normal.

Stats Freeze

Thu, 7 Dec 2006 20:44:00 GMT

Stats are stuck for some of our users. We are aware of the problem and are working on it. We will keep you updated.

Stats display v 1.5.1

Wed, 15 Nov 2006 05:09:00 GMT

After about 1,000 emails all requesting the same feature, we finally caught on. The stats display version 1.5.1 was rolled out tonight and features (are you ready stats junkies?) "Remove record" feature!

So finally you can go in remove errant files from your stats display. Please be cautious when doing this- we don't want to create a new problem while we fix another ("Dear support, I accidently deleted the stats for all my shows, HELP!!!!!!!!"). Ok? There's an "are you sure" popup that you have to confirm before the record is actually deleted, but still, pay attention to what you are doing.

Currently, when you delete a single episode's stats, it has to go in and re-parse the XML to draw that display again. I know, it's lame, but the double buffering array thing we are doing has given me a double headache, and well, you shouldn't have to use this feature too often and I wanted to get it out into your hands sooner rather then later.

Also, a slight performance boost was given to the XML loader in this version. Shows with several years of data still take 20+ seconds to load, but hopefully alot of you will notice snappier load times.

More great stuff to come!

Stats engine 3.0 turned on

Mon, 13 Nov 2006 22:49:00 GMT

It's been a long time coming (too long) but today we are upgrading our server software for statistics. This is primarily a behind-the-scenes upgrade intended to bring increased reliability to our statistics system, and to enable future release of highly desired features such as remove-your-own-files from stats (as in the case of an erroneous upload), the Libsyn Active Audience number, and eventually Geo-stats, among many others. With our new system we have noticed that the stats display takes a little longer to load for users with many files, and will be working to bring the display time down in that case.

We will release more updates as we update the front end display as well

Stretched out Stats

Sun, 12 Nov 2006 22:48:00 GMT

We are aware of the current stats issues people are seeing in which recent files show more days worth of stats than they have been available. This is due to some changes we have made to our network to distribute more files across more server farms, but the problem will be resolved when we upgrade our stats software within the next 24-48 hours.

Afternoon Service Interruptions / Slowdowns

Tue, 7 Nov 2006 23:03:00 GMT

From around 4 - 6pm EST the libsyn media delivery network experienced slowdowns and interruptions, causing slow connections, and/or timeouts when media files were requested.

At present we have associated these problems with at least one malicious IP address and are investigating further to see if we can pinpoint any other causes for the network-wide disruption.

Update: We are still investigating the true reason for last night- and very early this mornings- performance issues. I am fairly convinced the IP's that we blocked coincidentally helped relieve the issue, which was caused in fact by something else. We may have jumped the gun a little bit reporting that it was a malicious attack. Thanks for everyone's support as we keep wrestling with the logs and graphs and reports to find the real problem and make sure it's fixed.

Updating stats engine

Tue, 7 Nov 2006 03:28:00 GMT

Hey guys, you may notice some stats irregularities this evening as we're cutting over to a new processing engine. All steps towards a better, more reliable, more accurate stats system.

Thanks for your patience.

Stats Updating

Wed, 1 Nov 2006 02:15:00 GMT

Updates were made to stats tonight that put the cruncher a bit behind. Things should be caught up and restored in a couple hours.
Thanks

Slow downloads, time outs, 404's for some users

Mon, 30 Oct 2006 04:35:00 GMT

Sorry this post is late coming out. The team is working on some issues surrounding a flood of requests on part of the network. Many users are affected and we're working the issue.

Updates to be released as we figure it out. Thank you for your patience.

Update: The segment of our network seeing the problems today was caused by a propagation issue. New files were not getting out to delivery nodes quick enough which was causing a backup in requests, slow downloads and 404- file not found errors for new files. All the mounting traffic then started to cause older files to be hard to access as well, while the network was absorbing lots of requests it couldn't fill.

The nodes are "catching up" now with the new media, and will be turned back on as soon as that is complete.

The technical reason, for those who are interested, is being linked to the network cards on these media nodes. We are checking into the reasons why they seem to have issues, and will replace the hardware, or update the os, etc as needed.

As always, we are sorry for the disturbance to your audience for these issues, and regret it took us so long to get to the bottom of the problem.