8:30pm EST Learned from Network Solutions that the reason our DNS changes are taking so long to update is because they are doing maintenance at this time. Data center is working to restore the primary and backup systems. Back up soon.

If you were seeing that message, you can login and reupload now without any problems.

Even if you were not seeing that message, when you attempted to download new files, you may have been getting a 500 Internal Server Error message from time to time. That issue was also fixed yesterday.

Both issues were intermittent and it took a few days longer to track down with the holiday and everyone traveling. We were also slow to update this blog with the issue. Please accept our apologies.]]> Media Delivery Mon, 29 Dec 2008 13:05:00 GMT http://support.libsyn.com/index.php?post_id=417043# No http://support.libsyn.com/index.php?post_id=406617#

Update (6:05pm EST): The server repair is taking longer then expected, the team is working through the night to repair the service as quickly as possible. The download data is still being collected and no data has been lost as a result of this downtime.

Update (11:15 pm EST): A replacement server has been brought online and about 60% of the data has been copied over. We are attempting to bring up a backup image of the database from yesterday to allow users access to all stats leading up to this mornings failure while the remainder of the data is copying. If you login and see that the last updated date is behind, rest assured that it's still copying, or processing and will catch up very soon.]]> Stats Mon, 7 Jul 2008 13:44:00 GMT http://support.libsyn.com/index.php?post_id=356531# No http://support.libsyn.com/index.php?post_id=348148# Second, there were some upgrades made today to the libsyn.com stats display system to improve compatibility with Firefox 3.0. This code change caused some temporary issues for some users loading their stats. There was no interruption in stats collection or processing during this time- only an unusually slow load time.

Both issues have been resolved.]]> Bugs Tue, 10 Jun 2008 04:10:00 GMT http://support.libsyn.com/index.php?post_id=348148# No http://support.libsyn.com/index.php?post_id=344435# All systems have been restored and are now fully operational.]]> general Thu, 29 May 2008 22:13:00 GMT http://support.libsyn.com/index.php?post_id=344435# No http://support.libsyn.com/index.php?post_id=338676#

]]> announcements Mon, 12 May 2008 23:14:00 GMT http://support.libsyn.com/index.php?post_id=338676# No http://support.libsyn.com/index.php?post_id=335432# update: we are monitoring the FTP servers closely as they have been acting a little finicky all day. All is up and operational as of 8:30pm EST. FYI, web uploading has been fine throughout this and for most cases should be a suitable workaround for most cases if you should have trouble getting in through FTP.]]> Bugs Sat, 3 May 2008 14:15:00 GMT http://support.libsyn.com/index.php?post_id=335432# No http://support.libsyn.com/index.php?post_id=332248# ]]> Media Delivery Thu, 24 Apr 2008 15:11:00 GMT http://support.libsyn.com/index.php?post_id=332248# No http://support.libsyn.com/index.php?post_id=327905#

The development of our next generation stats system has been very exciting. We really can't wait to show off everything stats 2.0 will do! While we progress to that stage, there are some more measures we need to take to improve performance and stability. The first is to purge some old XML and Apache logs. If you are currently using the apache or xml logging feature, any log file older then January 2008 will be purged from the system on 4/18/2008. Following that date, the first of the month will mark the rollover period whereby only the oldest month will be purged. (eg. on 5/1/2008, February logs will be removed,etc, etc..). The download links for the logs will be moved to the 1.6 stats interface and the 1.5.x interface will be officially phased out. The access to historical data back to September 2006 will still be available through the stats interface- nothing is being removed from there. The vast majority of libsyn users will not be affected by this data retention policy.

Finally, while we are repairing the issues being reported of recent episodes not being picked up, the stats display may appear to be a bit erratic. We are restoring to a backup from earlier in the week and then reprocessing the weeks worth of data. We anticipate everything to be back to normal late this evening into the early morning hours.

Thanks to everyone for their feedback and suggestions in helping us build a better stats experience for all users.

We do not want to jump the gun on pointing the finger for what happened, it would seem that the recent interruptions with Amazon EC2- which we take advantage of for our stats processing- has caused some unexpected behavior. This would also account for why a swath of users across only some of the partitions were affected.

We have repaired this mishap as best we can and are moving forward with the work on the next generation stats system which will handle these types of issues in a much better manner. I apologize for the inconvenience this has caused.]]> Stats Mon, 7 Apr 2008 18:02:00 GMT http://support.libsyn.com/index.php?post_id=326140# No http://support.libsyn.com/index.php?post_id=323505# update: We will continue to monitor the load on the databases. The disruption was intermittent over the course of a couple hours in the evening on Monday. The team reacted quickly, resolved the issue, restoring the service to those experiencing problems logging in, navigating the website, and uploading media. If you uploaded any files between 9:30 - 11:00 PM EST and are either not seeing them show up in your account or are experiencing 404 file not found errors, you may need to try uploading again. ]]> podcasts Tue, 1 Apr 2008 02:34:00 GMT http://support.libsyn.com/index.php?post_id=323505# No http://support.libsyn.com/index.php?post_id=322248# To you straight from the mouths of our datacenter team:

"I will try to explain it as easy as possible. at 18:45EST 3-27-2008; The PFC3B line card in the Primary Supervisor of Core1 went into a LOOP; This caused an issue that did not allow Core1 to process Layer3 traffic across the MSFC3. Layer2 however was not affected in this matter which allowed Core2 to continue running normally and pass Layer2 traffic Across the Core1 Layer2 Back-bone. HSRP which is used as a counter-measure; still showed Core1 as functional even though it was experiencing a Loop.

at 19:00EST; a CISCO SMART-NET Tech has been dispatched with additional hardware (SUP, MSFC3, PFC3) in case a defective board was found.

At 19:35EST, Core1 was taken off-line in preparation for Cisco's Arrival.

Upon arrival at about 20:45EST; a debug was generated and CISCO has re-certified the Module. We are waiting on Final Explanation from CISCO as to what caused the PFC3 daughter board to act in the way that it has.

We have taken numerous Physical counter-measures to make sure that this will not happen again. Some of these counter-measures include modification to the HSRP algorythm as well as physical scripting.

Lastly; This incident only affected clients that the HSRP was Active on the Device. We have placed more sophisticated HSRP counter-measures."

Laymen's explanation: there was a hardware failure. Our side of the blame line is not having enough replication of our DNS records in another datacenter. That is being fixed today.

Sorry again for the downtime.]]> general Fri, 28 Mar 2008 14:51:00 GMT http://support.libsyn.com/index.php?post_id=322248# No http://support.libsyn.com/index.php?post_id=319639# this article), two major network carriers (Cogent and Telia) have "stopped talking to each other."   While Libsyn.com does push a decent amount of traffic through Cogent (which could mean problems for users on the Telia network requesting that content), we are connected to the net through multiple large providers.  The word from our main data center is that most, if not all, of the requests for libsyn content from users on the Telia network should get sent through alternate (non-cogent) channels.

If you or someone you know is having problems getting content from libsyn, please find out the IP address of the affected user (try http://whatismyip.com) and send it to Libsyn suport letting them know you're having issues.

Thanks!


]]> general Thu, 20 Mar 2008 20:08:00 GMT http://support.libsyn.com/index.php?post_id=319639# No http://support.libsyn.com/index.php?post_id=316392# Stats Tue, 11 Mar 2008 20:12:00 GMT http://support.libsyn.com/index.php?post_id=316392# No http://support.libsyn.com/index.php?post_id=314843#

1. lack of updates for new files
2. low numbers for past 9 months
3. speed to load stats UI
4. accuracy of data
5. user agents list (zune, appleTV, iPhone, miro, etc added)
6. real trending data over time.. per month/week/day downloads of episodes or entire show
7. general confusion from v.1.5.2

We will keep everyone updated as things progress.

Update: problem seems to be very widespread and affecting other customers of the amazon S3 service: details from Amazon Forum

The "origin" server for the libsyn CDN service is held on S3. It has historically been very very solid, and I'm sure will have the service back up very quickly. As of 9:30 EST they have not provided an ETA.

Update: Files being uploaded will not be available for download righ away as a result of this issue. Once everything is resolved on amazon's end, we will reload all the recent uploaded files.

Thank you for your patience.

Update: As of 10:30 EST, the CDN is now available again, and all files should now be available. If you notice problems with any media files, please contact support@libsyn.com.

We apologize for the inconvenience.

update: We have had been working all day between our origin data center and our CDN to resolve the issue. The problem continues to plague new files uploaded, and older files that may not have been downloaded in a while. We are looking to put some stop-gap fixes in place while the issues with the CDN get resolved.

We apologize for the inconvenience this issue has caused.

The slow updates, or "stalled stats" issue is a high priority concern that the team is working on improving. Thank you for your continued patience and support while we rebuild our stats system from the ground up.

While we are making a sweep of the terabytes of data we have stored, if you are experiencing "weird" issues like these please let us know immediately and we will investigate your issue specifically.

We are running some scripts now to recover as many of the files from backups as we can. Unfortunately there will likely be files which will need to be re-uploaded. We will contact the users directly who this pertains to via email.

There have been extra validation checks put into place to ensure this will never happen again.

Stay tuned for more updates to come.

This is not a simple problem- like a piece of hardware breaking that can be easily fixed. It is a bug deep in the new stats aggregator code. The team is working to resolve the issue as quickly as possible, however it may take several days or longer to determine the exact problem, roll back the system and restore the missed downloads.

We appreciate everyone's patience and understanding while we work through this. There is a light at the end of tunnel as we continue to grow our development team and build out newer and better tools and engines to publish, serve and track media.

In short, the total requests for files downloaded on may 30th, june 1st and june 2nd will be slightly lower then normal, but the les and unique ip counts shouldn't be affected. For more technical description, keep reading.

We are currently scrubbing the request queue and reducing any duplication of IP address and request for file by the same IP and same user agent from the same libsyn node server to 1 per day for may 30th ,june 1st, and june 2nd. What that means that for those 3 days worth of data, the total requests, LES impressions, and unique IP counts will be closer to each other then normal. LES and unique IP counts should be affected the least by this procedure. This is the only way we can correct the current inflation which resulted from a combination of database lock and an automated script that wasn't properly keeping track of which files it had already processed. This is a unique scenario that hasn't happened before, and monitoring has been put in place to prevent it from happening in the future.

I know this is alot more detail then most people need and can really use, but we wanted to be clear on what we were doing. We expect this whole process to take another couple days. There's hundreds or millions of lines to process and clean up and reinsert.

Again, sorry for the trip up, everything going forward from yesterday on should be accurate and we'll do our best to fix the error asap. Thank you

We are continuing to work hard to provide timely and accurate statistical data. The popularity of all of your shows has really kept us busy. Great job everyone.

To be honest with everyone, we've been having a hell of a time with stats over the past couple weeks. A reload of the libsynPRO stats over this past weekend blew up and caused problems for both libsynPRO and libsynINDIE users. Before that (and in still on going) there are some wayward bugs that are causing miscalculations for users that have deleted stats in the past, basically accounting for a mismatch in total impressions vs overall total distribution (a number that SHOULD be the same).

It's safe to say that with the recent growth libsyn has seen, the log processing system is in need of an overhaul. We have high confidence in our engine, but the data input piece has some flaws. New designs are being worked on as we speak. In the meantime, we are fixing as many pieces to handle the demand as possible, and working hard to keep everyone satisfied- providing as accurate and timely data as possible. As always, we understand the importance of the stats and we appreciate everyone's patients and understanding.

Everything we've been seeing points to corrupted data somewhere in the pipeline.

We will keep everyone updated on our findings and appreciate your patience while we work through the issues.

The stats system has been playing catchup for the last day. All data is being collected, it's in the queue to be processed, and you should be seeing timely updates once again when everything is caught up. In the meantime, it may appear your numbers are not increasing and new files may not yet appear in the stats display

There was an issue with one of our data providers which was causing some users, and especially archived files, to be slow to start and time out resulting in no downloads. The fail-over mechanism that should have recovered the issue has been fixed as well as the problem with the provider itself.

The archiving issue which has been causing files to not move to the archives has been resolved.

Libsyn users who are using Kiptronic were experiencing issues with insertions not working. That has been resolved.

Update: We are still investigating the true reason for last night- and very early this mornings- performance issues. I am fairly convinced the IP's that we blocked coincidentally helped relieve the issue, which was caused in fact by something else. We may have jumped the gun a little bit reporting that it was a malicious attack. Thanks for everyone's support as we keep wrestling with the logs and graphs and reports to find the real problem and make sure it's fixed.

Update: We've made some major changes to the network that we feel will balance things out much better. The distribution and balancing technique we were using needed some updating. We're going to need to see tomorrow at peek times to see if we really have things tweaked perfectly.

As luck would have it, while doing some other general maintanence over the weekend, where we were adding some redundency to other key parts of the system, the back kinda broke on a segment of the network. Most new users (who've signed up in the last month or so) were affected, so we're not really starting out with our best foot forward with a lot of you. I hope you can understand that these kinds of things happen to everyone, and hope you stick with us- as you can be assured we're going to stick with you- through thick and thin. We're going to keep killing ourselves (and hopefully not each other) to ensure the best, most reliable, most stable system for all you content creators.
-Dave Chekan

The stats processing is currently going through a massive overhaul. Part of the upgrades will include some new hardware. We understand the importance of timely, accurate statistics, and second to reliable media delivery, it's the thing you all really care most about. We're on it. it's going to take a little time, thank you for understanding.

The patch that had been applied to the previous nodes has now been applied. I don't know how many more times you're going to accept our apologizes on this (the most critical or critical) error. We now need to make sure the stickies around our monitors have prominence that say "Add to the new build script!!!" on every new fix and tweak.

Sorry to start off your day with this. Our hope is that we caught the issue in time as to not have affected too many downloads.

The move has been delayed some as our techs at the datacenter double and triple check wiring, configurations, and routing for all of our equipment.  We will make a new post here with at least 24 hours notice of when the actual upgrade is going to take place.

--

Our primary datacenter is upgrading and expanding (partially to meet our growing demands). We have been working through a series of updates and are a reaching a point where there will been some physical hardware shifted around this coming week. Through careful planning and our redundancy in multiple datacenters, we intend to keep mission critical functions, like serving media to your audience, up and functioning at all times. There will however be a short period on Tuesday, May 16th where you will be unable to upload new media, login to post shows or check stats for up to an hour.

We will be dialing in the exact times as this stage of the process grows closer, and will post an update with  the latest news. This estimate is the worst-case scenario, and our technical team will be in constant  communication with the folks "on the ground," ready to react to any issues that should arise during the transition.

As always, we want you all to have complete clarity and understanding of what we are doing and that we are focused on ensuring the least inconvenience and downtime possible. Thanks

• bold
• italics
• font colors

1. Numbered
2. lists

Now that it's the 21st century, we have finally put some work into our documentation wiki. http://docs.libsyn.com is now updated a little more (working on it even as we type this email) and we openly welcome all your libsyn zealots to come help us fill in the cracks. Now, we know you all would do it just for your unconditional love of podcasting, but we do have plans to reward the big contributors (that goes for the community forum as well). Everything from sticker, pins, t-shirts, free accounts, marty's car.. will be given as small tokens of our appreciation to those who have continued to give back to the community. Having a good set of documentation will greatly cut down on the number of support emails we currently get. We'll try to emerge a practical compliment between the forum and the wiki so new users (and old) can self diagnose and help themselves understand the common issues of libsyn. Then a quick stop at the http://support.libsyn.com blog for system alerts and issues, and then last resort an email.

We hope everyone is enjoying the new hardware that has been put in place over the last couple weeks. Good news coming from the Kiptronic camp as well in terms of new sponsors coming into the marketplace very soon as well as some awesome new features we'll be excited to share with everyone soon. I will take a quick minute to talk about one of those, and that is libsyn stats. As mentioned on our stats display for months now, RSS stat feeds are almost ready to roll. What is also going to happen, is those users who choose to sign up for the kiptronic marketplace will automatically get their libsyn stats fed into the kiptronic system and displayed to potential sponsors. IF YOU ARE NOT USING KIPTRONIC, YOUR STATS WILL NOT BE SHARED WITH THEM. That's very important to note that we will not disclose your information to anyone who you have not given express consent to. The way you authorize the sharing of your audience data with other libsyn partners (soon to come) is through the advanced settings tab. We are also considering releasing podcast rankings to the public for those who wish to opt in. We welcome your feedback on these topics, either via private email or in the public community forum.

As always, thank you all for letting us syndicate your content. It is an honor. Thank you for hangin with libsyn, through thick and thin, and helping to build the greatest network of podcasts in the 'verse. (Firefly fans, let me hear ya)

Leaving you with a little support podcast that was sent to us recently. Wouldn't it be funny if everyone sent us audio support requests?

let me answer that.. . no

At least one person has pointed out that a number of files that were archived on Friday were resulting in 404's. This was because of a short archive run that went astray. We believed we had restored all of the missing files, but if we missed one user, it is possible there are more. If you notice 404's on any of your archived files, please let us know right away.

Finally, while we are hustling behind the scenes here at libsyn hq, Dave M. has been on the road meeting with various friends in the industry. In this picture, Dave is discussing some funding options with our new financial analyst... stay tuned for more.

Thank you and sorry for taking an extra couple days to resolve this problem

To be clear, requests from all browsers (IE, Firefox, Safari, etc) and iTunes will continue to be honored while we are working through this issue

Aside from just being a long winded excuse for why we have been slow to answer support emails, or why the site was locked out to European users during their 'prime time' a few nights ago, or why earlier in the week we crippled our delivery capability by almost 50%, I would also like to share some near-term improvements that everyone should enjoy as the results of our work this week.

The bad.
So the big upgrade actually started last week at this time. I'm pretty sure we've said it so many times already, but currently, libsyn is still a big hobby for all of us here. No one's getting paid and we all have (somewhat demanding) day jobs... just like most of you podcasters, right? So, we all started last friday night putting into motion the "big switchover." We had been planning, and designing, and coding, and preparing for weeks leading up to this, and given the wall we were approaching in terms of storage, we had to start putting the designs into play. Friday was a very good prep day, set things up for a saturday switch, still leaving all day sunday for bug fixes. After a few unforeseen hangups on saturday, the big switchover was pushed off till sunday. The notion was to do it early sunday, giving us all day to bug fix before the big "sunday night rush" that we typically see from people uploading in time for monday morning (aka Hell Morning).

Needless to say, sunday morning turned into sunday evening and before we knew it, it was 2AM and our backs were against the wall. We HAD to start the switchover. We started the wheels in motion when, something that could best be compared to virtually tripping over the power cord happened- actually, it was more like reaching for the cold water tap and turning the hot. It wasn't a _mistake_ in action, instead, the right action was performed in the wrong place. The result: our highly tuned, prized media delivery network turned 180 degrees and ate itself for 20 seconds. The outcome was almost 1/2 the media mirrored on our system was vaporized. Gone. Poof. With less than 7 hours before the big monday morning rush, strung out on Red Bull and Bawls, 3 days behind on sleep, we gasped.

Within 20 seconds after realizing what had happened, the team composed themselves and took to the painful process of rebuilding the data structures. In order to backfill 10,000+ files (20-50MB each) across 30 servers quickly, we had no choice but to pull the plug on media delivery until we could get caught up. Believe me when I say making that call to cut the network is the absolute last thing we ever want to do. But, by 10AM we were back online and serving at around 75-80% capacity. Monday hit, and hurt, but overall we saw a less than 10% deduction in overall numbers for the day. The big traffic, which pushes well past midnight EST, makes the self-healing process for the network 100x harder. We licked our wounds overnight, but it wasn't enough to get the network back to 100% by Tuesday morning. Tuesday's usually go somewhat slower, between the big Mondays and Wednesdays of the week. But with our tail still between our legs a bit, it was another rough day. The feedburner timeouts where mitigated somewhat by some sharp collaboration between Rick of feedburner and our Marty Mulligan, but overall delivery and uploads and the like were slower than we wanted to see. The idea was to start all over again with the 'big switchover' tuesday night, however given that we wouldn't have enough manpower around wednesday morning (while part of the team was traveling) we opted for a wednesday after-work start. Wednesday went a little better now that we were back to 100%, but we were still getting dangerously close to a point of media storage that we didn't want to be. That clock was ticking and we knew we had to get the switchover started. Ever hear 'a watched pot never boils' ? Well watching an s-copy of 130gigs never finishes. The hours were pushing well past midnight, and once again, we were consumed with a few other gotchas that surfaced in preparation for the big move.

So then there was thursday. We had pretty much run out of time, and the move needed to happen. All the data was now synced between the old servers and the new cluster. Everyone had a good dinner and we sat down around 8PM. We all had our hit lists. The skype call was crystal clear that night. All systems were go for a midnight "plug-swap" .... ..

And then there's this blurr period.

Anyone ever notice that clocks skip everything between 10pm and 1am? It's true. And so, we were looking at 1:00am and still no action. Throughout the week, we had posted a notice in the support blog, but then kept pulling it offline as the dates and times kept changing. We figured anyone who actually follows that was pretty much fed up with us, and since we had calculated everything regarding the switchover, we figured 10-20 minutes of lock-out wouldn't affect anyone too much. We decided that in an hour we would make our move. We posted to support.libsyn.com that at 2AM est. we would lock down uploads, prevent anyone from logging in, and that everything would be fine in less than an hour. Now, here's where software engineering meets the real, cold, lonely world: 2am comes, we break from our virtual skype huddle and begin to execute like a killer attack squad. In 5 minutes we have all the scripts changed over that need to be changed, we're logging in and testing uploads, downloads, sideloads, and crossloads.. everything is great. Skype (y)'s abound. Talks of the (beer) celebration began. Hoopes even began to (flex). Then things got... well... weird. Pages were loading super slow. And one by one the little things didn't really work right anymore. We were now stuck mid-integration and a elusive bug was running the system amok. With all of our heads down in the code, we burrowed our way through the scripts and data, config files and schemas, trying to find the crux of the problem. This is where I (Dave the narrator of this story) sorta shut down.

I was thinking alot about the day job i had to wake up to in 4 hours. Hoopes and Marty were pretty much speaking in binary tongues as I slowly slipped from consciousness. I jolted up a moment later- damn it was 4am! We were locked out for 2 hours now and seemingly hopeless for being online anytime soon! This is where (pardon the expression) the boys showed their "juevos". I really don't even know what happened between 5 and 7:30, but the next thing I knew, we were just about ready to re-enable logins and let the world in to our new system. It was something miraculous... or Hoopes-tastic, but we turned the key, the engine roared, and we peeled out. Ya, the blinkers didn't really work yet, and there was a piece or two still hanging off, but we were able to drive, and today I can say, we have really opened her up and are having a pretty smooth ride. The pieces that we're still putting back together are new user signups, and anything involving the system-level user accounts: changing disk quota, password changes, etc. Quickcast was broken for bit, but is now working again as are all of the blog-page features. Overall this was a very strategic change in system architecture. Though the near term benefits may seem few, the long term gains from this switchover are huge.

The good
To start with, the uploads are now handled by a box that isn't doing much else, so uploads were flying today. We sampled at various times throughout the day, and even while posting record numbers for a Friday (typically a slow traffic day for podcasting) upload speed from all points in the country we tested were limited by our home internet connections and not the server at all. Media propagation has also dramatically increased. We were seeing < 1minute propagation to the 1st tier and < 3 minutes propagation to the end tier of our network throughout the day. What does that mean? The time from when you upload to the time your file is distributed, ready to be served to thousands of sets of eyes and ears has been greatly reduced. Finally, and most importantly, the startup performance of the downloads was very good today. This was a side effect we didn't anticipate as much, so before we scream success on this one, we want to give it a few days and really understand what is causing this performance increase. What I'm talking about basically, is the time it takes from when you click to when you hear/see something.

So, while I try to make it a point not to claim a win prematurely, or to get myself into trouble promising perfect, error-free service from here on out, I just wanted to report that after a long, difficult week, the team feels good with how things are sitting right now. We have many more changes in front of us, and anticipate demands on our network doubling before the end of the year. Can we keep up? Can we stay ahead of the game? Could you go and give your $ to someone else, and get a pretty good service for your dollar- the answer to all if these questions is "of course." But know this: the $ that you pay to libsyn is going into building something for all of us to use now and in the future. You can etch your name in a brick of this factory for self producers and publishers. Your content is fueling this fire, and your contributions are keeping the lights on. It's a great thing to be a part of, and we would like to thank you for making it happen. We get a lot of mail saying "thank you for your service." It really is _our_ service, together, and we all are the enablers of a great media revolution. Now more than ever the world needs your thoughts, voices, and hearts. So what are you waiting for? Start podcasting!

love
libsyn

To keep up with this growing demand, we're doing some re-structuring of our network. Beginning around 2am est, logins will be disabled, as will new signups. We will do everything we can to execute our plan quickly and carefully, to minimize the wait for those of you wanting to access your account. Media delivery and all of your blog pages and RSS feeds should go unaffected.

We appreciate your understanding, and your choice to be a part of the libsyn family. We will let you know when the maintenance is finished and account access is restored.

Thanks,
- Your friends at libsyn

The system has been restored to approx. 80% full capacity. We anticipate full recovery sometime early this afternoon. Historically, Monday's are intense traffic days, and that extra added attention will slow down the recovery a little bit.

Of course we have to have a record day, today of all days, and it has caused the end-end repair of the distribution network to take a little longer. Currently, somewhere between 1/2 and 3/4 our users should be back up to 100%. Others will still be experiencing some slow downloads, occasional timeouts, and a few 404's from time to time while the network is healing. Specifically, new uploads are taking longer than usual to get out to the whole system. If you use and outside feed to link to your libsyn media, it may help a little if you upload, then hold back on releasing the file into your feed until it has made it around a little bit. (Especially you big shows) A good check is to hit http://media.libsyn.com/media/username/filename in a webbrowser, and if the node your directed to is not media0, then it should be ok. The higher the number the better (>media20 or >media30).

• iTunes duration(Time) support!
• We're still working on the back end receiver script to determine proper durations for the files uploaded automatically. This is totally not that hard a thing to do.. it's coming
• iTunes keyword support
• iTunes explicit tag support
• iTunes subtitle support
• removal of author email as a required field
• This was just one of those misinterpretations of the RSS spec on my part initially. Now, if you prefer to not put an email address out into the wild, so be it. If you leave it blank, and have the Author field populated on your iTunes settings, it should display in ITMS with that Author. This is again our interpretation of the iTunes spec. Please let us know if it's not working out that way for you.

As always, thanks to everyone for their patience and understanding as we battle through these technical issues.

update

Please hang with us while we upgrade to handle you.

So the bad news is, about 1/2 the hits from the last few days haven't been getting counted into your numbers. The very good news is, we are now much better at recovering from the old logs and can back-fill the missing data. It still takes us a little while to reassemble the millions of file requests, so we appreciate it if you pass it along that everything will be restored, and the as of right now, we are counting correctly.

We are really sorry for the past few days, it seems when it rains it pours.. and it sucks.. Thanks for hanging in there with us.

Update (honest truth of what's going on):

The outcome:

If you've been following this blog, you'll know that since the summer we've been taking steps to add redundancy and robustness to our media delivery network.

By robustness we mean the number of downloads the network can sustain at any given time (we currently, by our most conservative estimates, serve around 3.5 million files each week). We have been and contiinue to beef up our infrastructure as the demand for your programs increases.

By redundancy we mean that should any single part of our system fail, your files will continue to be served. A few weeks ago we made a big move towards this redundancy, seperating our media delivery network from the main libsyn site.

Note that links to your media should be updated to point to http://media.libsyn.com/media/your-username/file-name to provide your listeners with the best / fastest performance possible. Old links (http://libsyn.com/media/your-username/file-name) will still work, but will result in a slightly slower connection to your file).

In addition to that seperation of media delivery from the main site, we've also added a failover for the media delivery, so incidents like last night, where a faulty power supply resulted in ~1 hour of slow connections and timeouts, will be much less likely.

Please let us know if you have any questions about the new linking standard, or any of the recent changes to the libsyn network. Thanks as always for your feedback and support

- Your Libsyn Support Team

A faulty power supply caused a disruption in part of our delivery system. We redirected around the problem while the power supply was being replaced. Now things are back in full. We will be implementing a redundancy here as well as better monitoring tools to prevent this in the future.

We apologize for the inconvenience. ]]> Media Delivery Fri, 6 Jan 2006 06:16:00 GMT http://support.libsyn.com/index.php?post_id=45906# support@libsyn.com No http://support.libsyn.com/index.php?post_id=45525# Boy do we have a year in store for you! While visions of sugarplums where dancing in everyone's head, our elves have been hard at work. The result of months of deep development are ~just~ about to come to fruition. While you all have been building your audiences, the technology has been coming together behind the scenes, and we are now proud to present to you, the libsyn + Kiptronic sponsorship program.

We've been working with our friends at Kiptronic since last July to bring libsyn users a program to match up their podcasts with paid sponsorships. We have planned carefully to ensure that the sponsorship program provides a fair and open marketplace. With libsyn, content creators always maintain control of their content. Both publishers and sponsors are able to easily set their own terms and negotiate deals that make sense for them. The program is, of course, 100% optional and won't cost you a dime unless you get paid!

Here's how it works:
The next time you login you will see a link to the libsyn + kiptronic sponsorship program. Click on the link for a description of the system, and you will be taken to your kiptronic profile account which is ready to be activated. It's free to try, and there is no obligation to accept any sponsors. When you and a sponsor agree on the terms of a sponsorship, will become active and the agreed upon audio will be inserted into episodes your program, until the end of the arrangement is reached.

The Good Stuff:
How much you make, how the spots sound, when they are played, and how much you get paid are all up to you: an agreement must be reached by both parties regarding the terms of any sponsorship before it goes into effect. You can do spoken endorsements in your own voice, or the spots can be provided by the sponsor. You can opt for the audio to be inserted into the beginning or the end of each episode, and you can negotiate your own prices for each spot. Because the sponsor's promo is inserted dynamically upon file download, you can have multiple sponsors with rotating spots, and make money from your entire archived catalog of episodes. You can have ads in your own voice, or choose to use spots created by the sponsors. Best of all, You set the price, you set the terms, and you always have final say as to whether a sponsorship goes through.

What lies ahead:
Take a minute and read through the information on the Kiptronic site, and fill out the profile information. What we need from you, if you are interested in finding paying sponsors for your podcast, is to register yourself as available through the Kiptronic system. Once we are able to get interested podcasters into the program, we will set the sponsors loose to start looking for podcasts.

Who are the sponsors?
While we have sponsors ready to participate in this program, we would like to exhibit to them the range and diversity of content that you all are currently creating. This will result in the best deal for you publishers, and will highlight the value of your creation to the sponsors. If capitalizing on your hard work and creativity sounds good to you, go login to https://www.libsyn.com right now and activate your account with the libsyn + kiptronic sponsorship program.

Thanks as always for your loyal support,

Your Friends at Liberated Syndication ]]> announcements Thu, 5 Jan 2006 00:26:00 GMT http://support.libsyn.com/index.php?post_id=45525# support@libsyn.com No http://support.libsyn.com/index.php?post_id=45470# Media Delivery Wed, 4 Jan 2006 20:50:00 GMT http://support.libsyn.com/index.php?post_id=45470# support@libsyn.com No http://support.libsyn.com/index.php?post_id=44070# Stats Thu, 29 Dec 2005 06:31:00 GMT http://support.libsyn.com/index.php?post_id=44070# support@libsyn.com No http://support.libsyn.com/index.php?post_id=41372# Things are functioning much better. We'll let you know once its all clear.]]> Media Delivery Fri, 16 Dec 2005 21:38:00 GMT http://support.libsyn.com/index.php?post_id=41372# support@libsyn.com No http://support.libsyn.com/index.php?post_id=39895# First off, this currently only affects a small % of libsyn users. This in no way affects the stats tracking, displaying, or eventual dissemination of the LAA (libsyn active audience) and libsyn stats system. For those of you who do take our XML logs, run some scripts you have made on them, and do whatever it is you want to do with them, we now have an updated feature. You can select the libsyn XML format, or standard Apache log format.

For new customers, you will have to enable this feature if you want downloadable logs to start building for you. We are creating the logs from the database that updates the flash stats display. We will also very soon have a public (or private) RSS feed for your stats that you will be able to subscribe to. This does not affect that either. For existing customers, if you currently don't use this feature, please do us and everyone else a favor and turn off stat logging. We have, by default, left logging on for ever existing user, but would greatly save disk space and system resources if we only logged for those who absolutely needed it.

For more on this subject, please see the forum post

Cogent is experiencing 2 fiber cuts in the north/southeast-US. The locations are: Between Houston and Tampa (Southeast) Ticket #347277 Between Philidelpha and DC (Northeast) Ticket #347375

As a result we have seen about a 20-30% decrease in traffic. We will keep you updated as things develop.

We appreciate all the patience that you guys have given us today.

http://libsyn.org will get you to the site right now, but unfortunately, that is not going to help your listeners that are having problems.
Update: Files are now being served via the .org domain. This does not help your subscribers, but may be of some assistance.]]> announcements Thu, 3 Nov 2005 18:13:00 GMT http://support.libsyn.com/index.php?post_id=31247# support@libsyn.com No http://support.libsyn.com/index.php?post_id=31207# soundoff post if you want humor.

Thanks for sticking with us folks. We know it's been up a roller-coaster year. If we didn't love hearing your voices screaming our names (mostly) in pleasure and (sometimes) in pain, we wouldn't get out of bed in the morning. Now that we got people pushing video, we can't wait to see where soapy will turn up next!

We got together with the folks at feedburner to fix this problem as soon as we leanred of it. If anyone is still having problems with their feed, please drop an email to support.

Thanks.

Could not ping audio.weblogs.com: Can't call the script because the name "pingSuite" hasn't been defined.

Server Responded: Can't find a sub-table named "weblogs".

In some cases, you'll see that despite the warning, your ping actually has gone through.

The main libsyn media delivery network has been greatly tweaked and enhanced to provide seamless data delivery in the event of hardware or software failure on any of the individual nodes. Currently serving over 2 million media files a week, or approx. 7 terrabytes of data, its been pretty good for the most part. Outside issues dealing with databases, general network outtages, and file uploading aside, the libsyn media delivery network has had a pretty good track record with download speeds usually topping anything we see from other media websites. This mornings issues have caused us to reevaluate the handling of the archive network and upgrade it to the level of robustness we have in place for the main media network.

We are trying desperately to get all caught up on email and support forum issues. If you email us, and don't hear back in the traditional amount of time, please give us a little bit more time to get everything caught up. We installed an email management system _right_ before 3 major events occurred. Nothing like baptism by fire I guess.. keeping tabs on the status of requests is easier, just making it work into our traditional workflows is taking a little time.

No more excuses, we'll just keep chipping away

Dave M. circa 2005 not posing

This apparently has never happened before. It's not just us either, parts of the internet can't get to other parts of the internet. Big companies being Big Jerks. I need coffee. below is better explanations by my more eloquent buddies.

Here's the drafts:

Marty put it like this:

Hey everyone... libsyn is up and running and has been since the migration last Sunday night. What you're experiencing is the result of a corporate catfight between 2 major networks who facilitate large amounts of inter-connectivity across the net.

Dave C said this:

Dear libsyn customers-

Sorry to invade your inbox with more email, however we wanted to let everyone know we are ontop of the current issue going on in Internet land. "This is history in the making" says our ISP. http://ask.slashdot.org/askslashdot/05/10/05/2247207.shtml

We are working to get around this issue which mostly seems to be affecting TimeWarner Roadrunner users specifically by taking the IP addresses of the users being affected, looking at the IP block of their provider, and manually adjusting our BGP tables. If all that went right over your head, no problem. Bottom line: if you are having troubles connecting to libsyn.com, we are not offline. Please email support@libsyn.com with your IP address, and we will work our best to reroute the connection so you can get in.

Also for anyone having trouble logging into the libsyn FTP servers, this could be an issue with the DNS not being fully propagated. If you are having this problem, try using the IP address instead of ftp.libsyn.com to be sure your ftp client is trying to connect to the right server.

We are pretty happy with the way things went today, took longer then we hoped, but the performance increase should be worth it

Please report issues immediately to support@libsyn.com

Ladies, you will all get a chance to thank M personally and buy him a birthday drink at the podcast expo this november.

M. circa 1996

We are doing our best to respond to every email as quickly as we can.

-Dave Chekan

We know many of you are receiving warnings from iTMS that there is a problem with your feed, preventing it from being updated in iTunes.

We have checked several of the feeds in question and found them to be valid. You can test your own feed by visiting FeedValidator.org.

We have contacted the good folks at Apple asking for clarification on what exactly the error it is, so we can determine what steps need to be taken to get your feeds updating in iTunes again

We are getting occasional reports of file downloads being cut-off. We are working to try and figure out a connection between the incidents of file truncation. Please email support@libsyn.com right away if you are experiencing similar issues.

The libsyn directory OPML feed can be found here: http://www.libsyn.com/directory/libsyn.opml

Libsyn users: to be included in this feed login to your account, click the settings tab, and check the box at the bottom of the page "list my podcast in the libsyn directory." Then click "save changes" and you will be ready to rock.

Note: You'll need to re-add any customizations you made to the page. Thanks to Paul Wilch for pointing out and fixing the problem.]]> general Thu, 25 Aug 2005 13:50:00 GMT http://support.libsyn.com/index.php?post_id=17977# support@libsyn.com No http://support.libsyn.com/index.php?post_id=17714# add a new post
make changes to the settings
modify old posts
Thank you.]]> RSS Tue, 23 Aug 2005 20:20:00 GMT http://support.libsyn.com/index.php?post_id=17714# support@libsyn.com No http://support.libsyn.com/index.php?post_id=17031# Just like Sam Beckett We're trying to set right what once went wrong. If you feel your stats numbers are being reported in error, please respond to this forum post asap and we will look into the issue. ]]> Stats Fri, 19 Aug 2005 03:47:00 GMT http://support.libsyn.com/index.php?post_id=17031# support@libsyn.com No http://support.libsyn.com/index.php?post_id=16854# We discovered a bug last week which was preventing QuickCasts from being created, and was at the same time leaving "phantom" files in some users /quickcast folder. This was resulting in inflated disk-usage reporting and for some, unwarranted "You've exceeded your disk quota" e-mails.

This bug has been fixed, and any files lingering in your /quickcast folder have been removed. Please let us know if you have any problems with the libsyn QuickCast feature (or any other libsyn feature for that matter).

Thanks!

This method is known as CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). Yeah, it's a mouthful, but it's also a widely-used method for preventing spam like we've been seeing on your blog pages. Post a comment here on the support blog to let us know your thoughts on this.

iTunes not accepting RSS feed
There was a problem with some RSS feeds that had special characters (like ampersand: &) in the iTunes keywords field. This has been resolved by removing any punctuation (non alpha-numeric characters) from the keywords.

Media Delivery Failing For Some Files
It was discovered that files with names containing multiple periods in a row (i.e. my.file.is....cool.mp3) were not being delivered. This problem has been resolved.

Multiple Language Support for RSS Feeds
It was pointed out to us that some international character sets get displayed as wacky encodings rather than the characters they're supposed to represent. The fix for this is in the works, and we'll make an announcement when it is complete.