8:30pm EST Learned from Network Solutions that the reason our DNS changes are taking so long to update is because they are doing maintenance at this time. Data center is working to restore the primary and backup systems. Back up soon.

All systems have been restored and are now fully operational.

To you straight from the mouths of our datacenter team:

"I will try to explain it as easy as possible. at 18:45EST 3-27-2008; The PFC3B line card in the Primary Supervisor of Core1 went into a LOOP; This caused an issue that did not allow Core1 to process Layer3 traffic across the MSFC3. Layer2 however was not affected in this matter which allowed Core2 to continue running normally and pass Layer2 traffic Across the Core1 Layer2 Back-bone. HSRP which is used as a counter-measure; still showed Core1 as functional even though it was experiencing a Loop.

at 19:00EST; a CISCO SMART-NET Tech has been dispatched with additional hardware (SUP, MSFC3, PFC3) in case a defective board was found.

At 19:35EST, Core1 was taken off-line in preparation for Cisco's Arrival.

Upon arrival at about 20:45EST; a debug was generated and CISCO has re-certified the Module. We are waiting on Final Explanation from CISCO as to what caused the PFC3 daughter board to act in the way that it has.

We have taken numerous Physical counter-measures to make sure that this will not happen again. Some of these counter-measures include modification to the HSRP algorythm as well as physical scripting.

Lastly; This incident only affected clients that the HSRP was Active on the Device. We have placed more sophisticated HSRP counter-measures."

Laymen's explanation: there was a hardware failure. Our side of the blame line is not having enough replication of our DNS records in another datacenter. That is being fixed today.

Sorry again for the downtime.

The stats system has been playing catchup for the last day. All data is being collected, it's in the queue to be processed, and you should be seeing timely updates once again when everything is caught up. In the meantime, it may appear your numbers are not increasing and new files may not yet appear in the stats display

There was an issue with one of our data providers which was causing some users, and especially archived files, to be slow to start and time out resulting in no downloads. The fail-over mechanism that should have recovered the issue has been fixed as well as the problem with the provider itself.

The archiving issue which has been causing files to not move to the archives has been resolved.

Libsyn users who are using Kiptronic were experiencing issues with insertions not working. That has been resolved.

We would like to give you a 5 minute overview of 'what happened this week'. For the majority of you, the transactions of this past week were fairly painless and transparent. For some, it's been a major headache. We are now on the other side of the mountain of what was one of the largest internal data structure migrations libsyn has seen to date. Trumped possibly only by the moving of our main webserver to a new data center last summer, this week we dramatically changed the internal system of how all your podcasts are stored, archived, retrieved, and served. This change is just step 1 of an ongoing improvement to the way we will be doing our media delivery network.

Aside from just being a long winded excuse for why we have been slow to answer support emails, or why the site was locked out to European users during their 'prime time' a few nights ago, or why earlier in the week we crippled our delivery capability by almost 50%, I would also like to share some near-term improvements that everyone should enjoy as the results of our work this week.

The bad.
So the big upgrade actually started last week at this time. I'm pretty sure we've said it so many times already, but currently, libsyn is still a big hobby for all of us here. No one's getting paid and we all have (somewhat demanding) day jobs... just like most of you podcasters, right? So, we all started last friday night putting into motion the "big switchover." We had been planning, and designing, and coding, and preparing for weeks leading up to this, and given the wall we were approaching in terms of storage, we had to start putting the designs into play. Friday was a very good prep day, set things up for a saturday switch, still leaving all day sunday for bug fixes. After a few unforeseen hangups on saturday, the big switchover was pushed off till sunday. The notion was to do it early sunday, giving us all day to bug fix before the big "sunday night rush" that we typically see from people uploading in time for monday morning (aka Hell Morning).

Needless to say, sunday morning turned into sunday evening and before we knew it, it was 2AM and our backs were against the wall. We HAD to start the switchover. We started the wheels in motion when, something that could best be compared to virtually tripping over the power cord happened- actually, it was more like reaching for the cold water tap and turning the hot. It wasn't a _mistake_ in action, instead, the right action was performed in the wrong place. The result: our highly tuned, prized media delivery network turned 180 degrees and ate itself for 20 seconds. The outcome was almost 1/2 the media mirrored on our system was vaporized. Gone. Poof. With less than 7 hours before the big monday morning rush, strung out on Red Bull and Bawls, 3 days behind on sleep, we gasped.

Within 20 seconds after realizing what had happened, the team composed themselves and took to the painful process of rebuilding the data structures. In order to backfill 10,000+ files (20-50MB each) across 30 servers quickly, we had no choice but to pull the plug on media delivery until we could get caught up. Believe me when I say making that call to cut the network is the absolute last thing we ever want to do. But, by 10AM we were back online and serving at around 75-80% capacity. Monday hit, and hurt, but overall we saw a less than 10% deduction in overall numbers for the day. The big traffic, which pushes well past midnight EST, makes the self-healing process for the network 100x harder. We licked our wounds overnight, but it wasn't enough to get the network back to 100% by Tuesday morning. Tuesday's usually go somewhat slower, between the big Mondays and Wednesdays of the week. But with our tail still between our legs a bit, it was another rough day. The feedburner timeouts where mitigated somewhat by some sharp collaboration between Rick of feedburner and our Marty Mulligan, but overall delivery and uploads and the like were slower than we wanted to see. The idea was to start all over again with the 'big switchover' tuesday night, however given that we wouldn't have enough manpower around wednesday morning (while part of the team was traveling) we opted for a wednesday after-work start. Wednesday went a little better now that we were back to 100%, but we were still getting dangerously close to a point of media storage that we didn't want to be. That clock was ticking and we knew we had to get the switchover started. Ever hear 'a watched pot never boils' ? Well watching an s-copy of 130gigs never finishes. The hours were pushing well past midnight, and once again, we were consumed with a few other gotchas that surfaced in preparation for the big move.

So then there was thursday. We had pretty much run out of time, and the move needed to happen. All the data was now synced between the old servers and the new cluster. Everyone had a good dinner and we sat down around 8PM. We all had our hit lists. The skype call was crystal clear that night. All systems were go for a midnight "plug-swap" .... ..

And then there's this blurr period.

Anyone ever notice that clocks skip everything between 10pm and 1am? It's true. And so, we were looking at 1:00am and still no action. Throughout the week, we had posted a notice in the support blog, but then kept pulling it offline as the dates and times kept changing. We figured anyone who actually follows that was pretty much fed up with us, and since we had calculated everything regarding the switchover, we figured 10-20 minutes of lock-out wouldn't affect anyone too much. We decided that in an hour we would make our move. We posted to support.libsyn.com that at 2AM est. we would lock down uploads, prevent anyone from logging in, and that everything would be fine in less than an hour. Now, here's where software engineering meets the real, cold, lonely world: 2am comes, we break from our virtual skype huddle and begin to execute like a killer attack squad. In 5 minutes we have all the scripts changed over that need to be changed, we're logging in and testing uploads, downloads, sideloads, and crossloads.. everything is great. Skype (y)'s abound. Talks of the (beer) celebration began. Hoopes even began to (flex). Then things got... well... weird. Pages were loading super slow. And one by one the little things didn't really work right anymore. We were now stuck mid-integration and a elusive bug was running the system amok. With all of our heads down in the code, we burrowed our way through the scripts and data, config files and schemas, trying to find the crux of the problem. This is where I (Dave the narrator of this story) sorta shut down.

I was thinking alot about the day job i had to wake up to in 4 hours. Hoopes and Marty were pretty much speaking in binary tongues as I slowly slipped from consciousness. I jolted up a moment later- damn it was 4am! We were locked out for 2 hours now and seemingly hopeless for being online anytime soon! This is where (pardon the expression) the boys showed their "juevos". I really don't even know what happened between 5 and 7:30, but the next thing I knew, we were just about ready to re-enable logins and let the world in to our new system. It was something miraculous... or Hoopes-tastic, but we turned the key, the engine roared, and we peeled out. Ya, the blinkers didn't really work yet, and there was a piece or two still hanging off, but we were able to drive, and today I can say, we have really opened her up and are having a pretty smooth ride. The pieces that we're still putting back together are new user signups, and anything involving the system-level user accounts: changing disk quota, password changes, etc. Quickcast was broken for bit, but is now working again as are all of the blog-page features. Overall this was a very strategic change in system architecture. Though the near term benefits may seem few, the long term gains from this switchover are huge.

The good
To start with, the uploads are now handled by a box that isn't doing much else, so uploads were flying today. We sampled at various times throughout the day, and even while posting record numbers for a Friday (typically a slow traffic day for podcasting) upload speed from all points in the country we tested were limited by our home internet connections and not the server at all. Media propagation has also dramatically increased. We were seeing < 1minute propagation to the 1st tier and < 3 minutes propagation to the end tier of our network throughout the day. What does that mean? The time from when you upload to the time your file is distributed, ready to be served to thousands of sets of eyes and ears has been greatly reduced. Finally, and most importantly, the startup performance of the downloads was very good today. This was a side effect we didn't anticipate as much, so before we scream success on this one, we want to give it a few days and really understand what is causing this performance increase. What I'm talking about basically, is the time it takes from when you click to when you hear/see something.

So, while I try to make it a point not to claim a win prematurely, or to get myself into trouble promising perfect, error-free service from here on out, I just wanted to report that after a long, difficult week, the team feels good with how things are sitting right now. We have many more changes in front of us, and anticipate demands on our network doubling before the end of the year. Can we keep up? Can we stay ahead of the game? Could you go and give your $ to someone else, and get a pretty good service for your dollar- the answer to all if these questions is "of course." But know this: the $ that you pay to libsyn is going into building something for all of us to use now and in the future. You can etch your name in a brick of this factory for self producers and publishers. Your content is fueling this fire, and your contributions are keeping the lights on. It's a great thing to be a part of, and we would like to thank you for making it happen. We get a lot of mail saying "thank you for your service." It really is _our_ service, together, and we all are the enablers of a great media revolution. Now more than ever the world needs your thoughts, voices, and hearts. So what are you waiting for? Start podcasting!

love
libsyn

Periodically, you may see one of the following messages upon publishing a post:

Could not ping audio.weblogs.com: Can't call the script because the name "pingSuite" hasn't been defined.

Server Responded: Can't find a sub-table named "weblogs".

In some cases, you'll see that despite the warning, your ping actually has gone through.

-Dave Chekan

Update from apple:
We just implemented an email notification system. If we have a problem with a feed AND the feed contains a contact email, we notify that person that there's a problem.
The message below goes out if the iTunes system can't reach the feed. Usually it's a temporary problem. But we send it out so the podcaster can check and see if everything is working ok. If it is, they can just ignore the message. If they keep getting it --- then there's some deeper problem that needs to be investigate.

We know many of you are receiving warnings from iTMS that there is a problem with your feed, preventing it from being updated in iTunes.

We have checked several of the feeds in question and found them to be valid. You can test your own feed by visiting FeedValidator.org.

We have contacted the good folks at Apple asking for clarification on what exactly the error it is, so we can determine what steps need to be taken to get your feeds updating in iTunes again

There were some browser compatibility issues with the latest template, "Ku-Blue." The template has been repaired. To apply these changes to your page, login to your libsyn account, click the settings tab, and then click templates. Then scroll down to the Ku-Blue template and click the link "use this template."

Note: You'll need to re-add any customizations you made to the page. Thanks to Paul Wilch for pointing out and fixing the problem.

We discovered a bug last week which was preventing QuickCasts from being created, and was at the same time leaving "phantom" files in some users /quickcast folder. This was resulting in inflated disk-usage reporting and for some, unwarranted "You've exceeded your disk quota" e-mails.

This bug has been fixed, and any files lingering in your /quickcast folder have been removed. Please let us know if you have any problems with the libsyn QuickCast feature (or any other libsyn feature for that matter).

Thanks!

Thanks to everyone for helping us track down and eliminate the bugs in this system. Here are some of the big ones we fixed this week:

iTunes not accepting RSS feed
There was a problem with some RSS feeds that had special characters (like ampersand: &) in the iTunes keywords field. This has been resolved by removing any punctuation (non alpha-numeric characters) from the keywords.

Media Delivery Failing For Some Files
It was discovered that files with names containing multiple periods in a row (i.e. my.file.is....cool.mp3) were not being delivered. This problem has been resolved.

Multiple Language Support for RSS Feeds
It was pointed out to us that some international character sets get displayed as wacky encodings rather than the characters they're supposed to represent. The fix for this is in the works, and we'll make an announcement when it is complete.