Hello the best of the podcasting world-

We would like to give you a 5 minute overview of 'what happened this week'. For the majority of you, the transactions of this past week were fairly painless and transparent. For some, it's been a major headache. We are now on the other side of the mountain of what was one of the largest internal data structure migrations libsyn has seen to date. Trumped possibly only by the moving of our main webserver to a new data center last summer, this week we dramatically changed the internal system of how all your podcasts are stored, archived, retrieved, and served. This change is just step 1 of an ongoing improvement to the way we will be doing our media delivery network.

Aside from just being a long winded excuse for why we have been slow to answer support emails, or why the site was locked out to European users during their 'prime time' a few nights ago, or why earlier in the week we crippled our delivery capability by almost 50%, I would also like to share some near-term improvements that everyone should enjoy as the results of our work this week.

The bad.
So the big upgrade actually started last week at this time. I'm pretty sure we've said it so many times already, but currently, libsyn is still a big hobby for all of us here. No one's getting paid and we all have (somewhat demanding) day jobs... just like most of you podcasters, right? So, we all started last friday night putting into motion the "big switchover." We had been planning, and designing, and coding, and preparing for weeks leading up to this, and given the wall we were approaching in terms of storage, we had to start putting the designs into play. Friday was a very good prep day, set things up for a saturday switch, still leaving all day sunday for bug fixes. After a few unforeseen hangups on saturday, the big switchover was pushed off till sunday. The notion was to do it early sunday, giving us all day to bug fix before the big "sunday night rush" that we typically see from people uploading in time for monday morning (aka Hell Morning).

Needless to say, sunday morning turned into sunday evening and before we knew it, it was 2AM and our backs were against the wall. We HAD to start the switchover. We started the wheels in motion when, something that could best be compared to virtually tripping over the power cord happened- actually, it was more like reaching for the cold water tap and turning the hot. It wasn't a _mistake_ in action, instead, the right action was performed in the wrong place. The result: our highly tuned, prized media delivery network turned 180 degrees and ate itself for 20 seconds. The outcome was almost 1/2 the media mirrored on our system was vaporized. Gone. Poof. With less than 7 hours before the big monday morning rush, strung out on Red Bull and Bawls, 3 days behind on sleep, we gasped.

Within 20 seconds after realizing what had happened, the team composed themselves and took to the painful process of rebuilding the data structures. In order to backfill 10,000+ files (20-50MB each) across 30 servers quickly, we had no choice but to pull the plug on media delivery until we could get caught up. Believe me when I say making that call to cut the network is the absolute last thing we ever want to do. But, by 10AM we were back online and serving at around 75-80% capacity. Monday hit, and hurt, but overall we saw a less than 10% deduction in overall numbers for the day. The big traffic, which pushes well past midnight EST, makes the self-healing process for the network 100x harder. We licked our wounds overnight, but it wasn't enough to get the network back to 100% by Tuesday morning. Tuesday's usually go somewhat slower, between the big Mondays and Wednesdays of the week. But with our tail still between our legs a bit, it was another rough day. The feedburner timeouts where mitigated somewhat by some sharp collaboration between Rick of feedburner and our Marty Mulligan, but overall delivery and uploads and the like were slower than we wanted to see. The idea was to start all over again with the 'big switchover' tuesday night, however given that we wouldn't have enough manpower around wednesday morning (while part of the team was traveling) we opted for a wednesday after-work start. Wednesday went a little better now that we were back to 100%, but we were still getting dangerously close to a point of media storage that we didn't want to be. That clock was ticking and we knew we had to get the switchover started. Ever hear 'a watched pot never boils' ? Well watching an s-copy of 130gigs never finishes. The hours were pushing well past midnight, and once again, we were consumed with a few other gotchas that surfaced in preparation for the big move.

So then there was thursday. We had pretty much run out of time, and the move needed to happen. All the data was now synced between the old servers and the new cluster. Everyone had a good dinner and we sat down around 8PM. We all had our hit lists. The skype call was crystal clear that night. All systems were go for a midnight "plug-swap" .... ..

And then there's this blurr period.

Anyone ever notice that clocks skip everything between 10pm and 1am? It's true. And so, we were looking at 1:00am and still no action. Throughout the week, we had posted a notice in the support blog, but then kept pulling it offline as the dates and times kept changing. We figured anyone who actually follows that was pretty much fed up with us, and since we had calculated everything regarding the switchover, we figured 10-20 minutes of lock-out wouldn't affect anyone too much. We decided that in an hour we would make our move. We posted to support.libsyn.com that at 2AM est. we would lock down uploads, prevent anyone from logging in, and that everything would be fine in less than an hour. Now, here's where software engineering meets the real, cold, lonely world: 2am comes, we break from our virtual skype huddle and begin to execute like a killer attack squad. In 5 minutes we have all the scripts changed over that need to be changed, we're logging in and testing uploads, downloads, sideloads, and crossloads.. everything is great. Skype (y)'s abound. Talks of the (beer) celebration began. Hoopes even began to (flex). Then things got... well... weird. Pages were loading super slow. And one by one the little things didn't really work right anymore. We were now stuck mid-integration and a elusive bug was running the system amok. With all of our heads down in the code, we burrowed our way through the scripts and data, config files and schemas, trying to find the crux of the problem. This is where I (Dave the narrator of this story) sorta shut down.

I was thinking alot about the day job i had to wake up to in 4 hours. Hoopes and Marty were pretty much speaking in binary tongues as I slowly slipped from consciousness. I jolted up a moment later- damn it was 4am! We were locked out for 2 hours now and seemingly hopeless for being online anytime soon! This is where (pardon the expression) the boys showed their "juevos". I really don't even know what happened between 5 and 7:30, but the next thing I knew, we were just about ready to re-enable logins and let the world in to our new system. It was something miraculous... or Hoopes-tastic, but we turned the key, the engine roared, and we peeled out. Ya, the blinkers didn't really work yet, and there was a piece or two still hanging off, but we were able to drive, and today I can say, we have really opened her up and are having a pretty smooth ride. The pieces that we're still putting back together are new user signups, and anything involving the system-level user accounts: changing disk quota, password changes, etc. Quickcast was broken for bit, but is now working again as are all of the blog-page features. Overall this was a very strategic change in system architecture. Though the near term benefits may seem few, the long term gains from this switchover are huge.

The good
To start with, the uploads are now handled by a box that isn't doing much else, so uploads were flying today. We sampled at various times throughout the day, and even while posting record numbers for a Friday (typically a slow traffic day for podcasting) upload speed from all points in the country we tested were limited by our home internet connections and not the server at all. Media propagation has also dramatically increased. We were seeing < 1minute propagation to the 1st tier and < 3 minutes propagation to the end tier of our network throughout the day. What does that mean? The time from when you upload to the time your file is distributed, ready to be served to thousands of sets of eyes and ears has been greatly reduced. Finally, and most importantly, the startup performance of the downloads was very good today. This was a side effect we didn't anticipate as much, so before we scream success on this one, we want to give it a few days and really understand what is causing this performance increase. What I'm talking about basically, is the time it takes from when you click to when you hear/see something.

So, while I try to make it a point not to claim a win prematurely, or to get myself into trouble promising perfect, error-free service from here on out, I just wanted to report that after a long, difficult week, the team feels good with how things are sitting right now. We have many more changes in front of us, and anticipate demands on our network doubling before the end of the year. Can we keep up? Can we stay ahead of the game? Could you go and give your $ to someone else, and get a pretty good service for your dollar- the answer to all if these questions is "of course." But know this: the $ that you pay to libsyn is going into building something for all of us to use now and in the future. You can etch your name in a brick of this factory for self producers and publishers. Your content is fueling this fire, and your contributions are keeping the lights on. It's a great thing to be a part of, and we would like to thank you for making it happen. We get a lot of mail saying "thank you for your service." It really is _our_ service, together, and we all are the enablers of a great media revolution. Now more than ever the world needs your thoughts, voices, and hearts. So what are you waiting for? Start podcasting!

love
libsyn

.

Regretfully, we are still having some account-management issues with our new system. Currently affected are new account signups, account password changes, and account quota updates. We hope to have all these issues worked out today. Thank you for your patience and understanding.

Hello good synners. The first quarter of 2006 has brought explosive growth to the libsyn network. Yesterday we delivered 1,042,739 downloads and as of this posting we are hosting more than 12,000 files on our main media network and almost 60,000 files in total.

To keep up with this growing demand, we're doing some re-structuring of our network. Beginning around 2am est, logins will be disabled, as will new signups. We will do everything we can to execute our plan quickly and carefully, to minimize the wait for those of you wanting to access your account. Media delivery and all of your blog pages and RSS feeds should go unaffected.

We appreciate your understanding, and your choice to be a part of the libsyn family. We will let you know when the maintenance is finished and account access is restored.

Thanks,
- Your friends at libsyn

We are still experiencing 404's on newly posted files. File propagation has not yet fully recovered from this weekend's issues. We are recommending that folks upload media, but hold out publishing to the rss feeds and blogs until the file has been confirmed to have reached the delivery network. You can do this in the "publish" tab by selecting "save draft", and then confirm that the file has propagated by checking http://media.libsyn.com/media/username/filename. We are sorry for the inconvenience and will keep you posted.

Today we implemented measures that should help avoid the recent time-outs being experienced by libsyn users who burn their feeds with FeedBurner. Thanks to the FeedBurner crew for being pro-active about working with us to resolve these issues. If you are a libsyn / feedburner user who was affected by this problem in the past, please let us know how things are working for you now.

We regretfully had some issues regarding our system upgrades this evening and are working to recover. As a result, data delivery will be slowed while we restore connectivity to a large part of our network. We will keep you posted. We apologize for the inconvenience.

---

Update: (10AM EST)

The system has been restored to approx. 80% full capacity. We anticipate full recovery sometime early this afternoon. Historically, Monday's are intense traffic days, and that extra added attention will slow down the recovery a little bit.

Thank you for hanging in there with us.

---

Update: (8:30PM EST)

Of course we have to have a record day, today of all days, and it has caused the end-end repair of the distribution network to take a little longer. Currently, somewhere between 1/2 and 3/4 our users should be back up to 100%. Others will still be experiencing some slow downloads, occasional timeouts, and a few 404's from time to time while the network is healing. Specifically, new uploads are taking longer than usual to get out to the whole system. If you use and outside feed to link to your libsyn media, it may help a little if you upload, then hold back on releasing the file into your feed until it has made it around a little bit. (Especially you big shows) A good check is to hit http://media.libsyn.com/media/username/filename in a webbrowser, and if the node your directed to is not media0, then it should be ok. The higher the number the better (>media20 or >media30).

As part of our ongoing house-cleaning effort to go back and rework some of the parts of our system which we didn't get quite right the first time around, we've revamped the publishing page to be a little closer to what we had in mind when we set out to make a system that was both easy to use, and extensible to the advanced user. We're not through yet, but we've refined it enough for one day and are going to let everyone start using it right away. We've try to hit on the major ones people have been after us for a while now. If you don't see your need on this list, please be a little patient, but also don't hesitate to bug us again about it.

The new features you'll find are:

• iTunes duration(Time) support!
• We're still working on the back end receiver script to determine proper durations for the files uploaded automatically. This is totally not that hard a thing to do.. it's coming
• iTunes keyword support
• iTunes explicit tag support
• iTunes subtitle support
• removal of author email as a required field
• This was just one of those misinterpretations of the RSS spec on my part initially. Now, if you prefer to not put an email address out into the wild, so be it. If you leave it blank, and have the Author field populated on your iTunes settings, it should display in ITMS with that Author. This is again our interpretation of the iTunes spec. Please let us know if it's not working out that way for you.

That's it for now, stay tuned for WYSIWYG text blog posting and better image and category management.

Libsyn friends: We're doing some relatively heavy-duty upgrades this weekend to continue to accommodate the explosive growth we're experiencing. So, while we're going to keep interruptions to an absolute minimum, weird things inevitably pop up. Please bring them to our attention if you catch something extra-wrong. Media delivery should continue uninterrupted. This upgrade is more toward users home directories, blog pages, rss serving, etc. So, anyway, rambling aside, keep your head on a swivel. Yours, - hoopes and the rest of the squad

Hey guys, Around 5pm eastern there was a glitch in our caching system for rss feeds. The feeds were temporarily unavailable. We did not see a significant drop in traffic, and things are again functioning properly. Thanks

Two hours worth of file transfers were replayed into the stats db, and some attendant display errors were corrected. At this time, stats should be fully back to normal. If you see anything contraindicators, please let us know. Also: in the past, when we have had to recover from logs, "phantom" files have occasionally crept into the stats displays of some users. I have added additional checks to attempt to prevent this, but if you see anything of the sort I'll clean it up and make the recovery process smarter for next time. As always, just drop a note to our support email box with a note about anything that doesn't look right.

We had a slight problem with our stats database around 12:23:06 this evening. We have not been tracking downloads real-time for the past hour. All data is being logged however, and we will be processing the raw logs as soon as possible.

The real-time tracking will be turned back on shortly and will start recording again. Sorry for the misqueue. We are updating our db tables to prevent this particular issue from happening again.

You may notice a new checkbox in your media tab today. An option an overwhelming number of users have asked for, is finally delivered: files that won't get archived.

Sticky files will stay in your current media, and therefore on our fast, primary media delivery network, for as long as you are a libsyn user. Please note, that any space a sticky file consumes will not be available for other files to use. (ie, if you have a 100MB account, and make 20MB file sticky, you will only ever at most have 80MB of space for future uploads)

Look for more small modifications and features to spill out over the next few days and weeks.

Between 11pm and 5am last night we had an issue with a misconfigured mail service. This was an administrative error that would have caused mail sent to us (at any @libsyn.com address) to be bounced. If you were seeking support during this time, or trying to contact us via e-mail for any reason, please re-send and accept our apologies for the mistake. Thanks

Around 3:00 EST this afternoon we started seeing a surge in traffic. We are moving traffic around and trying to do what we can to up our capacity. We will hopefully have things returned to normal asap.