We will release more updates as we update the front end display as well

Update: We are still investigating the true reason for last night- and very early this mornings- performance issues. I am fairly convinced the IP's that we blocked coincidentally helped relieve the issue, which was caused in fact by something else. We may have jumped the gun a little bit reporting that it was a malicious attack. Thanks for everyone's support as we keep wrestling with the logs and graphs and reports to find the real problem and make sure it's fixed.

Update: We've made some major changes to the network that we feel will balance things out much better. The distribution and balancing technique we were using needed some updating. We're going to need to see tomorrow at peek times to see if we really have things tweaked perfectly.

As luck would have it, while doing some other general maintanence over the weekend, where we were adding some redundency to other key parts of the system, the back kinda broke on a segment of the network. Most new users (who've signed up in the last month or so) were affected, so we're not really starting out with our best foot forward with a lot of you. I hope you can understand that these kinds of things happen to everyone, and hope you stick with us- as you can be assured we're going to stick with you- through thick and thin. We're going to keep killing ourselves (and hopefully not each other) to ensure the best, most reliable, most stable system for all you content creators.
-Dave Chekan

For now here's how it works. Login, under the Settings / Advanced section you'll see a check box to enable comment moderation. Once you turn that on, all comments left on your blog will be hidden until you go in and enable them. To enable the hidden comments, under the publish / previous posts screen, if you click on the comments viewer, you will see a button to enable the comments. When you have new comments, you will see a link like this when you first login (see screenshot).

The stats processing is currently going through a massive overhaul. Part of the upgrades will include some new hardware. We understand the importance of timely, accurate statistics, and second to reliable media delivery, it's the thing you all really care most about. We're on it. it's going to take a little time, thank you for understanding.

Fortunately (and unfortunately) the issue was the same as last time. The error we made this time, though, was that the fix was not properly incorporated into our build script for new media nodes. Just as we though we were helping the network by bringing on 30% more hardware, it turns around and bites us for not installing all the proper fixes.

The patch that had been applied to the previous nodes has now been applied. I don't know how many more times you're going to accept our apologizes on this (the most critical or critical) error. We now need to make sure the stickies around our monitors have prominence that say "Add to the new build script!!!" on every new fix and tweak.

Sorry to start off your day with this. Our hope is that we caught the issue in time as to not have affected too many downloads.

UPDATE:

The move has been delayed some as our techs at the datacenter double and triple check wiring, configurations, and routing for all of our equipment.  We will make a new post here with at least 24 hours notice of when the actual upgrade is going to take place.

--

Our primary datacenter is upgrading and expanding (partially to meet our growing demands). We have been working through a series of updates and are a reaching a point where there will been some physical hardware shifted around this coming week. Through careful planning and our redundancy in multiple datacenters, we intend to keep mission critical functions, like serving media to your audience, up and functioning at all times. There will however be a short period on Tuesday, May 16th where you will be unable to upload new media, login to post shows or check stats for up to an hour.

We will be dialing in the exact times as this stage of the process grows closer, and will post an update with  the latest news. This estimate is the worst-case scenario, and our technical team will be in constant  communication with the folks "on the ground," ready to react to any issues that should arise during the transition.

As always, we want you all to have complete clarity and understanding of what we are doing and that we are focused on ensuring the least inconvenience and downtime possible. Thanks

• bold
• italics
• font colors

1. Numbered
2. lists

In other news. we have been a little tardy the past few days in responding to emails and to forum posts. We are going to try to make it a general rule of thumb to get through emails before we hit the forums. The time to do both, AND make improvements to the system is so sparse, so in order we'll answer: emails, then forum, then person instant messages. The bigger problem, however, is that our system has bugs and we have poor documentation. The first thing is our constant "to-do" list and we will make improvements with over time. Now that it's spring- we're kicking our house cleaning into high gear. As for the other thing, documentation, here's where we're calling on all of you to help us out a little (if you have some time).

Now that it's the 21st century, we have finally put some work into our documentation wiki. http://docs.libsyn.com is now updated a little more (working on it even as we type this email) and we openly welcome all your libsyn zealots to come help us fill in the cracks. Now, we know you all would do it just for your unconditional love of podcasting, but we do have plans to reward the big contributors (that goes for the community forum as well). Everything from sticker, pins, t-shirts, free accounts, marty's car.. will be given as small tokens of our appreciation to those who have continued to give back to the community. Having a good set of documentation will greatly cut down on the number of support emails we currently get. We'll try to emerge a practical compliment between the forum and the wiki so new users (and old) can self diagnose and help themselves understand the common issues of libsyn. Then a quick stop at the http://support.libsyn.com blog for system alerts and issues, and then last resort an email.

We hope everyone is enjoying the new hardware that has been put in place over the last couple weeks. Good news coming from the Kiptronic camp as well in terms of new sponsors coming into the marketplace very soon as well as some awesome new features we'll be excited to share with everyone soon. I will take a quick minute to talk about one of those, and that is libsyn stats. As mentioned on our stats display for months now, RSS stat feeds are almost ready to roll. What is also going to happen, is those users who choose to sign up for the kiptronic marketplace will automatically get their libsyn stats fed into the kiptronic system and displayed to potential sponsors. IF YOU ARE NOT USING KIPTRONIC, YOUR STATS WILL NOT BE SHARED WITH THEM. That's very important to note that we will not disclose your information to anyone who you have not given express consent to. The way you authorize the sharing of your audience data with other libsyn partners (soon to come) is through the advanced settings tab. We are also considering releasing podcast rankings to the public for those who wish to opt in. We welcome your feedback on these topics, either via private email or in the public community forum.

As always, thank you all for letting us syndicate your content. It is an honor. Thank you for hangin with libsyn, through thick and thin, and helping to build the greatest network of podcasts in the 'verse. (Firefly fans, let me hear ya)

Leaving you with a little support podcast that was sent to us recently. Wouldn't it be funny if everyone sent us audio support requests?

let me answer that.. . no

Because archiving was off for a number of days, we have had to turn it back on again in phases. We've made multiple sweeps of your home directories, archiving files that are 30 days old, and the plan is to make another complete sweep again tonight, re-enable quotas, and then put archiving back on an automated schedule.

At least one person has pointed out that a number of files that were archived on Friday were resulting in 404's. This was because of a short archive run that went astray. We believed we had restored all of the missing files, but if we missed one user, it is possible there are more. If you notice 404's on any of your archived files, please let us know right away.

Finally, while we are hustling behind the scenes here at libsyn hq, Dave M. has been on the road meeting with various friends in the industry. In this picture, Dave is discussing some funding options with our new financial analyst... stay tuned for more.

The issues with the archives not processing, and new uploads 404-ing are temporary and should be resolved shortly

Again, your account will look like it's over quota and disabled from uploading, however you ARE still able to upload while we fix the archiving situation

Thank you and sorry for taking an extra couple days to resolve this problem

This is just a workaround while we get more optimizations and new hardware online.

To be clear, requests from all browsers (IE, Firefox, Safari, etc) and iTunes will continue to be honored while we are working through this issue

Following last weekends system maintenance, we had disabled the change password and reset password functionality on libsyn.com. As of this posting, the change and reset password functions have been re-enabled, so if you have any trouble loggin into the website or via FTP, please reset your password and try again.

We would like to give you a 5 minute overview of 'what happened this week'. For the majority of you, the transactions of this past week were fairly painless and transparent. For some, it's been a major headache. We are now on the other side of the mountain of what was one of the largest internal data structure migrations libsyn has seen to date. Trumped possibly only by the moving of our main webserver to a new data center last summer, this week we dramatically changed the internal system of how all your podcasts are stored, archived, retrieved, and served. This change is just step 1 of an ongoing improvement to the way we will be doing our media delivery network.

Aside from just being a long winded excuse for why we have been slow to answer support emails, or why the site was locked out to European users during their 'prime time' a few nights ago, or why earlier in the week we crippled our delivery capability by almost 50%, I would also like to share some near-term improvements that everyone should enjoy as the results of our work this week.

The bad.
So the big upgrade actually started last week at this time. I'm pretty sure we've said it so many times already, but currently, libsyn is still a big hobby for all of us here. No one's getting paid and we all have (somewhat demanding) day jobs... just like most of you podcasters, right? So, we all started last friday night putting into motion the "big switchover." We had been planning, and designing, and coding, and preparing for weeks leading up to this, and given the wall we were approaching in terms of storage, we had to start putting the designs into play. Friday was a very good prep day, set things up for a saturday switch, still leaving all day sunday for bug fixes. After a few unforeseen hangups on saturday, the big switchover was pushed off till sunday. The notion was to do it early sunday, giving us all day to bug fix before the big "sunday night rush" that we typically see from people uploading in time for monday morning (aka Hell Morning).

Needless to say, sunday morning turned into sunday evening and before we knew it, it was 2AM and our backs were against the wall. We HAD to start the switchover. We started the wheels in motion when, something that could best be compared to virtually tripping over the power cord happened- actually, it was more like reaching for the cold water tap and turning the hot. It wasn't a _mistake_ in action, instead, the right action was performed in the wrong place. The result: our highly tuned, prized media delivery network turned 180 degrees and ate itself for 20 seconds. The outcome was almost 1/2 the media mirrored on our system was vaporized. Gone. Poof. With less than 7 hours before the big monday morning rush, strung out on Red Bull and Bawls, 3 days behind on sleep, we gasped.

Within 20 seconds after realizing what had happened, the team composed themselves and took to the painful process of rebuilding the data structures. In order to backfill 10,000+ files (20-50MB each) across 30 servers quickly, we had no choice but to pull the plug on media delivery until we could get caught up. Believe me when I say making that call to cut the network is the absolute last thing we ever want to do. But, by 10AM we were back online and serving at around 75-80% capacity. Monday hit, and hurt, but overall we saw a less than 10% deduction in overall numbers for the day. The big traffic, which pushes well past midnight EST, makes the self-healing process for the network 100x harder. We licked our wounds overnight, but it wasn't enough to get the network back to 100% by Tuesday morning. Tuesday's usually go somewhat slower, between the big Mondays and Wednesdays of the week. But with our tail still between our legs a bit, it was another rough day. The feedburner timeouts where mitigated somewhat by some sharp collaboration between Rick of feedburner and our Marty Mulligan, but overall delivery and uploads and the like were slower than we wanted to see. The idea was to start all over again with the 'big switchover' tuesday night, however given that we wouldn't have enough manpower around wednesday morning (while part of the team was traveling) we opted for a wednesday after-work start. Wednesday went a little better now that we were back to 100%, but we were still getting dangerously close to a point of media storage that we didn't want to be. That clock was ticking and we knew we had to get the switchover started. Ever hear 'a watched pot never boils' ? Well watching an s-copy of 130gigs never finishes. The hours were pushing well past midnight, and once again, we were consumed with a few other gotchas that surfaced in preparation for the big move.

So then there was thursday. We had pretty much run out of time, and the move needed to happen. All the data was now synced between the old servers and the new cluster. Everyone had a good dinner and we sat down around 8PM. We all had our hit lists. The skype call was crystal clear that night. All systems were go for a midnight "plug-swap" .... ..

And then there's this blurr period.

Anyone ever notice that clocks skip everything between 10pm and 1am? It's true. And so, we were looking at 1:00am and still no action. Throughout the week, we had posted a notice in the support blog, but then kept pulling it offline as the dates and times kept changing. We figured anyone who actually follows that was pretty much fed up with us, and since we had calculated everything regarding the switchover, we figured 10-20 minutes of lock-out wouldn't affect anyone too much. We decided that in an hour we would make our move. We posted to support.libsyn.com that at 2AM est. we would lock down uploads, prevent anyone from logging in, and that everything would be fine in less than an hour. Now, here's where software engineering meets the real, cold, lonely world: 2am comes, we break from our virtual skype huddle and begin to execute like a killer attack squad. In 5 minutes we have all the scripts changed over that need to be changed, we're logging in and testing uploads, downloads, sideloads, and crossloads.. everything is great. Skype (y)'s abound. Talks of the (beer) celebration began. Hoopes even began to (flex). Then things got... well... weird. Pages were loading super slow. And one by one the little things didn't really work right anymore. We were now stuck mid-integration and a elusive bug was running the system amok. With all of our heads down in the code, we burrowed our way through the scripts and data, config files and schemas, trying to find the crux of the problem. This is where I (Dave the narrator of this story) sorta shut down.

I was thinking alot about the day job i had to wake up to in 4 hours. Hoopes and Marty were pretty much speaking in binary tongues as I slowly slipped from consciousness. I jolted up a moment later- damn it was 4am! We were locked out for 2 hours now and seemingly hopeless for being online anytime soon! This is where (pardon the expression) the boys showed their "juevos". I really don't even know what happened between 5 and 7:30, but the next thing I knew, we were just about ready to re-enable logins and let the world in to our new system. It was something miraculous... or Hoopes-tastic, but we turned the key, the engine roared, and we peeled out. Ya, the blinkers didn't really work yet, and there was a piece or two still hanging off, but we were able to drive, and today I can say, we have really opened her up and are having a pretty smooth ride. The pieces that we're still putting back together are new user signups, and anything involving the system-level user accounts: changing disk quota, password changes, etc. Quickcast was broken for bit, but is now working again as are all of the blog-page features. Overall this was a very strategic change in system architecture. Though the near term benefits may seem few, the long term gains from this switchover are huge.

The good
To start with, the uploads are now handled by a box that isn't doing much else, so uploads were flying today. We sampled at various times throughout the day, and even while posting record numbers for a Friday (typically a slow traffic day for podcasting) upload speed from all points in the country we tested were limited by our home internet connections and not the server at all. Media propagation has also dramatically increased. We were seeing < 1minute propagation to the 1st tier and < 3 minutes propagation to the end tier of our network throughout the day. What does that mean? The time from when you upload to the time your file is distributed, ready to be served to thousands of sets of eyes and ears has been greatly reduced. Finally, and most importantly, the startup performance of the downloads was very good today. This was a side effect we didn't anticipate as much, so before we scream success on this one, we want to give it a few days and really understand what is causing this performance increase. What I'm talking about basically, is the time it takes from when you click to when you hear/see something.

So, while I try to make it a point not to claim a win prematurely, or to get myself into trouble promising perfect, error-free service from here on out, I just wanted to report that after a long, difficult week, the team feels good with how things are sitting right now. We have many more changes in front of us, and anticipate demands on our network doubling before the end of the year. Can we keep up? Can we stay ahead of the game? Could you go and give your $ to someone else, and get a pretty good service for your dollar- the answer to all if these questions is "of course." But know this: the $ that you pay to libsyn is going into building something for all of us to use now and in the future. You can etch your name in a brick of this factory for self producers and publishers. Your content is fueling this fire, and your contributions are keeping the lights on. It's a great thing to be a part of, and we would like to thank you for making it happen. We get a lot of mail saying "thank you for your service." It really is _our_ service, together, and we all are the enablers of a great media revolution. Now more than ever the world needs your thoughts, voices, and hearts. So what are you waiting for? Start podcasting!

love
libsyn

Hello good synners. The first quarter of 2006 has brought explosive growth to the libsyn network. Yesterday we delivered 1,042,739 downloads and as of this posting we are hosting more than 12,000 files on our main media network and almost 60,000 files in total.

To keep up with this growing demand, we're doing some re-structuring of our network. Beginning around 2am est, logins will be disabled, as will new signups. We will do everything we can to execute our plan quickly and carefully, to minimize the wait for those of you wanting to access your account. Media delivery and all of your blog pages and RSS feeds should go unaffected.

We appreciate your understanding, and your choice to be a part of the libsyn family. We will let you know when the maintenance is finished and account access is restored.

Thanks,
- Your friends at libsyn

The system has been restored to approx. 80% full capacity. We anticipate full recovery sometime early this afternoon. Historically, Monday's are intense traffic days, and that extra added attention will slow down the recovery a little bit.

Of course we have to have a record day, today of all days, and it has caused the end-end repair of the distribution network to take a little longer. Currently, somewhere between 1/2 and 3/4 our users should be back up to 100%. Others will still be experiencing some slow downloads, occasional timeouts, and a few 404's from time to time while the network is healing. Specifically, new uploads are taking longer than usual to get out to the whole system. If you use and outside feed to link to your libsyn media, it may help a little if you upload, then hold back on releasing the file into your feed until it has made it around a little bit. (Especially you big shows) A good check is to hit http://media.libsyn.com/media/username/filename in a webbrowser, and if the node your directed to is not media0, then it should be ok. The higher the number the better (>media20 or >media30).

The new features you'll find are:

• iTunes duration(Time) support!
• We're still working on the back end receiver script to determine proper durations for the files uploaded automatically. This is totally not that hard a thing to do.. it's coming
• iTunes keyword support
• iTunes explicit tag support
• iTunes subtitle support
• removal of author email as a required field
• This was just one of those misinterpretations of the RSS spec on my part initially. Now, if you prefer to not put an email address out into the wild, so be it. If you leave it blank, and have the Author field populated on your iTunes settings, it should display in ITMS with that Author. This is again our interpretation of the iTunes spec. Please let us know if it's not working out that way for you.

The real-time tracking will be turned back on shortly and will start recording again. Sorry for the misqueue. We are updating our db tables to prevent this particular issue from happening again.

Sticky files will stay in your current media, and therefore on our fast, primary media delivery network, for as long as you are a libsyn user. Please note, that any space a sticky file consumes will not be available for other files to use. (ie, if you have a 100MB account, and make 20MB file sticky, you will only ever at most have 80MB of space for future uploads)

Please leave a comment (or spam, that'd be awesome) on this forum post letting us know your feelings.

Probably 99% of the incidents of 404's on file downloads people have been seeing the past several days are directly related to the archive network failures. The steps taken today and those we will hopefully be able to take tomorrow, should clear up all issues pertaining to downloading older files.

Another day, and more headaches with our switch. Cisco is dragging their heals with a replacement, and the move-in to the new datacenter is taking longer than we expected or wanted. The aspect of the service most affected is the archives, but during the peak hours, all downloads are suffering. We don't have that much more to relay other than we are working all angles we can right now to bring the download speeds back up and eliminate the timeouts.

We are doing two major upgrades to our Media Delivery Network over the next couple days which should hopefully put us back ahead of the rush for data currently going on in podcasting. There apparently is a physical hardware problem with our data switch, which is being replaced, in our primary data center. In addition, we are bringing new servers online in another new data center. Both these upgrades have been planned for a while- the recent explosion has just moved that timetable up a little bit. Since the beginning of December, our average traffic load has almost doubled. That ramp is much higher than we have seen previously.

As always, thanks to everyone for their patience and understanding as we battle through these technical issues.

Just to be clear. We were able to add the second port channel to our switch, however there seems to be some configuration or software issue which is preventing us from pushing up over the 1 gbps mark. Our certified network techs are working the issue and we will post when there is something to report

update

Libsynners have been noticing mounting issues with slow starts, timeouts, and less then awesome download speeds. We have identified one of the causes to be network saturation. We are working with our team in the datacenter to get another gigabit network connection in asap. Is podcasting blowing up in popularity? Well, the answer is in our MRTG graphs. Just 2500 podcasts are saturating a gigabit network connection... crazy.

Please hang with us while we upgrade to handle you.

Last week, when we had the power blow out in one of our important load balancer servers, we recovered and then put in a fail-over system which more of less included adding another server to do the mirror of what the load balancer does, and then split the load between the two (which was also meant the improve the performance of 'click .. .. .. .. .. . download'. So, the new box went it, all was well and good, and everything (including stats) seemed to be working fine. Well.. in the truth was, in fact, that the stats weren't making it back to the stats database from the new load balancer.

So the bad news is, about 1/2 the hits from the last few days haven't been getting counted into your numbers. The very good news is, we are now much better at recovering from the old logs and can back-fill the missing data. It still takes us a little while to reassemble the millions of file requests, so we appreciate it if you pass it along that everything will be restored, and the as of right now, we are counting correctly.

We are really sorry for the past few days, it seems when it rains it pours.. and it sucks.. Thanks for hanging in there with us.

Update (honest truth of what's going on):

The outcome:

Hello Libsynners,

If you've been following this blog, you'll know that since the summer we've been taking steps to add redundancy and robustness to our media delivery network.

By robustness we mean the number of downloads the network can sustain at any given time (we currently, by our most conservative estimates, serve around 3.5 million files each week). We have been and contiinue to beef up our infrastructure as the demand for your programs increases.

By redundancy we mean that should any single part of our system fail, your files will continue to be served. A few weeks ago we made a big move towards this redundancy, seperating our media delivery network from the main libsyn site.

Note that links to your media should be updated to point to http://media.libsyn.com/media/your-username/file-name to provide your listeners with the best / fastest performance possible. Old links (http://libsyn.com/media/your-username/file-name) will still work, but will result in a slightly slower connection to your file).

In addition to that seperation of media delivery from the main site, we've also added a failover for the media delivery, so incidents like last night, where a faulty power supply resulted in ~1 hour of slow connections and timeouts, will be much less likely.

Please let us know if you have any questions about the new linking standard, or any of the recent changes to the libsyn network. Thanks as always for your feedback and support

- Your Libsyn Support Team

UPDATE:

A faulty power supply caused a disruption in part of our delivery system. We redirected around the problem while the power supply was being replaced. Now things are back in full. We will be implementing a redundancy here as well as better monitoring tools to prevent this in the future.

We apologize for the inconvenience.

Boy do we have a year in store for you! While visions of sugarplums where dancing in everyone's head, our elves have been hard at work. The result of months of deep development are ~just~ about to come to fruition. While you all have been building your audiences, the technology has been coming together behind the scenes, and we are now proud to present to you, the libsyn + Kiptronic sponsorship program.

We've been working with our friends at Kiptronic since last July to bring libsyn users a program to match up their podcasts with paid sponsorships. We have planned carefully to ensure that the sponsorship program provides a fair and open marketplace. With libsyn, content creators always maintain control of their content. Both publishers and sponsors are able to easily set their own terms and negotiate deals that make sense for them. The program is, of course, 100% optional and won't cost you a dime unless you get paid!

Here's how it works:
The next time you login you will see a link to the libsyn + kiptronic sponsorship program. Click on the link for a description of the system, and you will be taken to your kiptronic profile account which is ready to be activated. It's free to try, and there is no obligation to accept any sponsors. When you and a sponsor agree on the terms of a sponsorship, will become active and the agreed upon audio will be inserted into episodes your program, until the end of the arrangement is reached.

The Good Stuff:
How much you make, how the spots sound, when they are played, and how much you get paid are all up to you: an agreement must be reached by both parties regarding the terms of any sponsorship before it goes into effect. You can do spoken endorsements in your own voice, or the spots can be provided by the sponsor. You can opt for the audio to be inserted into the beginning or the end of each episode, and you can negotiate your own prices for each spot. Because the sponsor's promo is inserted dynamically upon file download, you can have multiple sponsors with rotating spots, and make money from your entire archived catalog of episodes. You can have ads in your own voice, or choose to use spots created by the sponsors. Best of all, You set the price, you set the terms, and you always have final say as to whether a sponsorship goes through.

What lies ahead:
Take a minute and read through the information on the Kiptronic site, and fill out the profile information. What we need from you, if you are interested in finding paying sponsors for your podcast, is to register yourself as available through the Kiptronic system. Once we are able to get interested podcasters into the program, we will set the sponsors loose to start looking for podcasts.

Who are the sponsors?
While we have sponsors ready to participate in this program, we would like to exhibit to them the range and diversity of content that you all are currently creating. This will result in the best deal for you publishers, and will highlight the value of your creation to the sponsors. If capitalizing on your hard work and creativity sounds good to you, go login to https://www.libsyn.com right now and activate your account with the libsyn + kiptronic sponsorship program.

Thanks as always for your loyal support,

Your Friends at Liberated Syndication

Things are functioning much better. We'll let you know once its all clear.

First off, this currently only affects a small % of libsyn users. This in no way affects the stats tracking, displaying, or eventual dissemination of the LAA (libsyn active audience) and libsyn stats system. For those of you who do take our XML logs, run some scripts you have made on them, and do whatever it is you want to do with them, we now have an updated feature. You can select the libsyn XML format, or standard Apache log format.

For new customers, you will have to enable this feature if you want downloadable logs to start building for you. We are creating the logs from the database that updates the flash stats display. We will also very soon have a public (or private) RSS feed for your stats that you will be able to subscribe to. This does not affect that either. For existing customers, if you currently don't use this feature, please do us and everyone else a favor and turn off stat logging. We have, by default, left logging on for ever existing user, but would greatly save disk space and system resources if we only logged for those who absolutely needed it.

For more on this subject, please see the forum post