We are playing a bit of catch-up on support emails. Today was another banner day for traffic- nearly 8TB (terra-bytes) and we are paddling harder than ever to keep everything running. This weekends upgrades are sorely needed.

We are doing our best to respond to every email as quickly as we can.

dear libsynners- As part of our continuing effort to provide the best podcast delivery service, we are undergoing some major transitions and upgrades. During that upgrade period, which we intend to begin approx. 12:00 PM EST this Sunday 10/2/2005, you may experience some service disruption. Our game plan is to complete the transition with enough time to fully test the new setup with before the monday morning rush. Once complete, the IP address of NS1.libsyn.com and NS2.libsyn.com may be changing. We will update http://support.libsyn.com with the updated IP addresses should we need to change them. The intended results of this upgrade are faster page loads, decreased delay in media file download starts, decreased propagation delay from the time uploaded to full distribution across our delivery network, increased upload bandwidth to allow of fast file uploads, stronger faster and more reliable database access, and improved archive download speeds. We have been experiencing more growing pains than we would have liked these past few weeks, and we can't say enough about how much we appreciate everyone sticking with us. Thanks to the support of all of you, we are able to offer our service. Your continued patience and understanding through these tremendous times of growth and change is always appreciated.

I just want to acknowledge the slowdowns, followed by a short period of downtime, we experienced this morning. From around 9:00-10:30am we saw a significant performance hit on the main server. For about 15 minutes of that time, the site was completely down. I apologize for your frustration- we take libsyn uptime very seriously and have admittedly struggled the past month to maintain consistency throughout our network. The problem this morning was caused by a runaway media node that backed a lot of traffic up to the main server. I'll be implementing a threshold today to prevent this particular problem in the future.

First want to say thank you to everyone for your patience. We had a serious bug which took us almost an entire day to recover from. Everything looked 100% up to the last minute, then we had another issue popup. That issue, causing the most recent downtime (last several hours), has been repaired, and the full data recovery from Sunday's issue has completed.
Please let us know immediately if there are any issues. We are setting up a virtual call center so libsynners can get us on the phone quickly in the event of serious problems. Until then, email support@libsyn.com first, then try one of us on skype, then as a last resort you can get my cell number from webmayhem.com.
I personally want to commit to all of you that we are working on the issues that arise from serving this huge amount of data. I take responsibility for our recent performance flaws, and wish to make it right with each of you one at a time. Your satisfaction and enjoyment using libsyn is my priority.

-Dave Chekan

Hey Guys, Everything should be coming around very soon. It took well over twelve hours for the whole thing to normalize since the server downage yesterday. With podcasting there is never any slow of traffic, so there is no recoup time. Our email, was down so we are backed up on support emails, but we will get to everything. We are literally in the last stretch of getting to all systems go. We will let you guys know when we are 100% on here as always.. talk is cheap.. we'll get things back on track. Very sorry about the inconvenience.

Update: We are continuing to fight a battle to keep data moving through our network. You may experience some periods of media files being unavailable. All hands are aware of the situation and are working to repair the problems as quickly as possible. As always, we appreciate your support and patience.

---

Our servers took a nosedive this evening and experienced some downtime. The issue was network related and somewhat out of our control. What is in our control is a more robust fallback which is what we are moving to in the very near future. We apologize for the inconvenience.

we have the new hot rod running stats, and the old pinto is being retired

bluejack has been toiling away in the thrilling world of statistics aggregation and data mining and a bunch of other numbers crap

spot checking has shown the data to be identical to the old stats, so hopefully you won't even notice this switch except for the fact that stats are updated much more often

of course, you guys are the worlds best customers/people/friends/internetters, and i don't know where i was going with this sentence, but it must be because i am giddy that BJ has chipped this monkey off of my back with an ice pick

please let us know if you notice anything fishy (notify us as gently as possible please, you know my ego is fragile, and i haven't been hugged in a hot minute)


peace and chicken grease, ya heard?

love,
- hoopes

leave forum posts here:
http://libsyn.com/forum/viewtopic.php?t=980

Update from apple:
We just implemented an email notification system. If we have a problem with a feed AND the feed contains a contact email, we notify that person that there's a problem.
The message below goes out if the iTunes system can't reach the feed. Usually it's a temporary problem. But we send it out so the podcaster can check and see if everything is working ok. If it is, they can just ignore the message. If they keep getting it --- then there's some deeper problem that needs to be investigate.

We're still working with them to see what was causing the problem initially. There was a performance slow down today. It's a catch 22 thing to figure out which is causing which. We are working hard on the problem.

---

We know many of you are receiving warnings from iTMS that there is a problem with your feed, preventing it from being updated in iTunes.

We have checked several of the feeds in question and found them to be valid. You can test your own feed by visiting FeedValidator.org.

We have contacted the good folks at Apple asking for clarification on what exactly the error it is, so we can determine what steps need to be taken to get your feeds updating in iTunes again

If you donate $50.00 to the Red Cross Hurrican Katrina relief fund we will give you a 1 year account of the Podcast Basic service. Sign up for a new Podcast Basic account. Submit it, but stop at the payment processing section (paypal.com). Email us a copy of your donation receipt and we'll active your account for a year.
Is there anyone out there who is willing and able to match every dollar our new libsyn customers are donating?