Since we introduced the public RSS feed option and the ability to redirect your-username.libsyn.com/rss to that address, some of you have taken advantage of this feature to redirect your libsyn feed URL to your feedburner feed. This caused a slight problem, since feedburner gets your feed information by checking your libsyn feed (thus creating an infinite loop).

We got together with the folks at feedburner to fix this problem as soon as we leanred of it. If anyone is still having problems with their feed, please drop an email to support.

Thanks.

Update: We had to deal with 2 major outages today as a result of todays disaster. The technical explanation for why the redundant power did not kick in was due to a faulty ATS switch within one of the UPS backups. When the power went out, the UPS's kicked in, in battery mode. Once the juice started to get low, one of the UPS's automatic transfer switches didn't cut over to generator power. The core routers for the datacenter were being serviced by that UPS. Everything was fixed early this afternoon and running off the generators for about 7 hours. Then, when the mains power came back on, the switch failed again this time in the other direction, switching back. The UPS system is being serviced. We are all learning a thing or two about seldom used emergency systems, and will be monitoring everything very closely. We again apologize for the effects felt by this natural disaster. Thank you for your continued support

---

libsyn.com has been affected by Hurricane Wilma. Our datacenters are in florida, and apparently the power outage there brought us down. We are investigating as to why backup generators and other precautions did not prevent this outage. As soon as we know something we will let you know. We are sorry for the inconvenience, More updates on this coming.

A form of DOS attack hit our data center this evening around 8:50PM. Thankfully we have a very alert and responsive tech staff that reacted to the attack and has brought our network back online in less than half an hour.

Hey folks, We've had some requests for automatic RSS feed redirecting, and we've gotten around to putting it in there. To make it work, you'll put the URL you want your feed redirected to in the "public rss feed" setting under basic settings. It's not active yet, though, because I wanted people with a setting in there already to know what was going to happen.

With this setting active, we won't generate a libsyn rss feed anymore, it will just redirect username.libsyn.com/rss to the one set in your settings pane. I'm planning to make this active in about a week, but if you want to have it active sooner, email support, and we'll put you in there sooner. As always, with any questions, email support or post on the forum.

Yours,
- hoopes

Periodically, you may see one of the following messages upon publishing a post:

Could not ping audio.weblogs.com: Can't call the script because the name "pingSuite" hasn't been defined.

Server Responded: Can't find a sub-table named "weblogs".

These messages are coming from the audio.weblogs ping server. The best thing to do if you get this error is to wait a few minutes, and then try manually pinging them at http://audio.weblogs.com/pingform.html

In some cases, you'll see that despite the warning, your ping actually has gone through.

From 10am until around 1PM today you may have experienced slow connections to libsyn and probably even some timeouts. The cause of the problem stemmed from the way we were previously handling the archive media network. When we started, the archive network was 6:1 smaller than the main media delivery network. As we have grown, the amount of data in the archives continues to grow so we have had to expand the number of nodes. The network logic, however, hadn't been scaled with the addition of more servers.

The main libsyn media delivery network has been greatly tweaked and enhanced to provide seamless data delivery in the event of hardware or software failure on any of the individual nodes. Currently serving over 2 million media files a week, or approx. 7 terrabytes of data, its been pretty good for the most part. Outside issues dealing with databases, general network outtages, and file uploading aside, the libsyn media delivery network has had a pretty good track record with download speeds usually topping anything we see from other media websites. This mornings issues have caused us to reevaluate the handling of the archive network and upgrade it to the level of robustness we have in place for the main media network.

It looks like connections between cogent and level 3 are coming back up, as seen here: http://status.cogentco.com/ We are here monitoring everything, and working to keep things as smooth as possible. We'll keep you posted.

We are trying desperately to get all caught up on email and support forum issues. If you email us, and don't hear back in the traditional amount of time, please give us a little bit more time to get everything caught up. We installed an email management system _right_ before 3 major events occurred. Nothing like baptism by fire I guess.. keeping tabs on the status of requests is easier, just making it work into our traditional workflows is taking a little time.

No more excuses, we'll just keep chipping away

 

Dave M. circa 2005 not posing

It's a crap deal, two big internet service providers, cogent and level 3 are in a tiff. They are not talking to each other. I'm not trying to be funny, it's crappy whats going on. An email should have been sent out, it may have fallen into black whole that these providers have created.

Bottom line, for those of you that can't reach libsyn send us your ip addresses. We are passing those off to our datacenter guy who is manually rerouting how our site gets to you. Also important, according to our charts your shows are being served to MOST of your audience.

This apparently has never happened before. It's not just us either, parts of the internet can't get to other parts of the internet. Big companies being Big Jerks. I need coffee. below is better explanations by my more eloquent buddies.

Here's the drafts:

Marty put it like this:

Hey everyone... libsyn is up and running and has been since the migration last Sunday night. What you're experiencing is the result of a corporate catfight between 2 major networks who facilitate large amounts of inter-connectivity across the net.

Dave C said this:

Dear libsyn customers-

Sorry to invade your inbox with more email, however we wanted to let everyone know we are ontop of the current issue going on in Internet land. "This is history in the making" says our ISP. http://ask.slashdot.org/askslashdot/05/10/05/2247207.shtml

We are working to get around this issue which mostly seems to be affecting TimeWarner Roadrunner users specifically by taking the IP addresses of the users being affected, looking at the IP block of their provider, and manually adjusting our BGP tables. If all that went right over your head, no problem. Bottom line: if you are having troubles connecting to libsyn.com, we are not offline. Please email support@libsyn.com with your IP address, and we will work our best to reroute the connection so you can get in.

It seems many users of the Time Warner roadrunner internet service are having trouble reaching libsyn.com. It seems to be at the network level, there are connection problems and our ISP and techs are working to help solve the problem. Our network load shows that the majority of your audience is still receiving your podcasts and the issue seems bound to the Roadrunner customers.

For those of you who handle your own dns entries, you'll want to update the record for any domains pointed at libsyn.com. The new ip address is 38.100.193.69

Also for anyone having trouble logging into the libsyn FTP servers, this could be an issue with the DNS not being fully propagated. If you are having this problem, try using the IP address instead of ftp.libsyn.com to be sure your ftp client is trying to connect to the right server.

It was a bloody battle, but I think we came out on top. The migration of the main libsyn.com webserver to the new datacenter went off today without any impact to media file downloads. User login was disabled for longer than we would have liked and the forums were also disabled for a few hours this afternoon.

We are now waiting for the DNS change to propagate, so if you still see the site as having login disabled, it's just a matter of time until your local cache updates.

We are pretty happy with the way things went today, took longer then we hoped, but the performance increase should be worth it

Please report issues immediately to support@libsyn.com

Things got pushed back a little bit this morning due to an underestimate for the time to copy user log files. We are just about to truely start the switch. User logins and uploads will be temporatily disabled during the middle of the move. Media delivery should be maintained through the entire process, with the exception of a few spot down times.

next update when things are done and back to normal

Please join me in wishing one of the libsyn squad, Marty, a happy birthday. For a present, we gave him the task of migrating a core piece of libsyn to a new data center :)

Ladies, you will all get a chance to thank M personally and buy him a birthday drink at the podcast expo this november.

Ok Marty, NOW GET BACK TO WORK!

M. circa 1996