there is now an 'advanced options' section of the post table. This is where we will be putting new fields for everyone to play with. The first is just a big, generic text area where you can put other tags. The problem is you can't put your own custom tags in there yet, because you can't define your own namespaces. I guess that's the next thing to put in there. But in the meantime, you can put creative commons rdf tags in there, clean tags in there, keyword1 keyword2 keyword3 etc...

Note on spam: We seemed to have uncovered another access point into our comment system. We are working to patch the newly discovered issue as quickly as possible.

Our replacement switch arrived today and we will be installing tonight. We don't anticipate more than a few moments of disruption, but just wanted to give everyone the heads up. We have to schedule time with the team in the datacenter to ensure the switchover happens at the most opportune time. Our hopes is that this upgrade will give us more headroom to push even more data.

It's been more then 3 days now since the last spam comment has been placed on a libsyn blog page, so we can hopefully say that the hole the spammers were exploiting has been closed. We are now facing the dilemma of: 'should we go through and try to scrub the system of spam?' What's the line or definition of a spam post- er what if we delete something that's not spam. Is it worth it- or should we leave it up to you to remove your own spam ?'

Please leave a comment (or spam, that'd be awesome) on this forum post letting us know your feelings.

We finally have a bit of good news to report about the state of the libsyn delivery network. After a harddrive failure in one of our archive nodes added to our problems, we were able to rebuild the machine with a new harddrive, and in addition bring more nodes online for the archive network. We also moved some of our servers to a temporary switch while we await the replacement for our main one. We hope this keeps up with the daytime rush tomorrow as we have a new bank of media nodes coming online in the new datacenter in the afternoon(fingers crossed).

Probably 99% of the incidents of 404's on file downloads people have been seeing the past several days are directly related to the archive network failures. The steps taken today and those we will hopefully be able to take tomorrow, should clear up all issues pertaining to downloading older files.

Another day, and more headaches with our switch. Cisco is dragging their heals with a replacement, and the move-in to the new datacenter is taking longer than we expected or wanted. The aspect of the service most affected is the archives, but during the peak hours, all downloads are suffering. We don't have that much more to relay other than we are working all angles we can right now to bring the download speeds back up and eliminate the timeouts.

Thanks to everyone for their words of encouragement and patience.

We are doing two major upgrades to our Media Delivery Network over the next couple days which should hopefully put us back ahead of the rush for data currently going on in podcasting. There apparently is a physical hardware problem with our data switch, which is being replaced, in our primary data center. In addition, we are bringing new servers online in another new data center. Both these upgrades have been planned for a while- the recent explosion has just moved that timetable up a little bit. Since the beginning of December, our average traffic load has almost doubled. That ramp is much higher than we have seen previously.

As always, thanks to everyone for their patience and understanding as we battle through these technical issues.

Last night's upgrades prooved to help some, but was not enough of a solution to hold us over until we are able to purchase a new gigabit fiber connection or expand to a second datacenter. We apologize for the lack of communication today, but we pretty literaly were pulling our hair out all day trying to shuffle things around, on the phone with the DC and network providers, and doing everything in our power to keep the speeds up. Part of our build out plans include bringing alot more delivery nodes online in another location, geographically isolated from our current datacenter. We're working the best we can to get to that point asap.

Just to be clear. We were able to add the second port channel to our switch, however there seems to be some configuration or software issue which is preventing us from pushing up over the 1 gbps mark. Our certified network techs are working the issue and we will post when there is something to report

we're going to light up some new fiber tonight. Sorry for the last minute update, we just got word that we can go ahead now. Since things are pretty slammed still, we figure sooner better then later. We're hoping for no more than a few moments of downtime, but we're trying to brace for the worst. More updates soon

---

update

I'm not stupid enough to claim success at 4am (our lowest traffic time of the day) but things went well with the port upgrade, and while we took the main website down for a couple minutes we took the opportunity to tweak some database settings as well. Again, while it's the lull right now, everything is super smooth and fast. We will keep a close eye monitoring things through tomorrows peak times and see how the new headroom affects the performance. Sorry again for the growing pains... but I think we just spurted up a foot.

Yesterday we thought we had fully corrected the problem that was leading to low stats numbers for this past weekend. We were wrong. As of 7:45 pm, Wed, Jan 11, we *have* corrected the problem, and confirmed this correction, but still need to recover numbers from apache logs from the period of 8:00 monday morning until we finally got the new loadbalancer recording stats correctly. This recovery operation will take place overnight.

Libsynners have been noticing mounting issues with slow starts, timeouts, and less then awesome download speeds. We have identified one of the causes to be network saturation. We are working with our team in the datacenter to get another gigabit network connection in asap. Is podcasting blowing up in popularity? Well, the answer is in our MRTG graphs. Just 2500 podcasts are saturating a gigabit network connection... crazy.

Please hang with us while we upgrade to handle you.

Every libsyn users is encouraged to download and read the latest revision of our end-user agreement. Thanks to the keen eye of a few libsynners, we have consulted further with our legal team and changed some wordings, added some material, and taken a couple things out to further separate what is our property and what is yours. The short of it- anything you upload is yours and we make no claims on it. That's the way it has always been, just wanted to make sure that's how the lawyers see it in ink.

Here's the truth about what's going on with the stats recently: (believe it or not, i'm going to route it back to a power supply again)

Last week, when we had the power blow out in one of our important load balancer servers, we recovered and then put in a fail-over system which more of less included adding another server to do the mirror of what the load balancer does, and then split the load between the two (which was also meant the improve the performance of 'click .. .. .. .. .. . download'. So, the new box went it, all was well and good, and everything (including stats) seemed to be working fine. Well.. in the truth was, in fact, that the stats weren't making it back to the stats database from the new load balancer.

So the bad news is, about 1/2 the hits from the last few days haven't been getting counted into your numbers. The very good news is, we are now much better at recovering from the old logs and can back-fill the missing data. It still takes us a little while to reassemble the millions of file requests, so we appreciate it if you pass it along that everything will be restored, and the as of right now, we are counting correctly.

We are really sorry for the past few days, it seems when it rains it pours.. and it sucks.. Thanks for hanging in there with us.

Hey everyone, we are isolating issues we are having this morning with a couple of our databases and should have everything back to normal very soon. THank you for your patience

---

Update (honest truth of what's going on):

We have restored connectivity and everything is working from the users POV. There is still a bug somewhere which we are trying to track down that causes mounting connections to our db servers. The reason we were not alerted to the issues sooner, was that our monitoring application was not properly configured from our previous issues with the power supply.

The outcome:

We are able to continue to bug fix without impacting any users. Once things are finally normal, we will spend a great deal of time revisiting our entire monitoring approach. As always, thank you for your continued patience and support.

Hello Libsynners,

If you've been following this blog, you'll know that since the summer we've been taking steps to add redundancy and robustness to our media delivery network.

By robustness we mean the number of downloads the network can sustain at any given time (we currently, by our most conservative estimates, serve around 3.5 million files each week). We have been and contiinue to beef up our infrastructure as the demand for your programs increases.

By redundancy we mean that should any single part of our system fail, your files will continue to be served. A few weeks ago we made a big move towards this redundancy, seperating our media delivery network from the main libsyn site.

Note that links to your media should be updated to point to http://media.libsyn.com/media/your-username/file-name to provide your listeners with the best / fastest performance possible. Old links (http://libsyn.com/media/your-username/file-name) will still work, but will result in a slightly slower connection to your file).

In addition to that seperation of media delivery from the main site, we've also added a failover for the media delivery, so incidents like last night, where a faulty power supply resulted in ~1 hour of slow connections and timeouts, will be much less likely.

Please let us know if you have any questions about the new linking standard, or any of the recent changes to the libsyn network. Thanks as always for your feedback and support

- Your Libsyn Support Team

We are currently resolving some media node issues which is causing slow responses for media requests and timeouts. We will have it back shortly.

UPDATE:

A faulty power supply caused a disruption in part of our delivery system. We redirected around the problem while the power supply was being replaced. Now things are back in full. We will be implementing a redundancy here as well as better monitoring tools to prevent this in the future.

We apologize for the inconvenience.

Happy 2006 everybody!

Boy do we have a year in store for you! While visions of sugarplums where dancing in everyone's head, our elves have been hard at work. The result of months of deep development are ~just~ about to come to fruition. While you all have been building your audiences, the technology has been coming together behind the scenes, and we are now proud to present to you, the libsyn + Kiptronic sponsorship program.

We've been working with our friends at Kiptronic since last July to bring libsyn users a program to match up their podcasts with paid sponsorships. We have planned carefully to ensure that the sponsorship program provides a fair and open marketplace. With libsyn, content creators always maintain control of their content. Both publishers and sponsors are able to easily set their own terms and negotiate deals that make sense for them. The program is, of course, 100% optional and won't cost you a dime unless you get paid!

Here's how it works:
The next time you login you will see a link to the libsyn + kiptronic sponsorship program. Click on the link for a description of the system, and you will be taken to your kiptronic profile account which is ready to be activated. It's free to try, and there is no obligation to accept any sponsors. When you and a sponsor agree on the terms of a sponsorship, will become active and the agreed upon audio will be inserted into episodes your program, until the end of the arrangement is reached.

The Good Stuff:
How much you make, how the spots sound, when they are played, and how much you get paid are all up to you: an agreement must be reached by both parties regarding the terms of any sponsorship before it goes into effect. You can do spoken endorsements in your own voice, or the spots can be provided by the sponsor. You can opt for the audio to be inserted into the beginning or the end of each episode, and you can negotiate your own prices for each spot. Because the sponsor's promo is inserted dynamically upon file download, you can have multiple sponsors with rotating spots, and make money from your entire archived catalog of episodes. You can have ads in your own voice, or choose to use spots created by the sponsors. Best of all, You set the price, you set the terms, and you always have final say as to whether a sponsorship goes through.

What lies ahead:
Take a minute and read through the information on the Kiptronic site, and fill out the profile information. What we need from you, if you are interested in finding paying sponsors for your podcast, is to register yourself as available through the Kiptronic system. Once we are able to get interested podcasters into the program, we will set the sponsors loose to start looking for podcasts.

Who are the sponsors?
While we have sponsors ready to participate in this program, we would like to exhibit to them the range and diversity of content that you all are currently creating. This will result in the best deal for you publishers, and will highlight the value of your creation to the sponsors. If capitalizing on your hard work and creativity sounds good to you, go login to https://www.libsyn.com right now and activate your account with the libsyn + kiptronic sponsorship program.

Thanks as always for your loyal support,

Your Friends at Liberated Syndication

Wow, you guys are getting popular! Our servers having been smoking since coming back from the holidays.. must be all those iPods everyone got for christmas. We are bringing more hardware online to handle the increased load. The biggest problem currently is slow starts for media files. We are going to try some things to fix that as best we can while the new servers are being built and installed. Thank you everyone for your patience.