Update: We are still investigating the true reason for last night- and very early this mornings- performance issues. I am fairly convinced the IP's that we blocked coincidentally helped relieve the issue, which was caused in fact by something else. We may have jumped the gun a little bit reporting that it was a malicious attack. Thanks for everyone's support as we keep wrestling with the logs and graphs and reports to find the real problem and make sure it's fixed.

For now here's how it works. Login, under the Settings / Advanced section you'll see a check box to enable comment moderation. Once you turn that on, all comments left on your blog will be hidden until you go in and enable them. To enable the hidden comments, under the publish / previous posts screen, if you click on the comments viewer, you will see a button to enable the comments. When you have new comments, you will see a link like this when you first login (see screenshot).

UPDATE:

The move has been delayed some as our techs at the datacenter double and triple check wiring, configurations, and routing for all of our equipment.  We will make a new post here with at least 24 hours notice of when the actual upgrade is going to take place.

--

Our primary datacenter is upgrading and expanding (partially to meet our growing demands). We have been working through a series of updates and are a reaching a point where there will been some physical hardware shifted around this coming week. Through careful planning and our redundancy in multiple datacenters, we intend to keep mission critical functions, like serving media to your audience, up and functioning at all times. There will however be a short period on Tuesday, May 16th where you will be unable to upload new media, login to post shows or check stats for up to an hour.

We will be dialing in the exact times as this stage of the process grows closer, and will post an update with  the latest news. This estimate is the worst-case scenario, and our technical team will be in constant  communication with the folks "on the ground," ready to react to any issues that should arise during the transition.

As always, we want you all to have complete clarity and understanding of what we are doing and that we are focused on ensuring the least inconvenience and downtime possible. Thanks

• bold
• italics
• font colors

1. Numbered
2. lists

In other news. we have been a little tardy the past few days in responding to emails and to forum posts. We are going to try to make it a general rule of thumb to get through emails before we hit the forums. The time to do both, AND make improvements to the system is so sparse, so in order we'll answer: emails, then forum, then person instant messages. The bigger problem, however, is that our system has bugs and we have poor documentation. The first thing is our constant "to-do" list and we will make improvements with over time. Now that it's spring- we're kicking our house cleaning into high gear. As for the other thing, documentation, here's where we're calling on all of you to help us out a little (if you have some time).

Now that it's the 21st century, we have finally put some work into our documentation wiki. http://docs.libsyn.com is now updated a little more (working on it even as we type this email) and we openly welcome all your libsyn zealots to come help us fill in the cracks. Now, we know you all would do it just for your unconditional love of podcasting, but we do have plans to reward the big contributors (that goes for the community forum as well). Everything from sticker, pins, t-shirts, free accounts, marty's car.. will be given as small tokens of our appreciation to those who have continued to give back to the community. Having a good set of documentation will greatly cut down on the number of support emails we currently get. We'll try to emerge a practical compliment between the forum and the wiki so new users (and old) can self diagnose and help themselves understand the common issues of libsyn. Then a quick stop at the http://support.libsyn.com blog for system alerts and issues, and then last resort an email.

We hope everyone is enjoying the new hardware that has been put in place over the last couple weeks. Good news coming from the Kiptronic camp as well in terms of new sponsors coming into the marketplace very soon as well as some awesome new features we'll be excited to share with everyone soon. I will take a quick minute to talk about one of those, and that is libsyn stats. As mentioned on our stats display for months now, RSS stat feeds are almost ready to roll. What is also going to happen, is those users who choose to sign up for the kiptronic marketplace will automatically get their libsyn stats fed into the kiptronic system and displayed to potential sponsors. IF YOU ARE NOT USING KIPTRONIC, YOUR STATS WILL NOT BE SHARED WITH THEM. That's very important to note that we will not disclose your information to anyone who you have not given express consent to. The way you authorize the sharing of your audience data with other libsyn partners (soon to come) is through the advanced settings tab. We are also considering releasing podcast rankings to the public for those who wish to opt in. We welcome your feedback on these topics, either via private email or in the public community forum.

As always, thank you all for letting us syndicate your content. It is an honor. Thank you for hangin with libsyn, through thick and thin, and helping to build the greatest network of podcasts in the 'verse. (Firefly fans, let me hear ya)

Leaving you with a little support podcast that was sent to us recently. Wouldn't it be funny if everyone sent us audio support requests?

let me answer that.. . no

Because archiving was off for a number of days, we have had to turn it back on again in phases. We've made multiple sweeps of your home directories, archiving files that are 30 days old, and the plan is to make another complete sweep again tonight, re-enable quotas, and then put archiving back on an automated schedule.

At least one person has pointed out that a number of files that were archived on Friday were resulting in 404's. This was because of a short archive run that went astray. We believed we had restored all of the missing files, but if we missed one user, it is possible there are more. If you notice 404's on any of your archived files, please let us know right away.

Finally, while we are hustling behind the scenes here at libsyn hq, Dave M. has been on the road meeting with various friends in the industry. In this picture, Dave is discussing some funding options with our new financial analyst... stay tuned for more.

The issues with the archives not processing, and new uploads 404-ing are temporary and should be resolved shortly

This is just a workaround while we get more optimizations and new hardware online.

To be clear, requests from all browsers (IE, Firefox, Safari, etc) and iTunes will continue to be honored while we are working through this issue

Following last weekends system maintenance, we had disabled the change password and reset password functionality on libsyn.com. As of this posting, the change and reset password functions have been re-enabled, so if you have any trouble loggin into the website or via FTP, please reset your password and try again.

Hello good synners. The first quarter of 2006 has brought explosive growth to the libsyn network. Yesterday we delivered 1,042,739 downloads and as of this posting we are hosting more than 12,000 files on our main media network and almost 60,000 files in total.

To keep up with this growing demand, we're doing some re-structuring of our network. Beginning around 2am est, logins will be disabled, as will new signups. We will do everything we can to execute our plan quickly and carefully, to minimize the wait for those of you wanting to access your account. Media delivery and all of your blog pages and RSS feeds should go unaffected.

We appreciate your understanding, and your choice to be a part of the libsyn family. We will let you know when the maintenance is finished and account access is restored.

Thanks,
- Your friends at libsyn

The new features you'll find are:

• iTunes duration(Time) support!
• We're still working on the back end receiver script to determine proper durations for the files uploaded automatically. This is totally not that hard a thing to do.. it's coming
• iTunes keyword support
• iTunes explicit tag support
• iTunes subtitle support
• removal of author email as a required field
• This was just one of those misinterpretations of the RSS spec on my part initially. Now, if you prefer to not put an email address out into the wild, so be it. If you leave it blank, and have the Author field populated on your iTunes settings, it should display in ITMS with that Author. This is again our interpretation of the iTunes spec. Please let us know if it's not working out that way for you.

The real-time tracking will be turned back on shortly and will start recording again. Sorry for the misqueue. We are updating our db tables to prevent this particular issue from happening again.

Sticky files will stay in your current media, and therefore on our fast, primary media delivery network, for as long as you are a libsyn user. Please note, that any space a sticky file consumes will not be available for other files to use. (ie, if you have a 100MB account, and make 20MB file sticky, you will only ever at most have 80MB of space for future uploads)

Please leave a comment (or spam, that'd be awesome) on this forum post letting us know your feelings.

Probably 99% of the incidents of 404's on file downloads people have been seeing the past several days are directly related to the archive network failures. The steps taken today and those we will hopefully be able to take tomorrow, should clear up all issues pertaining to downloading older files.

Another day, and more headaches with our switch. Cisco is dragging their heals with a replacement, and the move-in to the new datacenter is taking longer than we expected or wanted. The aspect of the service most affected is the archives, but during the peak hours, all downloads are suffering. We don't have that much more to relay other than we are working all angles we can right now to bring the download speeds back up and eliminate the timeouts.

We are doing two major upgrades to our Media Delivery Network over the next couple days which should hopefully put us back ahead of the rush for data currently going on in podcasting. There apparently is a physical hardware problem with our data switch, which is being replaced, in our primary data center. In addition, we are bringing new servers online in another new data center. Both these upgrades have been planned for a while- the recent explosion has just moved that timetable up a little bit. Since the beginning of December, our average traffic load has almost doubled. That ramp is much higher than we have seen previously.

As always, thanks to everyone for their patience and understanding as we battle through these technical issues.

Just to be clear. We were able to add the second port channel to our switch, however there seems to be some configuration or software issue which is preventing us from pushing up over the 1 gbps mark. Our certified network techs are working the issue and we will post when there is something to report

update

Libsynners have been noticing mounting issues with slow starts, timeouts, and less then awesome download speeds. We have identified one of the causes to be network saturation. We are working with our team in the datacenter to get another gigabit network connection in asap. Is podcasting blowing up in popularity? Well, the answer is in our MRTG graphs. Just 2500 podcasts are saturating a gigabit network connection... crazy.

Please hang with us while we upgrade to handle you.

Update (honest truth of what's going on):

The outcome:

Boy do we have a year in store for you! While visions of sugarplums where dancing in everyone's head, our elves have been hard at work. The result of months of deep development are ~just~ about to come to fruition. While you all have been building your audiences, the technology has been coming together behind the scenes, and we are now proud to present to you, the libsyn + Kiptronic sponsorship program.

We've been working with our friends at Kiptronic since last July to bring libsyn users a program to match up their podcasts with paid sponsorships. We have planned carefully to ensure that the sponsorship program provides a fair and open marketplace. With libsyn, content creators always maintain control of their content. Both publishers and sponsors are able to easily set their own terms and negotiate deals that make sense for them. The program is, of course, 100% optional and won't cost you a dime unless you get paid!

Here's how it works:
The next time you login you will see a link to the libsyn + kiptronic sponsorship program. Click on the link for a description of the system, and you will be taken to your kiptronic profile account which is ready to be activated. It's free to try, and there is no obligation to accept any sponsors. When you and a sponsor agree on the terms of a sponsorship, will become active and the agreed upon audio will be inserted into episodes your program, until the end of the arrangement is reached.

The Good Stuff:
How much you make, how the spots sound, when they are played, and how much you get paid are all up to you: an agreement must be reached by both parties regarding the terms of any sponsorship before it goes into effect. You can do spoken endorsements in your own voice, or the spots can be provided by the sponsor. You can opt for the audio to be inserted into the beginning or the end of each episode, and you can negotiate your own prices for each spot. Because the sponsor's promo is inserted dynamically upon file download, you can have multiple sponsors with rotating spots, and make money from your entire archived catalog of episodes. You can have ads in your own voice, or choose to use spots created by the sponsors. Best of all, You set the price, you set the terms, and you always have final say as to whether a sponsorship goes through.

What lies ahead:
Take a minute and read through the information on the Kiptronic site, and fill out the profile information. What we need from you, if you are interested in finding paying sponsors for your podcast, is to register yourself as available through the Kiptronic system. Once we are able to get interested podcasters into the program, we will set the sponsors loose to start looking for podcasts.

Who are the sponsors?
While we have sponsors ready to participate in this program, we would like to exhibit to them the range and diversity of content that you all are currently creating. This will result in the best deal for you publishers, and will highlight the value of your creation to the sponsors. If capitalizing on your hard work and creativity sounds good to you, go login to https://www.libsyn.com right now and activate your account with the libsyn + kiptronic sponsorship program.

Thanks as always for your loyal support,

Your Friends at Liberated Syndication

First off, this currently only affects a small % of libsyn users. This in no way affects the stats tracking, displaying, or eventual dissemination of the LAA (libsyn active audience) and libsyn stats system. For those of you who do take our XML logs, run some scripts you have made on them, and do whatever it is you want to do with them, we now have an updated feature. You can select the libsyn XML format, or standard Apache log format.

For new customers, you will have to enable this feature if you want downloadable logs to start building for you. We are creating the logs from the database that updates the flash stats display. We will also very soon have a public (or private) RSS feed for your stats that you will be able to subscribe to. This does not affect that either. For existing customers, if you currently don't use this feature, please do us and everyone else a favor and turn off stat logging. We have, by default, left logging on for ever existing user, but would greatly save disk space and system resources if we only logged for those who absolutely needed it.

For more on this subject, please see the forum post

According to the folks at nanog:

Cogent is experiencing 2 fiber cuts in the north/southeast-US. The locations are: Between Houston and Tampa (Southeast) Ticket #347277 Between Philidelpha and DC (Northeast) Ticket #347375

As a result we have seen about a 20-30% decrease in traffic. We will keep you updated as things develop.

We then spent the rest of the day pushing on the registrar to accelerate reactivation of the domain. Things started coming back around 5:30pm today. The lessons have been learned, we are changing registrars.

We appreciate all the patience that you guys have given us today.

http://libsyn.org will get you to the site right now, but unfortunately, that is not going to help your listeners that are having problems.
Update: Files are now being served via the .org domain. This does not help your subscribers, but may be of some assistance.

Thanks for sticking with us folks. We know it's been up a roller-coaster year. If we didn't love hearing your voices screaming our names (mostly) in pleasure and (sometimes) in pain, we wouldn't get out of bed in the morning. Now that we got people pushing video, we can't wait to see where soapy will turn up next!

A form of DOS attack hit our data center this evening around 8:50PM. Thankfully we have a very alert and responsive tech staff that reacted to the attack and has brought our network back online in less than half an hour.

We are trying desperately to get all caught up on email and support forum issues. If you email us, and don't hear back in the traditional amount of time, please give us a little bit more time to get everything caught up. We installed an email management system _right_ before 3 major events occurred. Nothing like baptism by fire I guess.. keeping tabs on the status of requests is easier, just making it work into our traditional workflows is taking a little time.

No more excuses, we'll just keep chipping away

Dave M. circa 2005 not posing

For those of you who handle your own dns entries, you'll want to update the record for any domains pointed at libsyn.com. The new ip address is 38.100.193.69

Also for anyone having trouble logging into the libsyn FTP servers, this could be an issue with the DNS not being fully propagated. If you are having this problem, try using the IP address instead of ftp.libsyn.com to be sure your ftp client is trying to connect to the right server.

We are now waiting for the DNS change to propagate, so if you still see the site as having login disabled, it's just a matter of time until your local cache updates.

We are pretty happy with the way things went today, took longer then we hoped, but the performance increase should be worth it

Please report issues immediately to support@libsyn.com

next update when things are done and back to normal

Ladies, you will all get a chance to thank M personally and buy him a birthday drink at the podcast expo this november.

M. circa 1996

We are doing our best to respond to every email as quickly as we can.