To support some of the new features we are developing, we did a major rev upgrade of some of our core software. The results were great on all fronts, except our support ticket system. Nothing got lost, however all the ticket assignments that we use to keep things organized and put into queues so that they get answered in some reasonable amount of time, got reset.

This means there's going to be a couple good days worth of work to go back and re-queue everything. We appreciate your patience if support responses are slower than expected while we get things sorted out. Thank you.

Update: (3:30am) Logins enabled and full operations back online.  That last wave took longer then we would have liked, but the good news is  that media delivery persisted throughout the entire move.  We are striving and building to the point where everything has that level of geo isolation and redudency.  We will be there soon.

---

Update: (2:00am) Third wave moved and back up.  The final wave we're going to have to take our time with.  We're going to lockdown logins and uploads first.  Then when everyone currently uploading is complete, we'll gracefully bring everything down and then move it and bring it back as quickly as possible.  There are many moving parts in this last phase, we need to make sure they all go back together properly.

---

Update: (1:19am) Second round of machine transplants went very smooth.  It started around 1:00am and took about the same amount of time roundtrip.  Third wave up...

---

Update: (12:34am) The first wave of moves started a little behind schedule, but we're moving along as planned now.  The 2nd set are being powered down now.  From offline in old center to online in new was about 20 minutes for round 1.

---

Update:  Just want to make sure everyone follows what's going on tonight.  We are doing a migration in 4 waves, starting at 11pm EST.  The first 3 waves should not cause any noticable disruption to the service.  There may be some load issues that cause some slowdowns, but we don't think they'll be too bad.  The network should self-heal, and load balance around the machines in transit to the new facility.  The final wave, which should probably hit around 2am EST, will put the service into a temporary "lockdown" mode.  While we're locked down, media will continue to be delivered, but most other functions of the site will be down for 15-20 minutes.

We have lots of hands on deck to ensure this process happens as smoothly and seamlessly as possible.  Redbull in hand, we're in for a long night.  If there are any hiccups in our plan, we will react fast and recover.  Thank you for your understanding and support.

---

The maintenance that was announced (and then postponed) in this post, has been rescheduled for this Wednesday.  We will be performing some critical maintenance that will cause anywhere between 20-45 minutes of restricted access to libsyn publisher services.  RSS feeds will be unavailable for a short time, as will files in your /images and /_static folders.

Work will begin around 11pm EDT Wednesday night and should take a few hours. To re-iterate, during this time there will be a period of 20-45 minutes (most likely later in the night / around 2am) where some services become unavailable.

Media delivery should continue unfettered, and we will work as quickly and carefully as possible to minimize the impact on you and your listeners.

Thanks for understanding.

Around 5AM (EST) several of our servers were targeted by a ddos attack (distributed denial of service). As a result, the main libsyn page, uploads, your blog pages, rss feeds, the forums, and mail services were all affected. Links to media were affected, but not completely down (connects were slow due to the load shifting on to all the nodes that weren't affected). We sincerely apologize for the disruption. Our tech team at the datacenter did a great job working with the service providers to restore the service as quickly as they could. It unfortunately took a few hours. The nodes are coming back online now, and we should be back to full operating power shortly. Thank you for understanding and sticking with us through some rough waters this past week.

We are experiencing core network issues in one of our partner data centers, which is causing possible poor user experiences and denied access to services.  Our team is hard at work now to resolve the issue and restore full functionality to affected users at the earliest possible moment.  To those that are experiencing disruptions to service, our sincere apologies for any inconveniences this causes, and we will inform you immediately on this board of ongoing progress.

Login access for both the libsyn website and FTP servers will be disabled for a short spell at 1:00am EDT on Friday, June 16th.  We expect the maintenance will take under 30 minutes, at which point login access will be restored.

Thanks for your understanding.

Systems are solid and back to normal, with heavy traffic catching due to the outage that affected a portion of Libsyn users.  Unfortunately a portion of statistical data was lost for a short increment of time.  This situation has been remedied and the tech team is hard at work ensuring your feeds are always working, regardless of the technical challenge.

Update: Early Monday morning for about an hour, a segment of our users were unable to upload new content or have access to their blog pages. RSS feeds and media delivery did not suffer from the issue which was hardware related. The team maintained as much of the service as was possible while the hardware issue was resolved at the data center. Additionally some Libsyn users have experienced problems with the blog page, or logging in to upload new media. We apologize for the inconvenience caused by this interruption. As always we are striving to improve service and quickly react to unforeseen events. 

We are no longer calculating statistics on  the following file formats:  "jpg", "jpeg", "gif", "png", "html", "htm", "phtml", "xml".  This is  to counteract the inflation of audience statistics due to  the inclusion of these files in the media distribution folder, as well as to optimize the performance of our stats engine.  Images, as a general rule, should be uploaded using the "new post" form, or via FTP to your /images folder.  Any concerns about this revision should be sent to support@libsyn.com with subject: 7/11/06 Stats Revision.

Hey everyone, we need to get our heads out of the weeds and explain some poor performance over the last few days.  There have been some spots where media delivery has slowed, timed-out on the start, and generally been flakey.  A few culprits have popped up, and one by one we have been dealing with them.  Some of them have been hardware, some misbehaving software clients, others network related.  We wish we could proclaim that we have found the sliver and pulled it from the lion's paw... but in this case, the damn cat fell in a bit of a picker bush.

We appologize for the lack of communication on the matter, there have been a great many things going on consuming everyone's time.  Please take some comfort in knowing we are working all the issues and will restore peace and order to the podcasting world once again.