Mon, 24 November 2008 An issue has just arisen with one of our providers which is causing a Redirect Loop error. We've contacted our provider and are working on the issue. Thanks for your patience. Update: 11/25/08 1:30am EST We have resolved the issue with one of our service providers that was causing the 'redirect loop' error. All files are now being served. There was no interruption in data collection. Please contact support@libsyn.com if you have any other questions. Category: Media Delivery -- posted at: 8:29 PM Comments[4] |
Thu, 23 October 2008 Update 10/24/08 11:00am EST
We have completed performing maintenance on our Stats system after experiencing a hardware issue earlier this week. Stats have been restored across the Libsyn system and are now displaying for all users. We were able to use our secondary sources of data to update the system, which resolved the issue. Please contact Libsyn Support (support@libsyn.com) if you have any other questions.
Due to some hardware issues that occurred earlier in this week, we are currently running some maintenance on our stats system. Stats will appear to be frozen, but in actuality are rolling back about a week. Users will see stats progress forward through the night until they are full caught up later tonight. No data appears to have been lost. Please let us know if you have any questions. Thanks for your patience. Category: Stats -- posted at: 3:35 PM Comments[1] |
Thu, 2 October 2008 Update (6:00pm EST) - Our data center technicians have replaced a
faulty disk and are in the process of rebuilding a RAID array. Stats
data is continuing to be collected across the Libsyn system, and we
expect full functionality to be restored by tomorrow morning at the
earliest. We will continue to provide updates on the situation.
Update (10/03/08 1:00pm EST) - A faulty disk on the indie Stats log parser has been replaced. The Stats parser has begun to process data collected across the Libsyn system. Updated stats data will be available shortly. Category: Stats -- posted at: 5:44 PM Comments[1] |
Wed, 1 October 2008 The Libsyn Indie Stats updates are currently paused while backend maintenance is being performed on the hardware running this system. While numbers will not appear to change, all downloads are still being captured and no data is being lost. We will keep everyone posted on an ETA for the system to be back up. Once the Stats service has been restored, it may take several hours for the most recent Stats numbers to display while the Libsyn system updates user data.
Update (7:00pm EST) - The database update revealed hardware problems with one of the disks in the RAID array. The datacenter has not provided us with an exact ETA. Fortunately, the data that was processed up to that point was not affected, nor was data collection. The hardware is being serviced on-site and we expect to start processing in the very near future. We apologize for the inconvenience and appreciate your patience. Our statistics system has made some huge strides in the past year and we expect to see even bigger improvements coming soon. One of the safeguards recently put in place is the ability to retain data in an event like this one. Again, no current numbers have been impacted and, as soon as service is restored, the processor will catch up quickly. Category: Stats -- posted at: 11:04 AM Comments[1] |
Sat, 20 September 2008 The stats page was temporarily unavailable this morning due to an internal issue. There has been no data loss
and all new stats were collected during the interruption. The datacenter team has restored the service to all users. Please contact Libsyn Support if you have any questions. Category: Stats -- posted at: 8:29 AM Comments[1] |
Thu, 14 August 2008 We will be performing some maintenance on the database tonight beginning around 11pm eastern time. There may be a few short interruptions of service and we suggest users avoid logging into the system between 11pm and midnight. Category: announcements -- posted at: 5:52 PM Comments[0] |
Tue, 5 August 2008 The stats view is temporarily unavailable this morning due to server issues. There has been no data loss
and all new stats are still being collected. The datacenter team is
working as quickly as possible to restore the service to all users.
UPDATE: The stats display issue has been resolved. Stats are now displaying for all users. Please contact Libsyn Support at support@libsyn.com if you have any other questions. Category: Stats -- posted at: 9:06 AM Comments[0] |
Tue, 22 July 2008 We have experienced problems with one of our main Libsyn.com servers early this morning. Our back-up measures have taken effect but there have been momentary disruptions in service while we resolve the issue. Media delivery and user stats have not been affected. Category: general -- posted at: 2:29 PM Comments[0] |
Sun, 20 July 2008 We are currently experiencing issues with one of our main service
providers. Please be advised that new uploads will not become
available for download until these issues have been resolved. -------------------------------------------------------------------------------------- Update 7/21 9:00am The file serving issue with one of our service providers has been resolved and uploaded files are available for download. Please contact Libsyn Support (support@libsyn.com) if you have any questions. Category: general -- posted at: 1:57 PM Comments[1] |
Mon, 7 July 2008 The stats view may be unavailable from time to time this morning while the backend database is being worked on. There has been no data loss and all new stats are still be collected. The datacenter team is working as quickly as possible to restore the service to all users.
Update (6:05pm EST): The server repair is taking longer then expected, the team is working through the night to repair the service as quickly as possible. The download data is still being collected and no data has been lost as a result of this downtime. Update (11:15 pm EST): A replacement server has been brought online and about 60% of the data has been copied over. We are attempting to bring up a backup image of the database from yesterday to allow users access to all stats leading up to this mornings failure while the remainder of the data is copying. If you login and see that the last updated date is behind, rest assured that it's still copying, or processing and will catch up very soon. Category: Stats -- posted at: 9:44 AM Comments[3] |
Tue, 10 June 2008 There are 2 issues to report on. First, there have been intermittent database errors this evening resulting in this error message for users attempting to login to libsyn.com and to visitors to libsyn blog pages and rss feeds. The engineering team continues to monitor the servers closely and take necessary measures to prevent these errors from occurring again. Media delivery, stats tracking, ad serving, and libsynPRO where not affected by these database errors.
Second, there were some upgrades made today to the libsyn.com stats display system to improve compatibility with Firefox 3.0. This code change caused some temporary issues for some users loading their stats. There was no interruption in stats collection or processing during this time- only an unusually slow load time. Both issues have been resolved. Category: Bugs -- posted at: 12:10 AM Comments[2] |
Thu, 29 May 2008 We had to react to an emergency database issue this afternoon which took down the website for a couple hours. There were no interruptions to media downloading, stats processing, and no data was compromised or lost. Logins to libsyn.com were unavailable, and access to blog pages and RSS feeds was intermittent.
All systems have been restored and are now fully operational. Category: general -- posted at: 6:13 PM Comments[1] |
Mon, 12 May 2008 On Tuesday, May 13th between 3:00pm - 4:00pm EST web and FTP login will be disabled while we perform database maintenance. During this period, media downloads, RSS feeds, blog pages, and statistics tracking will not be affected. Logging in, uploading, and add comments to blog posts will disabled during this period of time. Category: announcements -- posted at: 7:14 PM Comments[0] |
Sat, 3 May 2008 We are investigating FTP upload issues reported this morning. Will update when there is more information
update: we are monitoring the FTP servers closely as they have been acting a little finicky all day. All is up and operational as of 8:30pm EST. FYI, web uploading has been fine throughout this and for most cases should be a suitable workaround for most cases if you should have trouble getting in through FTP. Category: Bugs -- posted at: 10:15 AM Comments[2] |
Thu, 24 April 2008 Between 3:30am and 10:30am on 4/24, requests for media files were intermittently resulting in "404 File Not Found" errors. The outage was caused by human error / a mis-configured startup sequence on one of our media redirectors. We apologize for the trouble. The problem was discovered around 10am and resolved by 10:30. Category: Media Delivery -- posted at: 11:11 AM Comments[0] |
Sat, 12 April 2008 As promised yesterday, we have pushed out a new version of stats interface. Responding to many user's feedback, we have added:
Category: Stats -- posted at: 9:10 PM Comments[1] |
Thu, 10 April 2008 Libsyn stats v.1.6 has been out for about a month now. The feedback we have gotten so far has been very positive and many suggestions we received are either in the works or planned for the near future. If you haven't already, please feel free to jump into the discussion on the forums. We are continuing to improve and make fixes to the current system, and will roll out a new release over the weekend.
The development of our next generation stats system has been very exciting. We really can't wait to show off everything stats 2.0 will do! While we progress to that stage, there are some more measures we need to take to improve performance and stability. The first is to purge some old XML and Apache logs. If you are currently using the apache or xml logging feature, any log file older then January 2008 will be purged from the system on 4/18/2008. Following that date, the first of the month will mark the rollover period whereby only the oldest month will be purged. (eg. on 5/1/2008, February logs will be removed,etc, etc..). The download links for the logs will be moved to the 1.6 stats interface and the 1.5.x interface will be officially phased out. The access to historical data back to September 2006 will still be available through the stats interface- nothing is being removed from there. The vast majority of libsyn users will not be affected by this data retention policy. Finally, while we are repairing the issues being reported of recent episodes not being picked up, the stats display may appear to be a bit erratic. We are restoring to a backup from earlier in the week and then reprocessing the weeks worth of data. We anticipate everything to be back to normal late this evening into the early morning hours. Thanks to everyone for their feedback and suggestions in helping us build a better stats experience for all users. update 4/11 12:30pm The prep work for the brief rollback we were planning for early this morning took longer than expected. We will be performing the rollback this afternoon and will post back here when it is complete.
update 4/11 6:20pm The rollback is complete. Stats should be current and up to date. We've recovered the missing data from over the last few days (thought there may be slight amounts of irrecoverable data from last weekend). Stay tuned also for some requested features we'll be pushing out in short order. Category: Stats -- posted at: 5:16 PM Comments[0] |
Mon, 7 April 2008 We have been tracking down the bug(s) which is causing the last episodes published by some users. We will keep you published as we learn more. update: we have temporarily stopped the stats engines- meaning no new updates of any files will occur for the time being. We are working on a plan to recover the data for those users affected over the last several days worth of data. We do not want to jump the gun on pointing the finger for what happened, it would seem that the recent interruptions with Amazon EC2- which we take advantage of for our stats processing- has caused some unexpected behavior. This would also account for why a swath of users across only some of the partitions were affected. Update (4/8/2007) 9:00PM: Stats processing resumed earlier this evening. Everything appears to be caught up, and for the most part, missing downloads have been restored. It is possible that a subset of new files got missed as we could not tell the exact time in which each EC2 engine got disrupted. We took a time from 1200 GMT on friday as our roll back point. There were measures taken to mitigate double counting of downloads for files which were tracking fine over that time period, but it is also possible that some double counts occured. We have repaired this mishap as best we can and are moving forward with the work on the next generation stats system which will handle these types of issues in a much better manner. I apologize for the inconvenience this has caused. Category: Stats -- posted at: 2:02 PM Comments[1] |
Mon, 31 March 2008 We are currently experiencing some database issues which is causing problems connecting and uploading new media. We are working to restore service as quickly as possible
update: We will continue to monitor the load on the databases. The disruption was intermittent over the course of a couple hours in the evening on Monday. The team reacted quickly, resolved the issue, restoring the service to those experiencing problems logging in, navigating the website, and uploading media. If you uploaded any files between 9:30 - 11:00 PM EST and are either not seeing them show up in your account or are experiencing 404 file not found errors, you may need to try uploading again. Category: podcasts -- posted at: 10:34 PM Comments[0] |
Fri, 28 March 2008 To you straight from the mouths of our datacenter team: "I will try to explain it as easy as possible. at 18:45EST 3-27-2008; The PFC3B line card in the Primary Supervisor of Core1 went into a LOOP; This caused an issue that did not allow Core1 to process Layer3 traffic across the MSFC3. Layer2 however was not affected in this matter which allowed Core2 to continue running normally and pass Layer2 traffic Across the Core1 Layer2 Back-bone. HSRP which is used as a counter-measure; still showed Core1 as functional even though it was experiencing a Loop. at 19:00EST; a CISCO SMART-NET Tech has been dispatched with additional hardware (SUP, MSFC3, PFC3) in case a defective board was found. At 19:35EST, Core1 was taken off-line in preparation for Cisco's Arrival. Upon arrival at about 20:45EST; a debug was generated and CISCO has re-certified the Module. We are waiting on Final Explanation from CISCO as to what caused the PFC3 daughter board to act in the way that it has. We have taken numerous Physical counter-measures to make sure that this will not happen again. Some of these counter-measures include modification to the HSRP algorythm as well as physical scripting. Lastly; This incident only affected clients that the HSRP was Active on the Device. We have placed more sophisticated HSRP counter-measures." Laymen's explanation: there was a hardware failure. Our side of the blame line is not having enough replication of our DNS records in another datacenter. That is being fixed today. Sorry again for the downtime. Category: general -- posted at: 10:51 AM Comments[0] |
Thu, 20 March 2008 As you may have already heard (and as reported in this article), two major network carriers (Cogent and Telia) have "stopped talking to each other." While Libsyn.com does push a decent amount of traffic through Cogent (which could mean problems for users on the Telia network requesting that content), we are connected to the net through multiple large providers. The word from our main data center is that most, if not all, of the requests for libsyn content from users on the Telia network should get sent through alternate (non-cogent) channels. If you or someone you know is having problems getting content from libsyn, please find out the IP address of the affected user (try http://whatismyip.com) and send it to Libsyn suport letting them know you're having issues. Thanks! Category: general -- posted at: 4:08 PM Comments[0] |
Tue, 11 March 2008 We launched v.1.6 of the libsyn stats service yesterday. Here's a 7 minute demonstration video to see what they are all about. Please excuse the fact that I have a cold.. and am not, in fact, the EVP of Stats.
Comments[1] |
Fri, 7 March 2008  Issues addressed in this release:
There are currently 2 partitions which contain many popular users and therefore taking longer to re-process that last 9 months worth of data. We've held off an extra couple weeks from releasing this version while we waited for all users to catch up. Our feelings now are why make the 90% suffer for the 10% that are behind. If you are unlucky enough to be in that 10%, please be patient while your data catches up. For the most recent issues some users have been experiencing this week regarding slow or no updates, those engines are working to catch up. You should see v.1.5.2 numbers updating over the weekend. Category: Stats -- posted at: 3:09 AM Comments[0] |
Thu, 28 February 2008 Just wanted to update everyone on how the rollback process is going. We're happy to report that about 85% of the reprocessing is complete, and our testing is showing that issues with missing data are being resolved as this process progressess. We are also very excited to report that the new stats UI is working, tested, and ready to roll as soon as the re-processing is complete.
Our goal to get the new data to all of you this week looks like it's going to slip into next week. There was a lot of data to run through and it's taking longer then we had hoped. The good news is that the new UI, combined with some new hardware to back it, is a lot faster and more responsive than the previous UI. So not only will your data be easier to understand, but it should load much faster as well. We will keep everyone updated as things progress. Category: Stats -- posted at: 10:43 PM Comments[4] |
Tue, 19 February 2008 Hi everyone. We are happy to report that the system wide stats rollback / recovery is moving right along. We expect the analysis and testing to be complete next week. At that point we'll release the updated numbers and UI updates into your dashboard. Thanks as always for hanging in there. Category: Stats -- posted at: 3:22 PM Comments[0] |
Tue, 19 February 2008 Over the past few weeks we've received a number of reports of users replacing existing files, but the new versions of those files never becoming available. We tracked down the root of this problem (related to a recent configuration change with our CDN provider) and got it fixed. Now, when you update an existing file, the new version should be available for download immediately, and the old version should no longer get served. Category: Bugs -- posted at: 1:49 PM Comments[0] |
Fri, 15 February 2008 We are currently investigating an error with recently uploaded media returning an XML error file instead of the media. The problem seems to be with our backend service provider.
We are working with the providers to resolve the issue as quickly as possible. Update: problem seems to be very widespread and affecting other customers of the amazon S3 service: details from Amazon Forum The "origin" server for the libsyn CDN service is held on S3. It has historically been very very solid, and I'm sure will have the service back up very quickly. As of 9:30 EST they have not provided an ETA. Update: Files being uploaded will not be available for download righ away as a result of this issue. Once everything is resolved on amazon's end, we will reload all the recent uploaded files. Thank you for your patience. Update: As of 10:30 EST, the CDN is now available again, and all files should now be available. If you notice problems with any media files, please contact support@libsyn.com. Category: Media Delivery -- posted at: 9:13 AM Comments[0] |
Mon, 11 February 2008 Dear Users, We have an update on the status of libsyn.com stats! We know stats accuracy is an ongoing issue and that libsyn users have had to exhibit a lot of patience and swallow a lot of frustration as we've worked through the problems over the years. As we said in a previous post, we have assembled a solid roadmap towards achieving stability and accuracy, and providing industry-leading statistical reports that tell you producers and network administrators exactly what you need to know about the consumption trends of your content. As promised, we took the first step of identifying and fixing the issues with the current libsyn.com stats (mainly, some episodes published after July 2007 not showing up in the system, or suddenly stop reporting downloads). We sifted through the code that processes the raw data into data we can use in the reports we provide to you, and believe we have identified the root of the problem, as well as the fix for it. We've set up a test environment, applied this fix, and are now in the process of re-processing the raw data from Summer 2007 forward and verifying that this "fixed" data is accurate. Depending on how that verification goes, the reprocessing could be finished in as soon as 7 - 10 days. While fixing the existing system is an important step, we are really excited about the development that is getting underway for a longer term solution. As we've said in the past, the current iteration of the stats application was built to be scalable, but it just has TOO many moving parts and missing pieces (for example the ability to speedily re-process data, or easily rollback and re-process from an arbitrary date). We are working on a plan that will provide scalability, flexibility for us in terms of administrating the application, and will give us the ability to continue to store historical data indefinitely (a wall that the current system was bound to come up against sooner or later). This is the "data warehouse" model we discussed in a previous post, and is a key component to our long-term plans for the libsynPRO enterprise service. The good news for all of our independent (i.e. libsyn.com) users is that the "libsynPRO" stats engine is also going to be the libsynINDIE stats engine and we are fast tracking the development and integration into both systems. It's still a little too early to give an accurate deliverable date, but we are shooting for early second quarter of this year. Finally, as a bonus we are working vigorously to implement an updated stats UI into the current libsyn.com dashboard. We were originally waiting for libsyn.com 2.0 to be done and in place before we started introducing new features, but the long-loading XML+Flash interface is pretty long in the tooth, and we figured out a way to shoehorn in the new UI into the old system, so we figured what the heck. We've beaten you guys up so much over the past year regarding stats performance and stability, the least we can do is SHOW you all the data we have been collecting but were unable to deliver in a meaningful way for so long. So, that's where things stand. We will keep you posted as the reprocessing progresses. Love, Libsyn PS - you'll notice that the libsyn.com stats updates are now happening less frequently than normal. We've had to tone-down the update cycle for those during the re-processing, since both the live system and the system running in the background work off of some of the same data Category: Stats -- posted at: 1:22 AM Comments[1] |
Mon, 21 January 2008 From about 8:00 to 8:40pm on tonight, some requests for files in the network were not being redirected properly, and in some cases, the wrong file was delivered. We were able to fix the problem as soon as it came to our attention. We apologize for any trouble this may have caused you or your listeners. Category: Bugs -- posted at: 8:58 PM Comments[0] |
Mon, 14 January 2008 Hi Everyone. First off: our apologies for this post not being made sooner. We were short-staffed over the weekend, and the most important thing (letting you all know something was up) got overlooked. There were two issues this weekend affecting some, but not all of Libsyn publishers: 1) Saturday Night / Early Sunday morning around 2:30am Eastern, the datacenter we colocate a bulk of libsyn servers in was performing some routing maintenance on their core routers. We were told to expect a more or less seamless experience, but as it worked out, there was an oversight on their end which resulted in some people not being able to connect (download files, upload files, or check RSS feeds). Alarms started going off on our end around 4am, and we worked with them though the night / into the day on Sunday repairing the problem. 2) During this same time one of the machines that houses a section of users' home directories had a disk that was becoming full. With all of our focus on the other problem, this disk filled up completely, meaning new files didn't get saved properly. Both of these issues could result in the "Access Denied" XML error that many of you are seeing. This basically means the file was not found on the CDN. Please let our support team know if you're getting this error and, if your file made it to your home directory, then we can force it out to the CDN. If the file isn't in your home directory, re-uploading it will fix the problem. We're sorry for the trouble this has caused anyone, and are committed to getting any broken files fixed as fast as possible. Note: We are also having some issues currently with FTP. It looks like a disk went bad on one of our FTP servers. If at all possible, please use Web upload until we update this post saying everything is fixed. When it rains, it pours. Thanks for hanging in there. Update: The FTP issue has been resolved, so feel free to use the FTP method for upload again. _________________________ If you are having trouble with access errors please include username AND filename when emailing support@libsyn.com. Thank You! Category: general -- posted at: 2:32 PM Comments[3] |
Thu, 13 December 2007 This is an update on what is being done to improve the stats - Over the holidays we will be changing where the statistics are collected. Previously the libsyn service was hosted on multiple cdns; one that was managed in house and then two to three other cdn providers. The statistics were gathered by the redirect media nodes, that would accept requests, refuse bad requests, redirect based on db references to where particular media files are hosted, and collect statistical data. This was not an efficient architecture decision as the redirect nodes were handling too many tasks. While all the stats data was being collected, the system was also introducing noise into our logs that would ultimately choke up the stats parsing engine. This would cause the the stats displays to report improperly or fail to report at all (spinning beach ball). We would then have to go in and remove the noise by hand. This took a significant amount of time for the repairs and for the re-running of data. We would then have to wait as the stats engine caught up to date. The reasons we managed multiple cdns were to keep our costs down for our users, and to allow us to scale our system from the in-house distribution platform launched in '04 to one based on industrial solutions. What has changed over the last couple of months is that we have migrated our network to a unified cdn partner. Now all media is served from one system. The media is no longer shuffled to different cdns based on demand (an inefficient overhead), or moved to slower servers when archived. Our download performance has been greatly improved for all media. We have also begun migrating our stats data storage from sql based databases to a higher grade platform. We are now migrating the collection of the stats data off of the redirect nodes. We will be collecting the statistics directly from our cdn partner. This data output is much cleaner and we believe this will improve the performance of the stats crunching engines greatly. We also have been working on a more streamlined reporting tool for the stats tab of your libsyn account. We have to take it one step at a time; first the consolidation of media which is now complete, then the change of collection method to data directly from the cdn. This should bring our performance to an industry best state. When this is completed we can finally overhaul our stats interface based on our users input, and the experiences we have gained from Wizzard Media's advertising initiatives. We are very positive about the plan we have in place and we will give you updates as they progress. So much of the work has been behind the scenes architecture changes, and we realize our users cannot see any immediate benefits. But we assure you we are committed to making libsyn the best publishing platform for podcasters. We are happy to say that in early 2008 you will see the fruition of all the work that has been done to fortify the stats system. Category: Stats -- posted at: 3:06 PM Comments[1] |
Mon, 3 December 2007 We have discovered a break in the stats engine which occurred on friday, 11/30, which have caused updates to backup for most users over the weekend. The systems are being restarted and should pickup the updates over the course of today and into tomorrow.
There is also the standing issue of missing data dating back over the last several months. That is a longer process to fix, please be patient while we work on that as well. We apologize for the inconvenience. Category: Stats -- posted at: 12:22 PM Comments[5] |
Sat, 1 December 2007 After spending the better part of the day yesterday trying to understand what was going on with the delivery network (for anyone who didn't notice, requests for newly uploaded files were failing with 503 and 504 errors), we were able to trace the problem to one of our service providers who had rolled out some new, less than stable code for our account. Normally we would let you all know if we were planning an upgrade or anything that might interrupt service, but this particular upgrade happened without notification for us, so we couldn't notify you. We apologize for the inconvenience the problems caused, and you can be sure we are making it very clear that something like this cannot happen again. The availability of your files is vital to both your livelihood and ours. We finally were able to get the new code rolled back, which almost immediately fixed the problem, around 12:30am this morning. If you find you are having problems with any of your files, please open a ticket with support, and make sure you include the URL of the file in question. Thanks Category: Media Delivery -- posted at: 7:19 PM Comments[1] |
Fri, 30 November 2007 We are currently experiencing issues with our file delivery CDN. You may experience intermittent issues as we are working with the data provider to fix the issue.
More updates to follow. update: We have had been working all day between our origin data center and our CDN to resolve the issue. The problem continues to plague new files uploaded, and older files that may not have been downloaded in a while. We are looking to put some stop-gap fixes in place while the issues with the CDN get resolved. We apologize for the inconvenience this issue has caused. Category: Media Delivery -- posted at: 11:39 AM Comments[3] |
|
Libsyn Support Blog Weblog of the status of the Liberated Syndication System.
|
|||||||||||||||||||||||||||||||||||||||||||||||||
