We are currently experiencing an extremely high and unusual load on our web-engines (the systems that generate feeds and blogs) – we have put the system into maintenance mode while we investigate.
UPDATE – 7PM EST – We’ve been able to mitigate the bad incoming traffic and have turned the logins back on. All systems are back to normal now.
On Monday May 16th starting at 8AM we will maintenance on the stats system. This maintenance includes some pretty serious hardware upgrades to the two key statistics systems we have – reporting and processing. The first part of this maintenance will affect stats reporting – so you’ll likely see red error boxes when trying to view your stats. We expect this part to last at most 15 minutes. The majority of the work is to the stats processing system – this will cause stats processing to be paused for up 8 hours.
We will update this post throughout the maintenance period – noting when stats reporting is back on and then again when stats processing has been restarted and finally when stats processing is all caught up.
UPDATE – MAY 16 – 8AM EST – Maintenance window is beginning – stats reporting will be the first thing affected, you will see red error messages in the stats dashboard for roughly 15-20 minutes.
UPDATE – MAY 16 – 8:15AM EST – Stats reporting is back on and starts processing is now paused – it will be paused for up to 8 hours.
UPDATE – MAY 16 – 4:45PM EST – Stats processing is being turned back on – we will be watching it carefully and will make another post after processing gets caught up.
UPDATE – MAY 16 – 11:30PM EST – Stats processing is progressing well and we except them to be fully caught up sometime overnight.
We’ll be doing some system maintenance at 8AM EST on Monday. We expect this maintenance to be completed quickly but we are setting a 30 minute time window for it. During this time logins and FTP will be disabled, while media downloads, feeds and blog pages should be unaffected. We will update this post on Monday morning when we start the maintenance and then afterwards when it’s completed.
UPDATE [850 Eastern]: Update went smoothly, everything is up and running as expected. If you notice anything unusual, please contact support.
We are currently experiencing a delay receiving raw logs from our CDN, which has stopped stats processing. Check back for an update as soon as we find out when log delivery will be restored.
UPDATE: [1730 Eastern] CDN technicians have been looking into the problem since early this morning. We are still waiting to hear back from them about log delivery being restored.
UPDATE: [1820 Eastern] Our CDN has informed us that the issue was related to a hardware failure. They have restored logs from yesterday and today from an unaffected backup system and we are in the process of downloading these logs. Processing should commence within the next hour. Over the next day, stats will be backfilling and will be caught up as soon as possible. We appreciate your patience and apologize for the inconvenience. If there are any changes, we will update this post.
UPDATE: [5/6/11 – 9AM EST] Stats processing caught up overnight and is functioning normally again.
