[Resolved] Emergency Maintenance miami-b14

4:03AM EST – We’re investigating an issue affecting users in miami-b14. Please hang on while we find an issue, we’ll update this blog post asap.

6:00 AM EST – We had a critical raid failure. We are on recovery mode and Webbies in miami-b14 are currently down. We are migrating each Webby’s data so that and starting them on other nodes.

7:30 AM EST – Webbies still being migrated.

4:00 PM EST - The majority of Webbies have been migrated. We have 5-6 Webbies till being migrated manually in order to ensure data integrity.

7:00 PM EST – The majority of Webbies were recovered yesterday. We had to do hard data recovery for the last few ones. We are working on the last 2 at this point.

4:00 AM – EST – All Webbies have been recovered. If your Webby has any issues still please contact support.

[Resolved] Webby Manager Maintenance

We’re working on an issue with the Webby Manager. We’ll update this post as soon as we get updates.

Note: All Webbies are up.

Update: The Webby Manager is back online, please send us a ticket if you run into any issues.

Thanks.

[resolved] miami-b07 connectivity issues

Some customers in miami-b07 node are experiencing connectivity issues. We’re investigating and will post updates soon.

Update 2:30PM EST: All webbies on miami-b07 are up and running. Problem resolved.

[Resolved] Miami – Connectivity Issues

[12:00 PM EST] We are experiencing connectivity issues with the Miami location and it’s also affecting connectivity between Webby Manager and Dallas data center.

We’re working on mitigating this issue, please stand by.

12:15 PM EST Connectivity has been resumed and all services are back up.

Connectivity issues with 67.23.79.x range

[09:45 PM EST] We are experiencing connectivity issues with the 67.23.79.x IP Range and it’s also affecting connectivity between Webby Manager and Dallas data center.

We have full hands on deck working on a fix.

Please standby for updates.

[09:55 PM EST] Connectivity has been restored.

Dallas01 – Network Issue

As of 10:50AM EST, we are working on bringing back the Webby Manager for the Dallas location.

Note: All Webbies are up.

11:02AM It seems we are having a network issue on the dallas site, we are investigating.

11:05AM The issue is directly affecting customers on dallas-01. We’re working on a fix.

11:15AM We are rebooting dallas-01 at the time.

11:50 Webbies are coming up at this time.

All Webbies up and running. Please open a support ticket if you have any further issues.

Miami: Level3 Connectivity Issues.

3:10AM EST We are seeing some Level 3 connectivity issues at this time, with about 70% packet loss to the Miami, FL location.

We’re working with level 3 to restore connectivity, we’ll keep monitoring and updating this ticket.

Update 3:14AM Connectivity has been restored.

We will keep monitoring this issue to avoid any re-occurrence.

Thanks,

Carlos.

Dallas Network Maintenance / Sunday Oct 24, 2010

To ensure the most reliable service, we are planning a maintenance window which may temporarily affect your services. Improvements made in a controllable and pro-active fashion will significantly reduce the risk of unplanned incidents. We appreciate your patience as we make these pro-active changes.

Window Start Date & Time (24h CST/CDT): Sunday Oct 24, 2010 00:01
Window End Date & Time (24h CST/CDT): Sunday Oct 24, 2010 05:00

Our Network Engineers will be upgrading the firmware and rebooting the routers in one of our Rack environments to improve reliability and scalability.
Servers and services downstream from the routers may experience an interruption in network traffic during the code upgrade process. This service interruption should be less than 10 minutes.

Note: Only the Dallas, TX Location is Affected. Miami, FL customers are not affected by this maintenance.

Miami-b07 reboot

We had to reboot miami-b07 as it hit a Xen bug. It has been rebooted and is back online. All Webbies are running.

Please send us a ticket if you have any issues with your Webby on b07.

Emergency miami-b13 maintenance

We are currently investigating a problem in miami-b13. More updates to follow.

Update: We had to perform a reboot on miami-b13. The server is stable now.

Update 1:08PM The server is not responding once again, we are working to stabilize it and move the affected users on to other nodes asap.