[Resolved] Dallas-02 Network Issue

We’ve identified a network issue on Dallas-02. We are investigating the cause and working to fix asap. Please stand by.

We had an issue with our internal xen network bridges. Issue has been fixed and we’re back.

Note: All Webbies remained running.

Miami-b11 Outage

We are experiencing an outage on Miami-b11. We are having critical issues with a storage array. Please hold while we work on bringing this server back.

Update: Issue is still being worked on. We will post more details soon.

Miami Network Downtime

We are still working to find the cause of the brief network outage in Miami. The onsite admins have restored connectivity. There was no downtime for webbies, only a loss of network connectivity.

UPDATE We thought things had been resolved, but the onsite admins are still working on the outage, bear with us, we’ll have things back online shortly.

UPDATE Our onsite admins have reported that we are 100% back. Outage has been resolved. Please open a ticket if you need any further assistance.

[Completed] Miami-b07 Maintenance

We are currently performing emergency maintenance on Miami-b07. We are checking the file system integrity to ensure no data loss.

More updates to follow.

UPDATE All maintenance is finished. If anyone has any problems, please submit a ticket.

[Completed] Wash-07

We are currently performing more patch and security updates on nodes in the Wash DC data center. Wash-07 is being rebooted to apply patches. Updates to follow.

UPDATE 17:15 EST Wash-07 is coming back online and all webbies are now booting back up. If anyone sees any issues once all webbies are online, please submit a support ticket and we’ll handle any issues.

[Completed] Wash-06 Reboot

We are currently performing more patch and security updates on nodes in the Wash DC data center. Wash-06 is being rebooted to apply patches. Updates to follow.

UPDATE 13:20 EST Wash-06 is back online and all webbies are coming back up.

[Completed] Wash-01 && Wash-04 Reboot

We are rebooting Wash-01 && Wash-04 to remediate some needed updates and system patches. There will only be a brief downtime. We’ll update here once finished.

UPDATE 16:45 EST Webbies are now coming back online on Wash-01

UPDATE 17:20 EST Webbies on Wash-01 should all be back online. Wash-04 is coming back up now.

UPDATE 17:27 EST All webbies on Wash-04 should now be back online.

If anyone is seeing any problems, please submit a ticket.

[Resolved] Wash-04 Connectivity Issues

We are currently checking on a connectivity issue to Wash-04. We’ll update once we know more.

UPDATE 14:20 EST We’ve identified a NIC port caused the system to lock up and we initiated a reboot to bring things back online. We are investigating if the problem is related to the switch port the node is connected to or the NIC card.

UPDATE 14:35 EST All webbies are coming back online, please open a ticket if you are still experiencing any problems.

[Resolved] DoS Attack Miami

We have mitigated a DoS Attack that was affecting network connectivity in the Miami datacenter. DoS Attack has been mitigated and all connectivity is back to 100%.

If you experience any further issues please get in touch with support.

[Resolved] Emergency Network Maintenance

Date: 06-02-2011
Start time: 11:20
Services Affected: Washington, DC – Public Network

Event Summary:

Webbynode Engineers along with Cisco TAC have identified a software bug on FCR01.WDC01 which is causing forwarding issues for public connectivity. Engineers will be performing an EMERGENCY code change on Jun 2nd, 2011 at 10:00 PM EDT to resolve this issue. The expected downtime is 20 minutes with the maintenance window being scheduled for up to 4 hours.

Start Time: 10:00pm EDT (6/2/2011)
End Time: 02:00am EDT (6/3/2011)
Expected Duration: 20 minutes

Customer Impact:

During this maintenance, customers will notice a complete loss of connectivity to their servers on the frontend network (public network). Backend network (private network) connectivity will NOT be impacted during this maintenance. While the upgrade duration is scheduled for 4 hours, we only expect around 20 minutes of downtime as the code is changed. Again, this will NOT impact the backend network (private network) for customer servers.

Best,

Webbynode

UPDATE 11:00PM EST: Connectivity has been restored at this point, but the maintenance window is still going. Please open a ticket if you still have issues.