Webbynode System Status

We hit a Xen bug on on miami-b12 and had to perform a short maintenance  from 1700-1730 EST.

All Webbies are back up now.

Please contact us if you have any issues.

Your Webbynode Team

We are experiencing a DDoS attack and working to mitigate it at this time.

Update 3:36PM EST: We have mitigated the attack and blocked the attackers.

We’ve hit a Xen bug on Miami-a05, we are rebooting the server and all Webbies should come back shortly.

Update 8:38 AM EST: All Webbies are now up and running.

If you experience any further issues please send us a ticket.

We will be doing a scheduled maintenance September 1st between the hours of 11AM to 3PM EST. During this time, we may perform reboots to apply new kernel patches we are putting in place.

Affected Services: None.

Update 1:00PM EST: The scheduled maintenance has been completed. No nodes needed to be restarted.

3:25pm ET We are experiencing some connectivity issues in the Miami-A Network.

3:25pm ET We are bringing console down for about 30 minutes while we apply a fix.

4:20pm ET We finished testing and console is working smoothly.

CURRENT AFFECTED SERVICES: Webby Console.

WEBBIES ARE UP AND RUNNING.

1am EST: Performing an emergency reboot on mia-b03. Shutting down all Webbies then rebooting.

1:40am EST: mia-b03 coming back up now.

1:50am EST: mia-b03 is now 100% back online. Sorry for the unexpected downtime.

4AM EST: We are experiencing multiple hard drive failures, and issues. We are investigating on-site.

6AM EST: Array is rebuilding, and doing a long checksum process. Webbies remain offline. This process may take a few hours.

7AM EST: We’re still troubleshooting this issue, waiting for the array processes to finish.

9:50AM EST: The array rebuild is at 78%.

2:30PM EST: The array rebuild has halted and we are now copying all Webbies off of mia-a06 to perform recoveries. More information will be posted here as we know more.

4:00PM EST: We are still copying Webbies off of mia-a06. We are trying to make backups of all Webbies before we attempt any risky RAID array recovery procedures.

4:30PM EST: We are 1/3 of the way through backing up all Webbies on mia-a06. Due to the varying size of Webbies we can’t say for sure how long the last 2/3′s is going to take. More to follow.

7:00PM EST: Webbies are coming back up one at a time as we restore them, emails are going out as Webbies are up.

1:00AM EST: We have been able to recover all the data from miami-a06 and moved Webbies to other nodes. We are still missing a few Webbies for the raw data to be recovered. We are now waiting for the process to finish in order to deploy those Webbies without any data loss.

9:00AM EST: After a long process of recovering raw data from Webbies, we have successfully recovered every Webby and moved it to other nodes. Everyone has been emailed and notified.

We thank everyone with the patience and understanding on this long and extensive process. We could have gotten Webbies back much faster while giving up on the data, but we decided going down to the block level and restore the data completely.

Please contact support if you have any further issues.

4AM EST: We are experiencing multiple hard drive failures, and issues. We are investigating on-site.

5AM EST: We had to replace 2 faulty drives in miami-a07, also had to reboot it. All Webbies are back online.

Please contact support if you’re having any issues and your Webby is on miami-a07.