Hardware Failure miami-a06 [in progress]

4AM EST: We are experiencing multiple hard drive failures, and issues. We are investigating on-site.

6AM EST: Array is rebuilding, and doing a long checksum process. Webbies remain offline. This process may take a few hours.

7AM EST: We’re still troubleshooting this issue, waiting for the array processes to finish.

9:50AM EST: The array rebuild is at 78%.

2:30PM EST: The array rebuild has halted and we are now copying all Webbies off of mia-a06 to perform recoveries. More information will be posted here as we know more.

4:00PM EST: We are still copying Webbies off of mia-a06. We are trying to make backups of all Webbies before we attempt any risky RAID array recovery procedures.

4:30PM EST: We are 1/3 of the way through backing up all Webbies on mia-a06. Due to the varying size of Webbies we can’t say for sure how long the last 2/3′s is going to take. More to follow.

7:00PM EST: Webbies are coming back up one at a time as we restore them, emails are going out as Webbies are up.

1:00AM EST: We have been able to recover all the data from miami-a06 and moved Webbies to other nodes. We are still missing a few Webbies for the raw data to be recovered. We are now waiting for the process to finish in order to deploy those Webbies without any data loss.

9:00AM EST: After a long process of recovering raw data from Webbies, we have successfully recovered every Webby and moved it to other nodes. Everyone has been emailed and notified.

We thank everyone with the patience and understanding on this long and extensive process. We could have gotten Webbies back much faster while giving up on the data, but we decided going down to the block level and restore the data completely.

Please contact support if you have any further issues.