Webbynode System Status

[0:30AM] We’re having connectivity issues with Webbies in the 67.23.79.* range. We have all hands on deck working to fix the problem.

[0:45AM] It was a Level3 Connectivity problem that is now resolved.

All Webbies are up and running.

[07:30PM EST] There was some packet loss at Atlanta’s Level 3 router. A ticket was opened and the problem is now solved. All Webbies remained up during the network outage.

[1PM EST] Our DC had an UPS overload and that caused a reboot of nodes miami-b01, b02, b03, b12, b13, b15 and b16. All webbies on those nodes were rebooted. Approximate downtime was 7 minutes and all Webbies are back up.

Please open a ticket if you are still experiencing problems.

[11:50 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 12:03AM Issue has been resolved.

[11:40 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 11:59 PM EST Issue has been solved, we’re back to optimal level3 performance.

We had a raid failure on miami-b12. The server is up and running, however the raid is running a full rebuild, so the disk access might be slower than usual.

We will keep monitoring the progress, and update this post.

Note: All Webbies are up.

Update 11PM EST: The rebuild has fully finished with no problems.

We are suffering from an issue where GitHub’s API is timing out. This is affecting any deployments that uses GitHub integration.

We’re actively working on this problem however we don’t have an ETA at this time.

Thanks for your patience!

Update 9:30AM EST: It seems it was a hiccup on GitHub’s end. Integration is available again.

[9:17PM EST] We hit a Xen bug with miami-b04, we will update this post as soon as its finished.

Update 9:22AM We had to reboot miami-b04 and Webbies are coming back online.

Please contact support if you have any issues.

We had a raid failure on miami-b11. We had to turn off the server for a hardware maintenance, and it is now complete.

The server is back up and Webbies are coming back online.

If you have any issues please contact support.

UPDATE 8:20PM :

We continue to have issues with miami-b11 we are now working on migrating everyone off miami-b11. We’ll update this shortly.

UPDATE 01/24 0:20AM :

All Webbies are up but we have disabled rebooting, stopping, starting and redeploying Webbies in the meantime. Once you’re migrated to another node, you’ll be ready to manage your Webby as usual.

Update 01/24 8:00PM :

We have finished migrating all Webbies to a new node. We managed to rescue all data, no data loss was suffered.