Webbynode System Status

[RESOLVED] Connectivity problems on 67.23.79.* IP range

fcourywebbynode — Mon, 12 Jul 2010 04:39:27 +0000

[0:30AM] We’re having connectivity issues with Webbies in the 67.23.79.* range. We have all hands on deck working to fix the problem.

[0:45AM] It was a Level3 Connectivity problem that is now resolved.

All Webbies are up and running.

Level3 Connectivity

fcourywebbynode — Fri, 18 Jun 2010 23:53:27 +0000

[07:30PM EST] There was some packet loss at Atlanta’s Level 3 router. A ticket was opened and the problem is now solved. All Webbies remained up during the network outage.

DC Power Supply problem

fcourywebbynode — Thu, 10 Jun 2010 17:58:25 +0000

[1PM EST] Our DC had an UPS overload and that caused a reboot of nodes miami-b01, b02, b03, b12, b13, b15 and b16. All webbies on those nodes were rebooted. Approximate downtime was 7 minutes and all Webbies are back up.

Please open a ticket if you are still experiencing problems.

Packet Loss

cartab — Mon, 05 Apr 2010 03:50:18 +0000

[11:50 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 12:03AM Issue has been resolved.

Level3 Connectivity

cartab — Wed, 03 Mar 2010 04:42:00 +0000

[11:40 PM EST] We are working with our network provider level 3 to solve a routing issue that has caused some packet loss for traffic going through this route.

Update: 11:59 PM EST Issue has been solved, we’re back to optimal level3 performance.

[Completed] miami-b12 Maintenance

cartab — Wed, 17 Feb 2010 18:30:20 +0000

We had a raid failure on miami-b12. The server is up and running, however the raid is running a full rebuild, so the disk access might be slower than usual.

We will keep monitoring the progress, and update this post.

Note: All Webbies are up.

Update 11PM EST: The rebuild has fully finished with no problems.

RESOLVED – GitHub Integration Unavailability

fcourywebbynode — Wed, 03 Feb 2010 14:20:28 +0000

We are suffering from an issue where GitHub’s API is timing out. This is affecting any deployments that uses GitHub integration.

We’re actively working on this problem however we don’t have an ETA at this time.

Thanks for your patience!

Update 9:30AM EST: It seems it was a hiccup on GitHub’s end. Integration is available again.

[Completed] miami-b14 Xen bug

cartab — Mon, 25 Jan 2010 17:10:00 +0000

[12:05 PM EST] We hit a Xen bug with miami-b14, we will update this post as soon as its finished.

Update 12:08 PM We had to reboot miami-b14 and Webbies are coming back online.

Please contact support if you have any issues.

[Completed] miami-b04 Xen bug

cartab — Mon, 25 Jan 2010 14:17:49 +0000

[9:17PM EST] We hit a Xen bug with miami-b04, we will update this post as soon as its finished.

Update 9:22AM We had to reboot miami-b04 and Webbies are coming back online.

Please contact support if you have any issues.

[Completed] miami-b11 Maintenance

fcourywebbynode — Sat, 23 Jan 2010 23:45:07 +0000

We had a raid failure on miami-b11. We had to turn off the server for a hardware maintenance, and it is now complete.

The server is back up and Webbies are coming back online.

If you have any issues please contact support.

UPDATE 8:20PM :

We continue to have issues with miami-b11 we are now working on migrating everyone off miami-b11. We’ll update this shortly.

UPDATE 01/24 0:20AM :

All Webbies are up but we have disabled rebooting, stopping, starting and redeploying Webbies in the meantime. Once you’re migrated to another node, you’ll be ready to manage your Webby as usual.

Update 01/24 8:00PM :

We have finished migrating all Webbies to a new node. We managed to rescue all data, no data loss was suffered.