[Resolved] Dallas-02 Network Issue
Posted: October 11, 2011 Filed under: Uncategorized Leave a comment »We’ve identified a network issue on Dallas-02. We are investigating the cause and working to fix asap. Please stand by.
We had an issue with our internal xen network bridges. Issue has been fixed and we’re back.
Note: All Webbies remained running.
Miami-b11 Outage
Posted: September 29, 2011 Filed under: Uncategorized Leave a comment »We are experiencing an outage on Miami-b11. We are having critical issues with a storage array. Please hold while we work on bringing this server back.
Update: Issue is still being worked on. We will post more details soon.
[Resolved] DoS Attack Miami
Posted: June 4, 2011 Filed under: Uncategorized Leave a comment »We have mitigated a DoS Attack that was affecting network connectivity in the Miami datacenter. DoS Attack has been mitigated and all connectivity is back to 100%.
If you experience any further issues please get in touch with support.
[Resolved] Emergency Network Maintenance
Posted: June 2, 2011 Filed under: Uncategorized Leave a comment »Date: 06-02-2011
Start time: 11:20
Services Affected: Washington, DC – Public Network
Event Summary:
Webbynode Engineers along with Cisco TAC have identified a software bug on FCR01.WDC01 which is causing forwarding issues for public connectivity. Engineers will be performing an EMERGENCY code change on Jun 2nd, 2011 at 10:00 PM EDT to resolve this issue. The expected downtime is 20 minutes with the maintenance window being scheduled for up to 4 hours.
Start Time: 10:00pm EDT (6/2/2011)
End Time: 02:00am EDT (6/3/2011)
Expected Duration: 20 minutes
Customer Impact:
During this maintenance, customers will notice a complete loss of connectivity to their servers on the frontend network (public network). Backend network (private network) connectivity will NOT be impacted during this maintenance. While the upgrade duration is scheduled for 4 hours, we only expect around 20 minutes of downtime as the code is changed. Again, this will NOT impact the backend network (private network) for customer servers.
Best,
Webbynode
UPDATE 11:00PM EST: Connectivity has been restored at this point, but the maintenance window is still going. Please open a ticket if you still have issues.
[Finished] Emergency maintenance on Wash-04
Posted: May 10, 2011 Filed under: System Status Leave a comment »We are experiencing an issue in Wash-04, the server will be coming back shortly. Updates will come as more information is available.
Update 10:55AM – Wash-04 is back online. All Webbies are coming back up. We hit a Xen kernel bug and had to restart the server.
[Resolved] DoS Attack Miami-B
Posted: May 7, 2011 Filed under: Uncategorized Leave a comment »We are working on mitigating an incoming attack in the Miami-B datacenter. Please wait for more updates as information becomes available.
Update: 2:30 PM: Attack has been mitigated, we’re back online. All Webbies are online. If you are having any issues please send a ticket.
[Resolved] Miami-B Datacenter Connectivity Issues
Posted: May 5, 2011 Filed under: System Status 5 Comments »We are experiencing a network issue in our Miami-DC. We are having intermittent network connectivity and packet loss.
We are working on it right now, we will update this post as soon as more information is available.
UPDATE 6:15PM EST: We are still having issues with connectivity, we have all hands on deck working on the problem. More updates following.
UPDATE 6:30PM EST: We are still working on this issue. Please hang on tight. We are working as fast as we can to get this resolved.
UPDATE 7:00PM EST: The network is now up. All Webbies are also up, and did not go down. We are are getting more information in detail so we can update this ticket.
UPDATE 7:55PM EST: The initial report is that we experienced an massive inbound DoS service attack, which seems to have been related to our primary uplink provider Cogent as a massive packet flood to our network. During this attack our core Cisco routers also locked up due to an un-fixed module bug by Cisco and prevented our main uplink to failover to our secondary which is Level3. After we changed everything to Level3 we came back up again which makes us believe that the packet flood was an issue created directly by Cogent and caused a Denial of Service to us. We are now on Level3 as a we wait for a report from Cogent and we will update this blog post when that information becomes available.
[Resolved] Washington-03 Maintenance
Posted: May 5, 2011 Filed under: System Status Leave a comment »As of 12:30AM EST we are investigating some server lockups that have cause the intermittent uptime. At this time we are investigating whether we had a RAID controller issue and we had to reboot the server. As of this time the server is up and running and operational, however we are now doing more investigations to solve the kernel panics we are receiving. If the fix doesn’t solve the problem we will be swapping the ram on the server even though the server reports no RAM errors.
If the server continues to give us more issues, we will then schedule a migration of every single customer to a new server.
Note: As of 12:30 AM EST All Webbies are up and running. We will update the Status post as we perform any reboots or changes.
Update: 3PM EST – We will be scheduling a memory swap on this server shortly. We will post the schedule here.
[Resolved] Wash-03 Maintenance
Posted: April 28, 2011 Filed under: Uncategorized Leave a comment »Hello,
We experienced a hardware issue on our raid on Wash-03, the server is up and running now. However we are investigating the matter closely, we will be updating this post as more information becomes available.
[Resolved] Emergency Maintenance miami-b14
Posted: April 11, 2011 Filed under: Uncategorized 3 Comments »4:03AM EST – We’re investigating an issue affecting users in miami-b14. Please hang on while we find an issue, we’ll update this blog post asap.
6:00 AM EST – We had a critical raid failure. We are on recovery mode and Webbies in miami-b14 are currently down. We are migrating each Webby’s data so that and starting them on other nodes.
7:30 AM EST – Webbies still being migrated.
4:00 PM EST - The majority of Webbies have been migrated. We have 5-6 Webbies till being migrated manually in order to ensure data integrity.
7:00 PM EST – The majority of Webbies were recovered yesterday. We had to do hard data recovery for the last few ones. We are working on the last 2 at this point.
4:00 AM – EST – All Webbies have been recovered. If your Webby has any issues still please contact support.
