Resolved -
Our vendor, Leaseweb, has replaced the failing switch and things should be normalizing now. We will provide a full RCA once Leaseweb makes it available.
We are also prioritizing moving these servers into our own data centers to avoid such unacceptable resolution times in the future.
Oct 29, 15:57 EDT
Update -
Leaseweb is replacing the failed switch and rebooting the affected nodes. Connectivity should return shortly.
Oct 29, 15:50 EDT
Update -
Leaseweb's data center engineer and the EOD team are actively working on the issue. We expect this to be resolved very soon.
Oct 29, 13:54 EDT
Identified -
Leaseweb, our vendor, has isolated this to a bad rack affecting a few nodes and they're working on restoring the network connectivity to resolve the packet loss. We expect this to be resolved very soon.
Oct 29, 13:05 EDT
Update -
We are continuing to investigate this issue.
Oct 29, 12:39 EDT
Investigating -
We are investing an issue in London affecting two nodes and working with our vendor to correct it as quickly as possible.
We apologize for any inconvenience this may cause.
Oct 29, 12:38 EDT