Fully Open Edge Cloud

[FINISHED] Incident: Jan, 9th, 2023 05:00 GMT: Premium.CDN, Canary.CDN and Chinary.CDN problems with network connectivity

Rapid.Space Maintenance. Read the article for real time information.
  • Last Update:2024-01-10
  • Version:001
  • Language:en

10/1/2024 12:15 GMT: Problem fixed

Finally, we found that the problem was linked to a faulty node in the new re6st production network. This faulty node was creating network issues on all nodes of the new re6st network, especially 2 frontend nodes (the 2 nodes removed from the cluster) and some backend nodes. For the impacted backend nodes, all the frontends could have problem to reach them.

Removing the faulty node from the network made the problem disappeared right away. We will now take the time to improve re6st network so that one faulty node cannot impact all other nodes. We will also improve the reporting of such issue to detect more easily issues like this.

 

10/1/2024 12:00 GMT: All CDNs nodes have connectivity issues

We found out that all nodes are impacted by the connectivity issues in all clusters.

The Rapid.Space team is debugging the situation.

10/1/2024 10:15 GMT: Premium.CDN faulty node in Europe and Japan removed from the cluster

With further analysis we found out that a problem occurred also on one of Japanese nodes, which has been removed from the cluster.

The Rapid.Space team is debugging the situation.

10/1/2024 08:30 GMT: Premium.CDN faulty node in Europe removed from the cluster

The faulty node has been removed from the cluster. We are continuing the investigation.

9/1/2024 05:00 GMT: Premium.CDN faulty node in Europe

One of the nodes in Europe have problems with the connectivity.