Fully Open Edge Cloud

[FIXED] Incident: Jul, 21st 2022 12:00 GMT: Canary cluster processing is underperforming

Rapid.Space Maintenance. Read the article for real time information.
  • Last Update:2022-08-01
  • Version:001
  • Language:en

01/08/2022 12:00 GMT: Canary cluster main node migration finished

The cluster migration has been finished successfully and it's operational in the new location.

Current slowness issues are solved, eventual changes in the infrastructure will be published in separate status notifications.

01/08/2022 10:00 GMT: Canary cluster main node migration

In order to achieve proper scalability of main Canary node it will be migrated today.

In order to achieve proper scalability of main Canary node it will be migrated today.

During the migration such services will be disrupted on Canary:

  • allocation of new instances
  • updates of instance

It will not impact currently configured instances serving content.

We will inform when the migration will be over.

21/07/2022 14:10 GMT: Canary cluster processing performance is stabilized - confirmation

While checking the logs we confirmed that the speed of processing frontend allocation and updates is acceptable.

We are planning to fully stabilize this situation next week, as soon as dedicated Compute Node will be delivered for this service.

21/07/2022 12:30 GMT: Canary cluster processing performance is stabilized

We dropped not critical debugging information while processing the Canary cluster, so the processing came back to normal. It's only short-term solution, which will allow us to find time for a better location for the root node of Canary cluster.

We will post here information about eventual changes.

21/07/2022 11:00 GMT: Canary cluster processing is underperforming

After maintenance on the Canary cluster we learnt that the machine is not performing correctly in the new location.

Currently there are problems with:

  • frontend allocation
  • frontend updates

There are no problems with serving currently configured frontends.