Scheduled provider maintenance: delayed GPU node provisioning in US East (Crusoe) and US South Central (Crusoe
resolved- Started
- Sep 10, 11:00 AM EDT
- Resolved
- Sep 10, 11:30 AM EDT
- Duration
- 30 min
Cerebrium's updates (2)
- update · Sep 10, 11:00 AM EDT
Our infrastructure provider Crusoe is performing maintenance on the Kubernetes control plane for the clusters behind our US East and US South Central Crusoe regions. What to expect: - Running apps and live traffic are not affected. - Scaling onto GPUs that are already provisioned continues to work normally. - Requests to provision new GPU nodes may fail for 5 to 10 seconds at a time and then retry. If your app needs to scale beyond the GPUs currently available in the region during this window, new replicas may take a few minutes longer than usual to become ready and requests may queue briefly. - Apps with cross-region enabled will spill over automatically as normal. What you can do: - Avoid deploying new versions of your apps in this window if you can. A deploy runs old and new replicas side by side and is the most common reason an app needs extra nodes. - If you have a known traffic peak in this window, consider raising your minimum replicas or scaling buffer from about 16:15 UTC so capacity is warm before the maintenance starts, and lowering it again afterwards.
- update · Sep 10, 11:30 AM EDT
Maintenance ended
What our checks recorded
We checked Cerebrium 6 times from Cerebrium's start time to its resolution, once every five minutes. Every one of them reached Cerebrium's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 223 ms, against 188 ms over the 24 hours before. Cerebrium posted the incident at Sep 10, 11:00 AM EDT.
Updates are Cerebrium's own words from its status page (status: resolved); we never add an inferred cause.