Elevated rate of 5xx in gcp-us-central1
major- Started
- Oct 8, 1:45 PM EDT
- Resolved
- Oct 8, 9:11 PM EDT
- Duration
- 7h 27m
turbopuffer's updates (4)
- investigating · Oct 8, 1:45 PM EDT
Some namespaces are experiencing an elevated rate of 5xx errors. We have identified the issue and are working to restore full service.
- identified · Oct 8, 2:27 PM EDT
Some write operations are intermittently failing due to an ongoing incident with [Google Cloud Storage](https://status.cloud.google.com/incidents/mSFVt6PiP557bjRyYsut).
- monitoring · Oct 8, 2:50 PM EDT
Error rates have recovered as Google rolls out a fix to Google Cloud Storage. We will continue monitoring until the upstream incident is resolved.
- resolved · Oct 8, 9:11 PM EDT
The issue upstream in Google Cloud Storage is resolved and us-central1 operations have returned to normal.
What our checks recorded
We checked turbopuffer 90 times from turbopuffer's start time to its resolution, once every five minutes. Every one of them reached turbopuffer's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 105 ms, against 84 ms over the 24 hours before. turbopuffer posted the incident at Oct 8, 1:45 PM EDT.
Updates are turbopuffer's own words from its status page (status: resolved); we never add an inferred cause.