Degraded ingestion for 20 projects
major- Started
- Sep 30, 5:10 AM EDT
- Resolved
- Sep 30, 11:30 AM EDT
- Duration
- 6h 20m
Langfuse's updates (3)
- identified · Sep 30, 5:10 AM EDT
S3 backpressure leads to S3 "please slow down" errors across projects with cm* prefix
- monitoring · Sep 30, 5:43 AM EDT
Offending workload was isolated, performance should be back to normal for other projects
- resolved · Sep 30, 11:30 AM EDT
All projects ingesting as normal
What our checks recorded
We checked Langfuse 77 times from Langfuse's start time to its resolution, once every five minutes. Every one of them reached Langfuse's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 204 ms, against 175 ms over the 24 hours before. Langfuse posted the incident at Sep 30, 5:10 AM EDT.
Updates are Langfuse's own words from its status page (status: resolved); we never add an inferred cause.