Slow trace data retirival across API and UI in EU
minor- Started
- Oct 21, 3:46 AM EDT
- Resolved
- Oct 21, 4:45 AM EDT
- Duration
- 59 min
Langfuse's updates (5)
- investigating · Oct 21, 3:46 AM EDT
Slow traces data serving across API and UI
- identified · Oct 21, 4:31 AM EDT
Slow clickhouse read replicas affecting performance across the service. Draining and replacing them.
- resolved · Oct 21, 4:45 AM EDT
All replicas look healthy again
- monitoring · Oct 21, 4:56 AM EDT
Looks like some timeout are still occurring. Monitoring.
- resolved · Oct 21, 5:39 AM EDT
should be finally resolved
What our checks recorded
We checked Langfuse 12 times from Langfuse's start time to its resolution, once every five minutes. Every one of them reached Langfuse's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 151 ms, against 141 ms over the 24 hours before. Langfuse posted the incident at Oct 21, 3:46 AM EDT.
Updates are Langfuse's own words from its status page (status: resolved); we never add an inferred cause.