Investigating elevated latency and some voice calls failing to connect
resolved- Started
- Nov 16, 12:32 PM EST
- Resolved
- Nov 16, 12:46 PM EST
- Duration
- 14 min
Decagon's updates (3)
- update · Nov 16, 12:32 PM EST
Starting at 9:32pm PT, we started seeing elevated latency on some endpoints and some voice calls fail to connect in the US. We are seeing signs of recovery as of 9:51pm PT, and will update with a more detailed impact sizing once available.
- resolved · Nov 16, 1:37 PM EST
Systems have remained stable since 9:51 AM PT. Our team continues to monitor chat and voice performance in the US region and will provide another update within 30 minutes.
- resolved · Nov 16, 2:32 PM EST
The issue affecting chat and voice in the US region has been resolved. Impact Window: 9:26 AM – 9:48 AM PT Cause: A configuration change caused degraded performance in one of our databases. The change has been reverted and additional safeguards applied. We’ve applied a fix to prevent recurrence and will continue to monitor. We’ll provide a preliminary post-incident review upon request within 48hrs. Thank you for your patience and understanding.
What our checks recorded
We checked Decagon 3 times from Decagon's start time to its resolution, once every five minutes. Every one of them reached Decagon's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 338 ms, against 150 ms over the 24 hours before. Decagon posted the incident at Nov 16, 12:26 PM EST.
Updates are Decagon's own words from its status page (status: resolved); we never add an inferred cause.