Investigating Issue with Intermittent Failures
resolved- Started
- Sep 14, 3:31 PM EDT
- Resolved
- Sep 15, 1:00 AM EDT
- Duration
- 9h 29m
Decagon's updates (6)
- update · Sep 14, 3:31 PM EDT
We have identified a likely cause for the intermittent failures in our admin portal and chat and voice conversations between 11:35am-11:40am PT, and are actively working to confirm it. There has been no further impact since the original window, and our engineering team continues to monitor closely while the investigation continues. We will post another update by 1:00 PM PT
- update · Sep 14, 4:01 PM EDT
We are still confirming the likely cause for the intermittent failures in our admin portal and chat and voice conversations between 11:35am-11:40am PT. We will post another update by 1:30 PM PT
- update · Sep 14, 4:28 PM EDT
We're continuing to investigate intermittent failures affecting the admin portal, chat, and voice conversations between 11:35–11:40am PT, as well as a brief secondary spike around 1:00pm PT. We've identified a likely contributing factor and are validating it now. We will post another update by 2 PM PT.
- update · Sep 14, 5:00 PM EDT
We're continuing to investigate the intermittent failures affecting the admin portal, chat, and voice conversations, and are actively working with our infrastructure partner to confirm and resolve the root cause. We'll share another update by 4 PM PT.
- update · Sep 14, 7:05 PM EDT
We have identified the root cause and are actively working on applying mitigations. We do not expect further customer impact at this time. We will resolve this incident once mitigations have finished rolling out. Next update by 10:00 PM PT.
- resolved · Sep 15, 1:00 AM EDT
We have identified the root cause of this issue and deployed a fix. Systems have stabilized and we are not seeing further impact to the Admin Dashboard, chat widget, or voice call initiation. All systems are operating normally, and we are continuing to closely monitor.
What our checks recorded
We checked Decagon 113 times from Decagon's start time to its resolution, once every five minutes. Every one of them reached Decagon's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 150 ms, against 157 ms over the 24 hours before. Decagon posted the incident at Sep 14, 3:31 PM EDT.
Updates are Decagon's own words from its status page (status: resolved); we never add an inferred cause.