Elevated webhook delivery latency
minor- Started
- Sep 24, 1:37 PM EDT
- Resolved
- Sep 24, 2:21 PM EDT
- Duration
- 44 min
Mux's updates (4)
- investigating · Sep 24, 1:37 PM EDT
We are currently investigating elevated webhook delivery times. Beginning at 16:40 UTC, some webhooks are being delivered with a delay of approximately 5 minutes. Video processing, playback, and API requests are not affected, and no webhook events are being dropped — delayed events will still be delivered. We will provide an update as soon as we have more information.
- identified · Sep 24, 1:44 PM EDT
We have identified the cause of the elevated webhook delivery times. An issue in one of our internal database clusters introduced a delay in the pipeline that queues webhook events, resulting in delivery delays beginning at 16:40 UTC. Video processing, playback, and API requests are unaffected, and no events have been lost — delayed webhooks are still being delivered. Delivery times are beginning to recover.
- monitoring · Sep 24, 1:51 PM EDT
Webhook delivery times have returned to normal. Between 16:40 UTC and 17:36 UTC, webhook events were delivered with delays averaging around 5 minutes, with some events delayed up to 15 minutes. Video processing, playback, and API requests were unaffected, and no events were lost — all delayed webhooks have been delivered.We have addressed the underlying issue in one of our internal database clusters and are monitoring to confirm.
- resolved · Sep 24, 2:21 PM EDT
This incident has been resolved. Between 16:40 UTC and 17:36 UTC, webhook events were delivered with delays averaging around 5 minutes, with some events delayed up to 15 minutes. Video processing, playback, and API requests were unaffected, and no events were lost — all delayed webhooks have been delivered.The underlying issue in one of our internal database clusters has been addressed, and delivery times have been stable since 17:36 UTC. We apologize for the disruption.
What our checks recorded
We checked Mux 9 times from Mux's start time to its resolution, once every five minutes. Every one of them reached Mux's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 190 ms, against 208 ms over the 24 hours before. Mux posted the incident at Sep 24, 1:37 PM EDT.
Updates are Mux's own words from its status page (status: resolved); we never add an inferred cause. The vendor's incident page ↗