Mux Data Missing Some EU Views
major- Started
- Nov 12, 4:00 PM EST
- Resolved
- Nov 13, 1:17 AM EST
- Duration
- 9h 18m
Mux's updates (6)
- investigating · Nov 12, 4:00 PM EST
We are investigating a missing views issue affecting customers collection in the EU.
- investigating · Nov 12, 4:19 PM EST
We are seeing roughly half of all collected views from the EU are missing, and have been missing for several months. Realtime metrics and alerting are unaffected. We are continuing to investigate.
- identified · Nov 12, 5:08 PM EST
The issue has been identified and a fix is being implemented.
- identified · Nov 12, 5:52 PM EST
We are collecting and storing all views from the EU as of 22:16 UTC. We are evaluating if any previous missing views are recoverable.
- identified · Nov 12, 8:08 PM EST
We are currently re-processing the last 7 days of available views that were missed. We expect this step to take several hours.
- resolved · Nov 13, 1:17 AM EST
Backfill complete and this incident has been resolved. Any missing views older than 7 days are unrecoverable. Time of impact for this incident is June 17th-November 5th for EU collection customers. We estimate 1/3 of views during this window are missing for affected customers, with more views affected in later months. Real-time metrics and alerting remain unaffected. We apologize for this data loss and are taking corrective action on our systems to prevent recurrence.
What our checks recorded
We checked Mux 112 times from Mux's start time to its resolution, once every five minutes. Every one of them reached Mux's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 369 ms, against 356 ms over the 24 hours before. Mux posted the incident at Nov 12, 4:00 PM EST.
Updates are Mux's own words from its status page (status: resolved); we never add an inferred cause. The vendor's incident page ↗