Degraded performance on some indexes
resolved- Started
- Oct 23, 12:15 PM EDT
- Resolved
- Oct 23, 2:54 PM EDT
- Duration
- 2h 39m
Marqo's updates (5)
- update · Oct 23, 12:15 PM EDT
Our alerting system has indicated that some indexes are experiencing elevated tail latencies and degraded success rates. Early indication seems that this is related to degradation from a service provider.
- update · Oct 23, 12:30 PM EDT
We are seeing error rates and latencies reduce. This corresponds to fixes reported by the service provider.
- update · Oct 23, 1:02 PM EDT
We're seeing latencies and error rates increase again. We're continuing to investigate.
- update · Oct 23, 1:25 PM EDT
Our observability systems indicate that service has returned to normal levels. However, we are yet to receive confirmation from the affected provider.
- resolved · Oct 23, 2:54 PM EDT
The affected provider has confirmed that the incident has been resolved.
What our checks recorded
We checked Marqo 32 times from Marqo's start time to its resolution, once every five minutes. Every one of them reached Marqo's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 130 ms, against 107 ms over the 24 hours before. Marqo posted the incident at Oct 23, 12:15 PM EDT.
Updates are Marqo's own words from its status page (status: resolved); we never add an inferred cause.