L
Outage recap · Labelbox

Labelbox outage — April 15, 2024

On April 15, 2024, Labelbox had a reported incident on Labelbox's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Streamable and non-streamable exports are experiencing increased latencies

minor
Started
Apr 15, 10:04 AM EDT
Resolved
Apr 15, 3:58 PM EDT
Duration
5h 53m

Labelbox's updates (4)

  1. identified · Apr 15, 10:04 AM EDT

    The issue has been identified and a fix is being implemented.

  2. monitoring · Apr 15, 12:10 PM EDT

    A fix has been implemented and we are monitoring the results.

  3. resolved · Apr 15, 3:58 PM EDT

    Exports latency is resolved. We'll be posting a post-mortem analysis of the incident within a week.

  4. postmortem · Apr 16, 6:26 PM EDT

    Dear Customers, We wanted to bring to your attention a recent latency issue with export functionality on the Labelbox platform. We’ll provide details on the cause and remediation as well as assurance on what actions we’re taking to prevent such incidents happening in the future. **Summary:** On April 15, 2024, around 14:00 UTC, we experienced latency issues with export functionality. An investigation was launched and the root cause was determined. Subsequently a fix was implemented and deployed within hours. There were some sporadic occurrences of login issues while the increased export latency mitigation was in progress. **Root Cause:** The root cause of the incident was due to high resource usage on the primary database and a resultant backlog in internal systems due to a sudden surge in exports, even as systems scaled to meet the increased load. **Resolution**: Upon identifying the root cause, a fix was implemented that directed export traffic to a replica thereby reducing load on the primary database. We have also updated the batch processing logic for exports to reduce the probability of overload. Systems were monitored to confirm problem mitigation. We have also identified and scheduled the following additional corrective actions to prevent the occurrence of these types of issues in the future. * Enhancing the concurrent message processing logic. * Improving the retry handling mechanism to prevent a backlog of unprocessed messages. We apologize for this inconvenience. We are here to make Labelbox the most reliable and responsive data-centric AI platform, and will provide support in any way we can for your continued use. ‌ Sincerely, The Labelbox team

Updates are Labelbox's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with Labelbox. This recap summarizes incidents published on Labelbox's official status page.