L
Outage recap · Labelbox

Labelbox outage — October 20, 2023

On October 20, 2023, Labelbox had a reported incident on Labelbox's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Labelbox platform temporarily unavailable to login

critical
Started
Oct 20, 7:01 PM EDT
Resolved
Oct 20, 8:54 PM EDT
Duration
1h 53m

Labelbox's updates (7)

  1. investigating · Oct 20, 7:01 PM EDT

    We are currently investigating this issue.

  2. identified · Oct 20, 7:05 PM EDT

    We've identified an issue with our CD system and are working on a fix.

  3. identified · Oct 20, 7:25 PM EDT

    A fix has been made and services are returning to a normal state. This will take some time before the platform is fully available.

  4. monitoring · Oct 20, 8:24 PM EDT

    A fix has been implemented and we are monitoring the results.

  5. monitoring · Oct 20, 8:31 PM EDT

    We are continuing to monitor for any further issues.

  6. resolved · Oct 20, 8:54 PM EDT

    This incident has been resolved. We are continuing to monitor for any further issues.

  7. postmortem · Oct 24, 4:06 PM EDT

    **Summary** An issue in our automated deployment tooling caused Labelbox services to become unavailable. The malfunction did not affect any data storage, database, network, security, or other infrastructure. A change to the automated deployment tooling was reverted and services were automatically restored.  **Details** A configuration change was made Friday, October 20th at approximately 15:45 PT \(22:45 UTC\) that was believed to be benign. However, when this change propagated through the configuration system it had unintended consequences. This resulted in Labelbox services becoming unavailable for approximately one hour. **The malfunction did not affect any data storage, database, network, security, or other infrastructure, and no data was lost or affected by the outage.** The change to the automated deployment tooling was reverted and services were automatically restored. The restoration of the services took some time to stabilize. The incident was deemed fully resolved at 16:56 PT \(23:56 UTC\) ‌ Labelbox has already completed several changes to prevent this type of issue from being able to occur again.

Updates are Labelbox's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with Labelbox. This recap summarizes incidents published on Labelbox's official status page.