Labelbox platform temporarily unavailable to login
critical- Started
- Oct 20, 7:01 PM EDT
- Resolved
- Oct 20, 8:54 PM EDT
- Duration
- 1h 53m
Labelbox's updates (7)
- investigating · Oct 20, 7:01 PM EDT
We are currently investigating this issue.
- identified · Oct 20, 7:05 PM EDT
We've identified an issue with our CD system and are working on a fix.
- identified · Oct 20, 7:25 PM EDT
A fix has been made and services are returning to a normal state. This will take some time before the platform is fully available.
- monitoring · Oct 20, 8:24 PM EDT
A fix has been implemented and we are monitoring the results.
- monitoring · Oct 20, 8:31 PM EDT
We are continuing to monitor for any further issues.
- resolved · Oct 20, 8:54 PM EDT
This incident has been resolved. We are continuing to monitor for any further issues.
- postmortem · Oct 24, 4:06 PM EDT
**Summary** An issue in our automated deployment tooling caused Labelbox services to become unavailable. The malfunction did not affect any data storage, database, network, security, or other infrastructure. A change to the automated deployment tooling was reverted and services were automatically restored. **Details** A configuration change was made Friday, October 20th at approximately 15:45 PT \(22:45 UTC\) that was believed to be benign. However, when this change propagated through the configuration system it had unintended consequences. This resulted in Labelbox services becoming unavailable for approximately one hour. **The malfunction did not affect any data storage, database, network, security, or other infrastructure, and no data was lost or affected by the outage.** The change to the automated deployment tooling was reverted and services were automatically restored. The restoration of the services took some time to stabilize. The incident was deemed fully resolved at 16:56 PT \(23:56 UTC\) Labelbox has already completed several changes to prevent this type of issue from being able to occur again.
Updates are Labelbox's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗