W
Outage recap · WellSaid Labs

WellSaid Labs outage — September 18, 2024

On September 18, 2024, WellSaid Labs had a reported incident on WellSaid Labs's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Studio Outage

major
Started
Sep 18, 7:00 PM EDT
Resolved
Sep 18, 7:20 PM EDT
Duration
21 min

WellSaid Labs's updates (4)

  1. identified · Sep 18, 7:00 PM EDT

    The issue has been identified and a fix is being implemented.

  2. monitoring · Sep 18, 7:15 PM EDT

    A fix has been implemented and we are monitoring the results.

  3. resolved · Sep 18, 7:20 PM EDT

    This incident has been resolved.

  4. postmortem · Sep 18, 8:12 PM EDT

    # Summary WellSaid Labs Studio experienced an outage due to a traffic routing policy change made at the load balancer level. The change, intended to reduce the latency of clip generation and downloading, configured the routing rules for the Studio API and Studio web to services that failed to be updated leading to the load balancer being unable to route incoming requests to the backend services. # Timeline * 2024-09-18 22:57 UTC : Infrastructure change begins to roll out * 2024-09-18 23:00 UTC : Routing rules for studio production services are updated * 2024-09-18 23:00 UTC : Studio page and API requests begin failing * 2024-09-18 23:05 UTC : Outage detected by automated systems and infrastructure team is alerted * 2024-09-18 23:06 UTC : Cause of outage identified * 2024-09-18 23:10 UTC : Fix identified * 2024-09-18 23:15 UTC : Changes to fix routing rules begin to roll out * 2024-09-18 23:18 UTC : Routing changes applied to production * 2024-09-18 23:18 UTC : Services begin to spin back up to handle traffic * 2024-09-18 23:19 UTC : Infrastructure team is able to access studio web and call the studio API * 2024-09-18 23:20 UTC : System reports healthy status # Impact All users attempting to load pages within the WellSaid Labs studio would have experienced failing requests. Those attempting to generate clips would have been unaffected. # Resolution The infrastructure team removed and recrated the failing backend services using the automated deployment pipeline allowing the routing layer to properly reach them. # Follow-Up ## Next steps The infrastructure team is working on changes to ensure alignment between the different layers involved in routing requests to the studio services and tightening the dependency links between them such that the new backends must exist before traffic is attempted to be routed to them.

Updates are WellSaid Labs's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with WellSaid Labs. This recap summarizes incidents published on WellSaid Labs's official status page.