W
Outage recap · WellSaid Labs

WellSaid Labs outage — November 9, 2024

On November 9, 2024, WellSaid Labs had a reported incident on WellSaid Labs's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Developer TTS API authorization failures

minor
Started
Nov 9, 8:41 PM EST
Resolved
Nov 9, 9:02 PM EST
Duration
21 min

WellSaid Labs's updates (4)

  1. identified · Nov 9, 8:41 PM EST

    Some TTS Developer API nodes have started performing requests to internal TTS service with incorrect api keys.

  2. monitoring · Nov 9, 8:44 PM EST

    Keys between the two services have been synced and we are seeing successful requests coming from the Developer API gateway. We are continuing to monitor to ensure all nodes reflect this change.

  3. resolved · Nov 9, 9:02 PM EST

    All Developer TTS API gateway nodes are properly communicating with internal TTS service.

  4. postmortem · Nov 9, 9:45 PM EST

    # Summary During a migration of our internal TTS service infrastructure, our external facing TTS Developer API experienced a partial outage due to authentication issues with the new TTS service. The root cause was identified as the use of invalid authentication keys from the Developer API which had not been updated to reflect those within the new internal TTS service. This resulted in a service degradation leading to 19 failed requests. # Timeline * 2024-11-10 01:39 UTC : Migration of internal TTS service to new infrastructure begins * 2024-11-10 01:41 UTC : Developer TTS API Gateway nodes begin to pick up new routing change * 2024-11-10 01:41 UTC : The nodes routing to the new TTS service begin to fail requests * 2024-11-10 01:42 UTC : Monitoring system shows failing requests between the two services * 2024-11-10 01:42 UTC : Routing logic changed back to existing infrastructure * 2024-11-10 01:43 UTC : Root cause of mismatching keys is identified * 2024-11-10 01:43 UTC : Monitoring system alert for failed requests to Developer TTS fires * 2024-11-10 01:44 UTC : Impact analysis begins * 2024-11-10 01:44 UTC : Developer TTS API Gateway successfully authenticates to existing internal TTS service * 2024-11-10 01:44 UTC : Existing authentication keys used by Developer TTS API Gateway are manually added to new TTS service * 2024-11-10 01:51 UTC : Verification of existing keys against new TTS infrastructure is performed * 2024-11-10 02:00 UTC : Impact analysis finds only Developer TTS API is affected * 2024-11-10 02:01 UTC : Routing logic changed back to new infrastructure * 2024-11-10 02:02 UTC : Developer TTS API Gateway begins to successfully authenticates to internal TTS service * 2024-11-10 02:05 UTC : Infrastructure team continues to monitor both services # Impact * Some users of the Developer TTS API hitting nodes connecting to the new TTS service would have experienced server errors while attempting to generate new clips, in total 19 requests over a 3 minute time frame failed # Action Items * A planned extension of this infrastructure migration is to implement automatic rotation and refreshing of API keys between the two services to remove the need for manual syncing

Updates are WellSaid Labs's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with WellSaid Labs. This recap summarizes incidents published on WellSaid Labs's official status page.