W
Outage recap · WellSaid Labs

WellSaid Labs outage — March 5, 2024

On March 5, 2024, WellSaid Labs had a reported incident on WellSaid Labs's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Increased latency & error rates for API customers

major
Started
Mar 5, 12:12 PM EST
Resolved
Mar 2, 5:00 PM EST
Duration
—

WellSaid Labs's updates (2)

  1. resolved · Mar 2, 5:00 PM EST

    Elevated latency and error rates impacting all API customers. Incident Start: 2024-03-02 13:48 PT Incident End: 2024-03-03 13:48 PT Incident Duration: 24hrs Impact: Roughly 20% of API Customer traffic experienced increased latency and/or error rates

  2. postmortem · Mar 5, 12:19 PM EST

    ## Incident Summary Elevated latency and error rates impacting all API customers. The errors were a result of a single gateway pod serving traffic in a faulty state after it failed to initialize a system component properly. ## Impact Roughly 20% of API customer traffic experienced increased latency and/or error rates for a period of 24 hours. ## Root Cause The source of error rates and latency was pinned down to a single pod responsible for handling API customer traffic. This pod in particular failed to initialize a middleware component properly but continued to unsuccessfully serve traffic. Upon termination of the problematic pod, service was restored. ## Preventative Measures * Improve gateway initialization logic and handling of failure scenarios * Improve logging severity to ensure relevant errors trigger alerts accordingly * Adjust alerting policies around error rates and latency for gateway components

Updates are WellSaid Labs's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with WellSaid Labs. This recap summarizes incidents published on WellSaid Labs's official status page.