Outage in aws us-east-1
major- Started
- Oct 20, 3:58 AM EDT
- Resolved
- Oct 20, 7:18 PM EDT
- Duration
- 15h 19m
turbopuffer's updates (5)
- investigating · Oct 20, 3:58 AM EDT
[AWS us-east-1 is experiencing a major outage and many services are affected. ](https://health.aws.amazon.com/health/status) turbopuffer clients in aws-us-east-1 will see elevated error rate and latencies during this outage. We're looking at mitigating the impact.
- monitoring · Oct 20, 4:38 AM EDT
We have deployed mitigations and are monitoring the situation.
- monitoring · Oct 20, 10:32 AM EDT
AWS is experiencing a major disruption in region us-east-1 impacting multiple services (<https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A>). This caused turbopuffer to become unavailable in that region starting at 7:50 UTC. We've rolled out mitigations which resulted in a full recovery of turbopuffer at 8:20 UTC (despite the ongoing AWS outage). An existing mitigation to guard against EC2 instance launch issues has prevented the long-running EC2 degradation from further affecting turbopuffer availability. The AWS incident is ongoing. We will continue monitoring the cluster's health.
- monitoring · Oct 20, 5:39 PM EDT
The turbopuffer service remains healthy in the aws-us-east-1 region. We continue to closely monitor the situation while AWS continues to report service disruption in the region.
- resolved · Oct 20, 7:18 PM EDT
AWS has reported that the upstream incident is resolved.
Updates are turbopuffer's own words from its status page (status: resolved); we never add an inferred cause.