Data Center Failure Impacting Capacity
minor- Started
- Jul 1, 7:01 PM EDT
- Resolved
- Jul 1, 8:35 PM EDT
- Duration
- 1h 34m
Groq's updates (5)
- investigating · Jul 1, 7:01 PM EDT
We are currently investigating a potential issue at one of our US Central data centers that is impacting system capacity. Users **may experience higher latencies** due to reduced resources. Our engineering team is working to identify the root cause and remediate the problem.
- identified · Jul 1, 7:11 PM EDT
We have **identified a cooling system failure** at one of our **US Central** data centers that is causing **reduced capacity**. The team is **working on restoring capacity**. Users **continue to see elevated latencies for specific models**.
- identified · Jul 1, 7:27 PM EDT
A power loss issue at approximately 22:20pm UTC led to a subsequent **cooling system failure** at one of our **US Central** data centers that is causing **reduced capacity**, primarily for the `openai/gpt-oss-20b` model. The team is **working on restoring capacity**.
- monitoring · Jul 1, 7:48 PM EDT
We have **redistributed and allocated more capacity to production models** in **the US Central region.** Users should **see performance return to normal**. We’re now **monitoring** the infrastructure to ensure stability.
- resolved · Jul 1, 8:35 PM EDT
The **capacity issue has been resolved**. All services are operating at normal capacity. The incident was caused by a **power loss issue that led to a cooling system failure at one of our US Central data centers**. We apologize for any disruption and **appreciate your patience**.
What our checks recorded
We checked Groq 19 times from Groq's start time to its resolution, once every five minutes. Every one of them reached Groq's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 175 ms, against 210 ms over the 24 hours before. Groq posted the incident at Jul 1, 7:01 PM EDT.
Updates are Groq's own words from its status page (status: resolved); we never add an inferred cause.