Errors and stalled predictions on some models
minor- Started
- Nov 17, 10:02 AM EST
- Resolved
- Nov 17, 10:23 PM EST
- Duration
- 12h 21m
Replicate's updates (5)
- investigating · Nov 17, 10:02 AM EST
We are currently investigating this issue. Cloudflare is aware of, and investigating, an issue affecting predictions for some models on Replicate. Customers may see prediction requests return HTTP 500 errors, and some predictions may remain in a processing state without completing. Some API responses are also missing version information, which can cause validation errors in client libraries. We are working to understand the full impact and mitigate this problem.
- identified · Nov 17, 11:32 AM EST
The issue has been identified and a fix is being implemented. Current impact: prediction requests for some models may return HTTP 500 errors, and some predictions may stay in a processing state. Customers using the API may also see validation errors on responses.
- identified · Nov 17, 4:53 PM EST
The issue has been identified and a fix is being implemented. Current impact: predictions on OpenAI models are back to normal. Predictions on other models may still return HTTP 500 errors or stay in a processing state while we continue to work on a fix.
- monitoring · Nov 17, 9:56 PM EST
A fix has been implemented and we are monitoring the results.
- resolved · Nov 17, 10:23 PM EST
This incident has been resolved.
What our checks recorded
We checked Replicate 148 times from Replicate's start time to its resolution, once every five minutes. Every one of them reached Replicate's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 305 ms, against 286 ms over the 24 hours before. Replicate posted the incident at Nov 17, 10:02 AM EST.
Updates are Replicate's own words from its status page (status: resolved); we never add an inferred cause. The vendor's incident page ↗