Forge Intermittent Web Page Unavailability
major- Started
- Nov 5, 10:22 AM EST
- Resolved
- Nov 5, 1:10 PM EST
- Duration
- 2h 48m
Weights & Biases's updates (7)
- investigating · Nov 5, 10:22 AM EST
We are investigating an issue where the forge site web pages are intermittently unavailable due to underlying rate limit issues.
- identified · Nov 5, 10:37 AM EST
We have identified the technical root cause and are looking into a patch to resolve the rate limit errors that are blocking user flows.
- identified · Nov 5, 11:34 AM EST
We are deploying a temporary configuration change around the integration rate limits to mitigate the impact while we are working on a larger architectural change to resolve the larger root cause. We will monitor for the impact of the mitigating fix and will update when we are able to push out the more comprehensive patch.
- identified · Nov 5, 12:00 PM EST
We have not seen a significant improvement from our initial mitigations, but we are finalizing testing on the more comprehensive patch that will be rolled out shortly pending verification. We will update at that time, and monitor the incident from then on.
- monitoring · Nov 5, 12:20 PM EST
We have deployed the comprehensive patch, and are seeing improvements on our internal metrics and external error rates. We will continue monitoring the results to validate the impact.
- monitoring · Nov 5, 12:28 PM EST
We are seeing stability return back to normal for the affected services, and are continuing to monitor uptime.
- resolved · Nov 5, 1:10 PM EST
We are seeing a return to nominal performance for the last 30 mins along with corresponding internal metrics, and will mark this incident as resolved.
What our checks recorded
We checked Weights & Biases 34 times from Weights & Biases's start time to its resolution, once every five minutes. Every one of them reached Weights & Biases's site, so this incident did not show up as a failed page load. Our check loads the public site; it cannot see an API error or a broken feature. Median response time was 571 ms, against 640 ms over the 24 hours before. Weights & Biases posted the incident at Nov 5, 10:22 AM EST.
Updates are Weights & Biases's own words from its status page (status: resolved); we never add an inferred cause. The vendor's incident page ↗