High rate of errors on control plane
major- Started
- May 5, 7:35 PM EDT
- Resolved
- May 5, 8:23 PM EDT
- Duration
- 48 min
Braintrust's updates (5)
- investigating · May 5, 7:35 PM EDT
We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.
- identified · May 5, 8:08 PM EDT
We have identified the cause and are working with an upstream provider on a fix.
- monitoring · May 5, 8:10 PM EDT
We are actively monitoring the fix.
- resolved · May 5, 8:23 PM EDT
We are no longer seeing issues.
- resolved · May 5, 8:41 PM EDT
The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.
Updates are Braintrust's own words from its status page (status: resolved); we never add an inferred cause.