Major outage on DeepL Pro
critical- Started
- Feb 12, 5:51 AM EST
- Resolved
- Feb 12, 12:09 PM EST
- Duration
- 6h 18m
DeepL's updates (8)
- investigating · Feb 12, 5:51 AM EST
We are investigating this issue.<br><br><b>Affected Services:</b> Document Translations
- investigating · Feb 12, 8:42 AM EST
We are still investigating the cause of the outage.
- investigating · Feb 12, 11:17 AM EST
We are continuing to investigate this issue.
- monitoring · Feb 12, 11:36 AM EST
A fix has been implemented and we are monitoring the results.
- monitoring · Feb 12, 11:48 AM EST
We are continuing to monitor for any further issues.
- monitoring · Feb 12, 11:49 AM EST
We are continuing to monitor for any further issues.
- resolved · Feb 12, 12:09 PM EST
This incident has been resolved.
- postmortem · Feb 12, 12:25 PM EST
# Summary On February 12th 2025, DeepL experienced a prolonged outage affecting document translation for both Pro and free users across the website, apps, API. The incident timeline: * 10:13 UTC - The service stopped working for Pro users * 12:20 UTC - The service stopped working for free users * 16:36 UTC - The functionality was fully restored for all users We acknowledge the significant disruption caused by this outage and apologize for its negative impact. We are committed to preventing similar incidents in the future, as described below. ## Incident detection and resolution An on-call engineer was alerted for failures in document translations. Upon investigations, a slow down in the system processing uploaded documents was found to be the cause. Engineers from the document translation and infrastructure teams started looking into a resolution. Recent deployments were ruled out quickly. To gain further insights into the cause, we enhanced observability into the state of the running service in production. This revealed a nontrivial bug in the document translation service making it tougher to debug in our pre-production environments and due to the rarity it passed through our internal test suite. The document translation service relies on an internal message queueing system. We found that while processing a document, the system would fail without accepting or rejecting the message in the queue. This caused the queue to build up and the uploaded documents to never be successfully processed. We rolled out a fix quickly. This restored service and documents in the backlog started processing successfully. We were confident that the issue was resolved at this point. ## Measures to mitigate future outages Due to the severity of this outage, we are dedicating significant engineering resources to prevent similar incidents in the future. Below are the three primary measures identified in our internal post-mortem. **Improved observability:** Diagnosing the root cause took significantly longer because the issue arose in an unmonitored segment of our stack that had never failed before. To prevent similar issues, we are implementing measures that enhance visibility into the entire document translation service’s internal stack, ensuring faster problem detection and resolution in the future. **Extended failover capabilities:** DeepL maintains a robust platform, reinforced by automatic failovers for critical services and fallback systems in different locations. We are actively expanding these failover mechanisms to cover additional services, including Document Translation, and are prioritizing these efforts to ensure greater overall system reliability. **Refined incident response protocols:** In addition to technical measures, we are enhancing our incident response protocols to minimize disruption during extended outages. Our efforts focus on earlier internal escalation to mobilize additional engineering resources and speed up incident resolution as well as more proactive customer communication throughout any prolonged incident to keep our customers informed.
Updates are DeepL's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗