Outage recap · DeepL

DeepL outage — February 12, 2025

On February 12, 2025, DeepL had 2 reported incidents on DeepL's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Major outage on DeepL Pro

critical
Started
Feb 12, 5:51 AM EST
Resolved
Feb 12, 12:09 PM EST
Duration
6h 18m

DeepL's updates (8)

  1. investigating · Feb 12, 5:51 AM EST

    We are investigating this issue.<br><br><b>Affected Services:</b> Document Translations

  2. investigating · Feb 12, 8:42 AM EST

    We are still investigating the cause of the outage.

  3. investigating · Feb 12, 11:17 AM EST

    We are continuing to investigate this issue.

  4. monitoring · Feb 12, 11:36 AM EST

    A fix has been implemented and we are monitoring the results.

  5. monitoring · Feb 12, 11:48 AM EST

    We are continuing to monitor for any further issues.

  6. monitoring · Feb 12, 11:49 AM EST

    We are continuing to monitor for any further issues.

  7. resolved · Feb 12, 12:09 PM EST

    This incident has been resolved.

  8. postmortem · Feb 12, 12:25 PM EST

    # Summary On February 12th 2025, DeepL experienced a prolonged outage affecting document translation for both Pro and free users across the website, apps, API. The incident timeline: * 10:13 UTC - The service stopped working for Pro users  * 12:20 UTC - The service stopped working for free users * 16:36 UTC - The functionality was fully restored for all users We acknowledge the significant disruption caused by this outage and apologize for its negative impact. We are committed to preventing similar incidents in the future, as described below. ## Incident detection and resolution An on-call engineer was alerted for failures in document translations. Upon investigations, a slow down in the system processing uploaded documents was found to be the cause. Engineers from the document translation and infrastructure teams started looking into a resolution. Recent deployments were ruled out quickly. To gain further insights into the cause, we enhanced observability into the state of the running service in production. This revealed a nontrivial bug in the document translation service making it tougher to debug in our pre-production environments and due to the rarity it passed through our internal test suite.  The document translation service relies on an internal message queueing system. We found that while processing a document, the system would fail without accepting or rejecting the message in the queue. This caused the queue to build up and the uploaded documents to never be successfully processed. We rolled out a fix quickly. This restored service and documents in the backlog started processing successfully. We were confident that the issue was resolved at this point.  ## Measures to mitigate future outages Due to the severity of this outage, we are dedicating significant engineering resources to prevent similar incidents in the future. Below are the three primary measures identified in our internal post-mortem. **Improved observability:** Diagnosing the root cause took significantly longer because the issue arose in an unmonitored segment of our stack that had never failed before. To prevent similar issues, we are implementing measures that enhance visibility into the entire document translation service’s internal stack, ensuring faster problem detection and resolution in the future. **Extended failover capabilities:** DeepL maintains a robust platform, reinforced by automatic failovers for critical services and fallback systems in different locations. We are actively expanding these failover mechanisms to cover additional services, including Document Translation, and are prioritizing these efforts to ensure greater overall system reliability. **Refined incident response protocols:** In addition to technical measures, we are enhancing our incident response protocols to minimize disruption during extended outages. Our efforts focus on earlier internal escalation to mobilize additional engineering resources and speed up incident resolution as well as more proactive customer communication throughout any prolonged incident to keep our customers informed.

Updates are DeepL's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

Degraded performance on DeepL Pro, DeepL Free

minor
Started
Feb 12, 7:15 PM EST
Resolved
Feb 12, 8:49 PM EST
Duration
1h 33m

DeepL's updates (3)

  1. investigating · Feb 12, 7:15 PM EST

    We are investigating this issue.<br><br><b>Affected Services:</b> Translations, Document Translations, Write, Login, Login (SSO)<br><b>Additional Info:</b> We have some internal problems that result in error messages on the website. Translations still work

  2. resolved · Feb 12, 8:49 PM EST

    This incident has been resolved.

  3. postmortem · Feb 13, 9:51 AM EST

    An error introduced in our internal DNS server configuration meant that some services were unable to route internal traffic. This resulted in errors being displayed on the website and login page. It was also not possible to add people to a subscription during the incident. The incident was first detected on 2025-02-13 12:18 AM CET and fixed on 2025-02-13 2:53 AM CET.

Updates are DeepL's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with DeepL. This recap summarizes incidents published on DeepL's official status page.