S
Outage recap · SuperAnnotate

SuperAnnotate outage — February 23, 2026

On February 23, 2026, SuperAnnotate had a reported incident on SuperAnnotate's official status page. Below is what was reported — impact, timing, and the AI services the same upstream outage could affect.

Orchestrate – Item event-triggered actions are not firing, Work Management - workload number of assignments are not updated.

major
Started
Feb 23, 1:21 AM EST
Resolved
Feb 23, 2:01 AM EST
Duration
40 min

SuperAnnotate's updates (5)

  1. investigating · Feb 23, 1:21 AM EST

    We are currently investigating an issue where item-based event triggers in Orchestrate are not executing as expected, and the workload assignment counts in are not reflecting recent updates. Our team is actively working to identify the root cause and resolve the issue. Further updates will be shared as they become available.

  2. identified · Feb 23, 1:49 AM EST

    The issue has been identified.

  3. monitoring · Feb 23, 1:50 AM EST

    The Issue has been resolved, all the affected jobs have been triggered, will take some time to finish.

  4. resolved · Feb 23, 2:01 AM EST

    The issue is confirmed to be fully resolved. All the jobs have started execution.

  5. postmortem · Mar 13, 1:32 AM EDT

    ### **Summary** On February 23 2026, a production Kafka Connect connector responsible for streaming MySQL binlog events temporarily stopped processing new events while remaining in a RUNNING state. Because the connector continued to report a healthy status and did not emit errors, the condition was not immediately detected by monitoring. The issue was resolved by recreating the connector, which re-established the binlog replication stream and restored normal processing. ### **Impact** Data ingestion into downstream production systems was delayed during the incident window. Scope: A single production Kafka Connect connector responsible for MySQL binlog ingestion. Observed behavior: The connector remained in RUNNING state while throughput dropped to zero. No connector errors were generated and the system continued reporting the task as healthy. As a result, binlog events were not ingested for approximately 2 hours and 50 minutes, delaying the availability of newly generated data in downstream systems. ‌ ### **Timeline** * February 23, 03:53 UTC – Last successful batch of binlog events processed. * February 23, 05:30 UTC – Connector observed in RUNNING state with zero throughput. Investigation initiated. * February 23, 06:00 UTC – Connector restart attempted. Processing did not resume. * February 23, 06:10 UTC – Temporary connector instance created to validate behavior. * February 23, 06:20 UTC – Existing connector instances removed to reset state. * February 23, 06:35 UTC – Connector configuration recreated. * February 23, 06:39 UTC – MySQL replication logs detected and terminated a stale dump thread with the same server\_id. * February 23, 06:43 UTC – Binlog processing resumed. * February 23, 06:48 UTC – Throughput returned to normal levels. * February 23, 06:53 UTC – Processing confirmed stable. ### **Root Cause** Expected behavior: If a binlog replication stream becomes unavailable or stops delivering events, the connector should detect the failure and transition to a failed state or attempt recovery. Actual behavior:  The LoadBalancer idle timeout was configured to 600 seconds, while the application timeout was 601 seconds. As a result, the network layer expired the idle connection before the connector detected it, leaving the task in a RUNNING state while binlog delivery had already stopped. When the connector was recreated, MySQL identified the previous replication dump thread as stale and terminated it, allowing a new replication stream to be established. ### **Resolution** The connector was removed and recreated, which reinitialized the MySQL replication stream. During initialization MySQL terminated the stale replication dump thread and established a new connection. Binlog processing resumed immediately and the backlog was processed normally. ### **Pipeline Improvements Implemented** * Added monitoring for sustained zero throughput and stalled offset progression even when connectors report RUNNING. * Eliminate silent connector stalls by correcting timeout mismatches, adding progress-based monitoring and alerting. * Evaluating automated recovery policies to restart connectors when binlog progress stalls beyond a defined threshold. * Reviewing TCP keepalive and timeout configuration to reduce the risk of undetected half-open replication connections.

Updates are SuperAnnotate's own words from its status page (status: postmortem); we never add an inferred cause. The vendor's incident page ↗

AIDownCheck is not affiliated with SuperAnnotate. This recap summarizes incidents published on SuperAnnotate's official status page.