Root cause summary

The data quality test duplicate_subs on table mysql.default.subscriptions aborted due to a data freshness issue. The upstream pipeline airflow.ingest_subs_daily had not run successfully since 2026-04-20T03:00:00Z, leading to stale data that prevented the DQ test from executing correctly. This is a recurring issue, as a similar incident was observed on 2026-04-26T10:59:14.187000+00:00.

Evidence chain

1. The data quality test duplicate_subs on mysql.default.subscriptions reported an 'Aborted' status at 2026-05-30T12:04:12.786952Z.
2. Multiple other data quality tests on various tables also reported 'Aborted' status around the same time, indicating a broader system issue rather than isolated data corruption.
3. The upstream pipeline airflow.ingest_subs_daily for mysql.default.subscriptions shows its last_run_at as 2026-04-20T03:00:00Z, which is significantly older than the incident trigger time (2026-05-30T12:04:08.253000+00:00). This suggests that the pipeline has not refreshed the data in mysql.default.subscriptions for over a month.
4. A past incident on 2026-04-26T10:59:14.187000+00:00 for the same table mysql.default.subscriptions and test case duplicate_subs had a similar root cause, noting an aborted test due to an upstream pipeline issue and a stale last_run_at for airflow.ingest_subs_daily.

Remediation steps

1. Investigate the airflow.ingest_subs_daily pipeline to determine why it has not been running as scheduled or why its status is not accurately reflecting recent runs. Check scheduler logs and pipeline configuration.
2. Manually trigger the airflow.ingest_subs_daily pipeline to refresh the data in mysql.default.subscriptions.
3. Review and enhance the monitoring for the airflow.ingest_subs_daily pipeline to promptly detect missed runs or data freshness issues.
4. Re-run the duplicate_subs data quality test on mysql.default.subscriptions after the upstream pipeline has successfully completed a run.