GitHub Actions Outage: Unpacking pull_request Failures and Engineering Goals for CI/CD Reliability

A recent discussion in the GitHub Community highlighted a critical incident where GitHub Actions workflows failed to enqueue for pull_request events on a private repository. This disruption prevented essential status checks from running, effectively blocking Pull Requests (PRs) from merging and impacting development velocity. For any software engineer, ensuring the reliability of CI/CD pipelines is among the most important engineering goals examples, and this incident underscores the challenges in maintaining robust automation.

The Challenge: Unresponsive CI/CD for Pull Requests

The user, thomasjustesq-dev, reported that their private repository, thomasjustesq-dev/penumbra, experienced a severe outage where pull_request events were not triggering GitHub Actions workflows. While workflow_dispatch events continued to function normally, new PRs showed zero associated Actions runs, causing required status checks like build-and-test and spec-guard to never appear. This meant PRs were perpetually blocked from merging under branch protection rules.

Adding to the complexity, one specific pull_request workflow run (ID 31125024043) became stuck in a queued state for hours and could not be cancelled. Attempts to cancel via API or CLI resulted in errors like 409 Cannot cancel a workflow re-run that has not yet queued or Cannot cancel a workflow run that is completed, despite the API still reporting status: queued. The raw API snapshot for the stuck run clearly showed its indeterminate state:

{"conclusion":null,"created_at":"2026-08-06T18:05:11Z","event":"pull_request","head_branch":"automation/claim-reconcile","html_url":"https://github.com/thomasjustesq-dev/penumbra/actions/runs/31125024043","id":31125024043,"run_started_at":"2026-08-06T20:33:06Z","status":"queued","updated_at":"2026-08-06T20:33:06Z"}

The incident also involved a mysterious "phantom workflow" named .NET Core Desktop, which had no corresponding YAML file in the repository but was registered and active. Though manually disabling it did not resolve the core issue, it raised questions about platform integrity and unexpected workflow registrations.

Root Cause: Actions Runner Controller (ARC) Glitches

A GitHub staff member, ebndev, later clarified the root cause: some Actions Runner Controller (ARC) runner pods became stuck in an idle state. This issue prevented the GitHub-hosted runners from picking up new workflow jobs, leading to the observed enqueueing failures and perpetually queued runs.

Resolution and Future Reliability Engineering Goals

The immediate resolution for affected users involved manually deleting the stuck ARC pods using kubectl or redeploying the Actions Runner Controller application. This action prompted ARC to automatically create replacement runners, restoring normal workflow processing. GitHub confirmed that future releases of Actions Runner and Actions Runner Controller would include an automatic recovery mechanism, aiming to prevent such manual interventions in the future.

This incident serves as a stark reminder for every software engineer OKR to include robust CI/CD reliability. While platform-level issues are outside direct control, understanding their impact and the proposed solutions is vital. Teams can benefit from comprehensive engineering dashboard examples that track CI/CD health, queue times, and runner availability to quickly detect and respond to similar anomalies. Prioritizing system resilience and automated recovery mechanisms should be paramount in any organization's engineering goals examples.

Key Takeaways for Software Engineers

• Monitor CI/CD Health: Implement proactive monitoring for workflow queue times, runner status, and job failures.
• Understand Runner Architecture: Familiarity with how GitHub-hosted or self-hosted runners (especially ARC) operate can aid in faster diagnosis during outages.
• Plan for Resilience: While GitHub works on automatic recovery, consider fallback strategies or communication plans for CI/CD disruptions.
• Report Detailed Issues: The original post's detailed breakdown of symptoms, environment, and API snapshots was crucial for rapid diagnosis by the GitHub team.