When CI/CD Stalls: Uncancelable GitHub Actions Runs Block Software Development Projects

In the fast-paced world of software development, continuous integration and continuous delivery (CI/CD) pipelines are the lifeblood of efficient project execution. Yet, even the most robust systems can encounter unexpected roadblocks. A recent discussion on GitHub Community highlights a critical bug within GitHub Actions that saw workflow runs become permanently stuck in a 'Queued' state, rendering them uncancelable and severely impacting the progress and planning a software development project.

The Uncancelable Queue: A Development Blocker

The issue, reported by rafaeldeak-lab, involved two manually dispatched GitHub Actions runs for the dzn-network repository. Both runs were for the workflow named "DZN Owner Console Preview" on a specific feature branch. Despite being dispatched, they remained indefinitely in a 'Queued' state, exhibiting several alarming characteristics:

• Zero Jobs: Neither run initiated any jobs.
• No Runner Assignment: No runner was ever assigned to process the workflows.
• No Steps Executed: Workflow steps never started.
• Complete Stasis: No artifacts, deployments, or external operations were initiated.

One of the runs had been stuck for over two days, effectively halting any related development or release verification processes. This kind of unexpected halt can throw off an entire planning a software development project timeline, creating significant delays and frustration.

Cancellation Failures and Deeper Issues

What made this bug particularly insidious was the complete inability to cancel the stuck runs. Attempts via multiple methods consistently failed:

• GitHub Website UI: Displayed a "Failed to cancel workflow" error.
• Standard REST API: Returned an HTTP 500 error.
• Force-Cancel REST API: Also returned an HTTP 500 error.

The user even dispatched a replacement run using the same workflow, branch, and concurrency group. Instead of replacing or cancelling the original pending run, this new run also became stuck in the queue, exacerbating the problem and preventing further attempts to avoid multiple unexpected executions.

Concurrency Configuration

The workflow utilized a concurrency group, which is designed to manage simultaneous runs. However, the cancel-in-progress: false setting indicates that new runs should not automatically cancel existing ones within the group. While this setting explains why the new run didn't cancel the old ones, it doesn't account for the runs being stuck with zero jobs or the HTTP 500 errors on manual cancellation.

concurrency:
  group: dzn-owner-console-preview-${{ github.ref }}
  cancel-in-progress: false

Impact and Resolution

The ongoing issue severely blocked development and release verification for the project. The user's inability to clear these corrupted records from their side, coupled with the fear of unexpected execution of the stuck runs, led to a direct plea for GitHub staff intervention to purge the backend run records and clear the concurrency state.

This incident underscores the critical importance of reliable CI/CD infrastructure for effective planning a software development project and maintaining developer productivity. When core automation tools fail in such a fundamental way, it can bring an entire project to a standstill, highlighting the need for robust error handling and recovery mechanisms within these platforms. For teams relying heavily on automated workflows, such persistent issues can quickly derail project timelines and necessitate a re-evaluation of their development strategies.