Ensuring Accurate Software Project Measurement: Addressing Stale GitHub Actions API Data

In the fast-paced world of continuous integration and deployment, accurate and real-time data is paramount for effective software project measurement and maintaining high developer quality. A recent discussion in the GitHub Community highlights a critical issue affecting the reliability of GitHub Actions workflow run data, with significant implications for deployment pipelines.

Intermittent Stale Data from GitHub Actions API and GUI Filters

A community member, smathson, reported an intermittent bug where the GitHub Actions API for listing workflow runs (actions/workflows/{workflow_id}/runs) returns month-old, partial results when filtered by parameters like branch, event, or status. This isn't just an API-specific problem; another user, eirikomn, confirmed seeing the same stale data in the GitHub GUI when applying branch filters, indicating a broader backend consistency issue.

The core problem manifests as inconsistent responses to identical requests made within seconds of each other. For example, 18 out of 20 API calls might return current data (e.g., a run from today with total_count: 1231), while 2 calls intermittently return significantly older data (e.g., a run from a month ago with total_count: 658 or 952). Each inconsistent response carries a unique x-github-request-id, suggesting that these are distinct backend responses rather than client-side caching issues.

Impact on Production Deployments

The severity of this bug is highlighted by its direct impact on deployment processes. smathson reported that this issue caused a production deploy workflow to inadvertently select and use a month-old build. Such an occurrence can severely compromise developer quality and lead to critical production incidents, underscoring the need for robust validation steps in CI/CD pipelines.

Reproducing the Issue

smathson provided a script to reproduce the intermittent behavior, demonstrating the varying total_count and created_at timestamps:

for i in $(seq 1 20); do
  out=$(gh api -i "repos/vpinball/vpinball/actions/workflows/vpinball.yml/runs?branch=master&event=push&per_page=1")
  printf '%s %s %s %s
' "$(date -u +%H:%M:%SZ)" \
  "$(grep -i '^x-github-request-id' <<<"$out" | awk '{print $2}' | tr -d '\r')" \
  "$(grep -o '"total_count":[0-9]*' <<<"$out")" \
  "$(grep -o '"created_at":"[^"]*"' <<<"$out" | head -1)"
  sleep 3
done

Similar reports from other public repositories (oprypin/nightly.link#94 and koala73/worldmonitor#8617) suggest this is not an isolated incident.

Community-Proposed Workarounds and Best Practices

While GitHub investigates the issue (support ticket #4817519 has been filed), the community has offered practical workarounds to mitigate the risks:

• Client-Side Filtering: Instead of relying solely on API filters, fetch a broader set of workflow runs (without filters) and then apply your desired filters client-side. This approach, while potentially requiring more data transfer, ensures you're working with the most complete and up-to-date dataset. Remember to follow pagination to retrieve all relevant runs.
• "Fail-Closed" Deployment Strategy: For critical deployments, implement a robust validation process. Instead of silently falling back to an older build, ensure your deployment pipeline fails if the expected, current build cannot be reliably identified.
  • Determine the intended commit SHA for deployment independently.
  • After selecting a candidate workflow run, fetch that exact run by its ID using GET /repos/{owner}/{repo}/actions/runs/{run_id}.
  • Verify critical details such as its head_sha, repository, workflow, event, and successful completion against your intended build.
  • Download artifacts only from this explicitly verified run ID.
  • If any verification fails, stop the deployment. This ensures that only a deliberately chosen rollback (with its explicitly selected SHA) can proceed.
• Provide Request IDs to Support: When encountering this issue, capture and provide the x-github-request-id from both the current and stale responses to GitHub Support. These IDs are crucial for tracing requests through GitHub's backend services and diagnosing the root cause.

This incident serves as a valuable reminder for teams engaged in agile development retrospective to regularly review their CI/CD practices for resilience against unexpected API behaviors. By adopting these proactive measures, teams can safeguard their developer quality and ensure the integrity of their software project measurement, even when facing intermittent platform inconsistencies.