Granular CI Metrics at Risk: GitHub's 'Merge as Stack' and Performance Dashboard Metrics

GitHub's "Merge as Stack" feature, designed to streamline the integration of multiple pull requests, is encountering a significant challenge with how it interacts with continuous integration (CI) systems. A recent community discussion highlights a critical gap: the feature emits only a single webhook for a push event to the trunk branch, even when several PRs are merged simultaneously. This behavior breaks fundamental assumptions for many CI setups, particularly those relying on granular, commit-level triggers for accurate performance dashboard metrics.

The Challenge: Single Webhook, Multiple Commits

The core of the issue, raised by user ghthor, is that traditional CI systems expect a separate push event webhook for each commit pushed to the trunk branch. This allows CI pipelines to be triggered individually for every change, enabling precise workload isolation and detailed analysis. However, when "Merge as Stack" is used, multiple PRs (and thus multiple underlying commits) are merged into the trunk, yet only one webhook is dispatched. This aggregation of events means that CI systems miss individual commit triggers, leading to incomplete or inaccurate data for performance dashboard metrics related to code changes and CI efficiency.

Impact on Monorepos and Change Impulse Calculations

Ghthor's specific use case illustrates the severity of this problem. Managing a monorepo, their team performs "change impulse calculations" per commit to the trunk. This approach is crucial for isolating CI workloads to the minimal set of changes caused by a specific commit. With "Merge as Stack" pushing multiple PRs and emitting only a single webhook, these granular calculations are fundamentally broken. The CI system only receives a trigger for the "top of stack" merge, losing the ability to attribute CI workload and impact to individual PRs or commits within the stack. This directly affects the ability to generate reliable performance dashboard metrics for individual contributions and the overall health of the monorepo.

Why Granular Webhooks Matter for Developer Productivity

The ability to trigger CI per commit is not just a technical detail; it's a cornerstone of modern developer productivity and effective monitoring. Granular CI triggers allow teams to:

• Isolate failures: Quickly identify which specific commit introduced a bug or broke the build.
• Optimize CI resources: Run targeted tests based on the exact changes in a commit, reducing unnecessary compute time.
• Generate accurate metrics: Provide precise data for performance dashboard metrics, such as commit-to-deploy time, CI success rates per commit, and the impact of individual changes on build duration.
• Support advanced tooling: Enable sophisticated systems that perform analyses like ghthor's change impulse calculations, which are vital for large, complex repositories.

Without these granular triggers, teams risk losing visibility into the true performance of their CI/CD pipelines and the impact of individual developer contributions. This can lead to difficulties in optimizing workflows, troubleshooting issues, and accurately reporting on key data engineer KPIs related to code integration and deployment.

Community Feedback and Next Steps

The discussion received an automated response confirming that the product feedback has been submitted to GitHub's product teams. While there's no immediate solution or workaround provided, the feedback highlights a significant pain point for developers leveraging advanced CI strategies, especially in monorepo environments. GitHub encourages users to upvote, comment, and add more details, emphasizing that community input is instrumental in guiding product improvements.

As development teams increasingly rely on sophisticated CI/CD pipelines and detailed analytics to drive efficiency, addressing issues like the "Merge as Stack" webhook behavior becomes crucial. It directly impacts the fidelity of data used for performance dashboard metrics and the overall effectiveness of development workflows.

Are you experiencing similar challenges with GitHub's "Merge as Stack" or other features impacting your CI metrics? Share your insights and workarounds in the comments below.