Jobs in a queue:max concurrency group stop being scheduled and cannot be cancelled

[Soju06]

Jobs in one job-level concurrency group stop being dispatched. They sit at status=pending with runner_name=null and an empty steps array, while no job in that group is in progress. Cancelling a run that holds one of them is also ignored.

Happened twice on the same day: 194 and 139 minutes.

Private repo, org-level self-hosted runners.

Config

Shared by 7 job definitions across 5 workflows:

concurrency:
  group: promotion-pipeline-critical
  queue: max
  cancel-in-progress: false

What we see

Job 99240622687 (run 33305095508):

queued          09:57:18Z
runner assigned 13:36:54Z   <- 220 minutes later
completed       13:40:55Z   (4.0 minutes of actual work)

The group was idle from 11:17:46Z, when the last job in it finished, until 13:36:54Z. Nothing in the group ran during those 139 minutes. Six jobs were queued behind it, all with runner_name=null and steps: [].

At 12:56:58Z, job 99260854039 in a different workflow requested the same runner label and was assigned a runner immediately. Runner supply was fine. These jobs were never offered to it.

Cancellation is affected too

• Run 33308212239, which had zero jobs created: cancelled in 8 seconds.
• Run 33299662484, whose only outstanding job was one of these pending ones: cancel requested at 13:12:09Z, still status=pending at 13:55Z. updated_at frozen at 13:12:32Z. Its commit-values job had been pending for 158 minutes at that point.

Notably, by 13:36Z a different job in the same group had already been dispatched, so the group was moving again — but this run stayed stuck and uncancellable.

Ruled out

Runner shortage	Same label served at 12:56 and 13:32
Org-wide load	Zero jobs in_progress across the org while 8 were queued
Queue limit	7 pending; queue: max allows 100
Unmet needs	All 25 upstream jobs succeeded; the if: condition evaluates true
Another job holding the group	Scanned every non-completed run: none in progress
Platform incident	githubstatus.com all green throughout

Both occurrences cleared on their own. Neither cancellation had any effect.

Questions

1. Why do jobs stay pending with no runner assignment while the group is unoccupied and the queue is far under the limit?
2. Why is a cancel request ignored for a run holding one of these jobs, when a run with no jobs cancels in seconds?
3. Is this known for queue: max groups shared across multiple workflows?

Docs say ordering is FIFO by the time each job started waiting on the group, and that ordering is not guaranteed — but this is not an ordering problem. Nothing was dispatched at all for over two hours.

[github-actions[bot]]

Thank you for your interest in contributing to our community! We currently only accept discussions created through the GitHub UI using our provided discussion templates. Please re-submit your discussion by navigating to the appropriate category and using the template provided.

This discussion has been closed because it was not submitted through the expected format. If you believe this was a mistake, please reach out to the maintainers.

[jasongin]

Thanks for the detailed report and the run/job IDs. I investigated the relevant scheduling telemetry.

For job 99240622687 in run 33305095508, the concurrency group was not idle during the reported interval. Between 11:17:46Z and 13:36:52Z, approximately 20 jobs that were already ahead of it in the shared promotion-pipeline-critical queue acquired and released the group in succession. Most were publish-production jobs from older runs of another workflow sharing the same group.

When the preceding job released the group at 13:36:52Z, job 99240622687 was notified and acquired the group within milliseconds. It was then assigned to a runner at 13:36:54Z. We found no concurrency-service, queue, or runner-assignment errors in this sequence. The available runner observed earlier could not accept this job because it had not yet reached the front of the concurrency queue.

I cannot tell from the provided information exactly why those intervening jobs were absent from your inventory. When inspecting a group shared across workflows, make sure the query includes older queued runs from every workflow, follows all API pagination, and does not exclude runs that complete while the inventory is being collected.

I also investigated cancellation of run 33299662484. The cancellation request was received successfully. However, its pending commit-values job had an if expression beginning with always(). GitHub Actions re-evaluates job conditions when a run is canceled; this expression still evaluated to true, so the job was intentionally allowed to remain queued and later run. This is the documented behavior of always().

If commit-values should not run after cancellation, remove always() or add an explicit cancellation check, for example:

if: ${{ !cancelled() && <existing dependency conditions> }}