ORG: Reconcile triage board Status #29

Workflow file for this run

.github/workflows/org_reconcile_board_status.yml at 7d19a27

	# Scheduled org job: reconcile Org Triage board Status and labels from live state
	# ------------------------------------------------------------------------------
	#
	# Not a reusable workflow — it runs here on a schedule and reaches the org
	# project through the API. There is no per-repo shim, because PR events fire in
	# the owning repo and this job deliberately does not depend on them: a
	# reconciler is idempotent, so a missed event self-heals on the next pass
	# instead of leaving a card wrong forever. See simp/puppetsync#99.
	#
	# Board: https://github.com/orgs/simp/projects/11
	#
	# Two passes, in order, over the same snapshot of the board:
	#
	# 1. Status -- move member PRs (and issues they close) between columns
	# 2. Labels -- `community` on outside contributions (simp/puppetsync#98),
	# `needs-attention` on items idle past their column's threshold
	# (simp/puppetsync#99, staleness extension). Thresholds are
	# keyed off the Status the first pass just set.
	#
	# ==============================================================================
	# Mapping (members only)
	#
	# Member PR, draft ................................ In Progress
	# Member PR, ready for review ..................... In Review
	# Member PR, changes requested .................... In Progress
	# Member PR, approved but unmerged ................ In Review
	# Issue with an open member PR that closes it ..... In Progress
	# Anything authored from outside the org .......... UNTOUCHED
	#
	# "Member" means the author's login appears in the organization's member list,
	# fetched once per run. It deliberately does NOT use the PR's `authorAssociation`
	# field: that is credential-dependent, and an App installation token reading a
	# public repo gets the same degraded answer as an anonymous client. Verified on
	# simp/pupmod-simp-ssh#229 -- a user PAT reports MEMBER while both an App token
	# and an unauthenticated request report CONTRIBUTOR. Trusting it made the mapping
	# treat 29 org members as outside contributors, doing about a third of its work
	# with no signal.
	#
	# One intentional narrowing: an outside collaborator on a single repo who is not
	# an org member now counts as an outside contribution. That matches the rule we
	# actually want ("is this one of us?") rather than the repo-scoped relationship
	# authorAssociation described.
	#
	# Outside contributions are left alone deliberately. Marking a drive-by draft
	# "In Progress" would imply we are working it, and would drop it out of the
	# triage queue before anyone had looked. They stay in New, carrying the
	# `community` label, until a human picks them up.
	#
	# A consequence worth knowing: New ends up holding only issues and outside
	# contributions — i.e. exactly the things that need a human decision. A member's
	# own PR has a knowable state, so it does not need triaging.
	#
	# ==============================================================================
	# What it will never do
	#
	# * Move an item out of `Blocked`, `Parked` or `Done`. Those are deliberate
	# human placements; a reconciler that overrode them would undo the board
	# review's decisions on its next tick. All three names are asserted to exist
	# at start-up, so renaming a column fails the run instead of silently
	# disarming the guard.
	# * Touch any field other than Status, or any label other than `community` and
	# `needs-attention`.
	# * Remove `community`. It records a permanent property of the contribution.
	# * Close anything. A stale-closer would fire hardest on first-time
	# contributors -- exactly the people the board must not lose. It labels and
	# surfaces; a human closes.
	# * Put `needs-attention` on anything in `Todo`, `Ready`, `Parked`, `Blocked`
	# or `Done`. Sitting there is expected; the periodic board review covers it.
	# * Leave a hand-applied `needs-attention` in place. The label is JOB-OWNED --
	# its description reads "applied and removed automatically" -- so the job
	# removes it from any column above and from any item whose human activity is
	# known to be inside the threshold, whoever applied it. Flag a Blocked or
	# Parked item for the review some other way (a comment, an assignee).
	# * Create a label. `POST .../issues/N/labels` would silently create a missing
	# label with an arbitrary colour; each (repo, label) is confirmed to exist
	# first, and a repo without it is a per-item failure instead.
	# * Revert an issue it previously promoted. Issue promotion is one-way —
	# see the note below.
	# * Reconcile an item the credential cannot view. The API reports those with
	# `type: REDACTED` ("A project may contain items that a user does not have
	# permission to view"), which is a permission signal: the run aborts on it,
	# because a partial view would apply some moves and skip the rest in silence.
	# Content that is genuinely gone (deleted, or moved out of the installation's
	# reach) shows as a readable type with null content instead, and is warned
	# about and skipped -- no grant recovers it, so aborting would deadlock the
	# board.
	# * Move anything to `Done`, or reconcile a PR once it is no longer open. That
	# is handled by the board's built-in "Pull request merged" and **"Item
	# closed"** workflows, both currently enabled. Those two are load-bearing: if
	# they are ever disabled, merged PRs will accumulate in `In Review` and this
	# job will not clear them.
	#
	# Status option IDs are looked up by name at run time. Adding or renaming a
	# column mints new option IDs, so hardcoding them would break silently.
	#
	# ==============================================================================
	# Issue promotion is deliberately one-way
	#
	# An issue moves to `In Progress` when an open member PR closes it. If that PR
	# is later closed unmerged, or the `Closes #N` line is removed, the issue is NOT
	# moved back: it keeps whatever Status it has and must be reset by hand.
	#
	# This is a deliberate choice, not an oversight. Nothing distinguishes "the
	# reconciler promoted this" from "a human put this In Progress because they are
	# working on it without a PR yet" — and auto-reverting would sweep the latter
	# back into triage. Since the wrong behaviour there is worse than the stale
	# label, the reconciler declines to guess. The staleness pass (simp/puppetsync#99)
	# is the mechanism that surfaces an issue sitting In Progress with nothing
	# behind it.
	#
	# Linked PRs are read from the issue's own `closedByPullRequestsReferences`, not
	# derived from PRs that happen to be on the board — so an issue is promoted even
	# when its PR was never added as a project item.
	#
	# ==============================================================================
	# Labels
	#
	# `community` (sticky) -- applied to every OPEN issue or PR on the board whose
	# author is a human (`User`, not `Bot`) outside the org member list. Never
	# removed by this job. It is applied here rather than by the per-repo intake
	# workflow because the event payload's `author_association` reports private org
	# members as CONTRIBUTOR, which would have labelled most of the team's own work
	# as outside contributions. Only board items are labelled: an outside PR in a
	# repo without the intake workflow (simp/puppetsync#96) is not seen here.
	#
	# `needs-attention` (transient) -- applied when an item has had no HUMAN
	# activity for longer than its column's threshold, and removed as soon as it
	# has. Thresholds, in days, from the board plan:
	#
	# `community` item (anywhere it is eligible) ... 7
	# New, or no Status .............................. 7 (triage isn't happening)
	# In Review ...................................... 14 (one review cycle)
	# In Progress .................................... 30
	# Todo / Ready / Parked / Blocked / Done ......... never
	#
	# A flat threshold does not work: measured 2026-08-27, "14 days idle" would
	# have flagged 83% of the board. Staleness is only meaningful relative to what
	# the item is waiting on.
	#
	# "Human activity" is the newest of: the item's creation (if by a human), and
	# any comment, review, commit, ready-for-review / convert-to-draft, reopen,
	# assignment, review request or retitle whose actor is a `User`. Actors of type
	# `Bot` are ignored (Renovate rebases, CI comments, Copilot), and LABELED /
	# UNLABELED events are not fetched at all -- so applying `needs-attention` does
	# not reset the clock it measures, and the label cannot flap. A commit whose
	# author has no GitHub account counts as activity (failing towards "fresh").
	#
	# The timeline is read 100 events deep (the API maximum), newest first. If all
	# 100 are bot events, the newest human event is older than the whole window,
	# which yields a LOWER BOUND on idleness rather than the value: the label is
	# added when even that bound exceeds the threshold, and is otherwise left
	# exactly as it is -- neither added nor removed on a guess.
	#
	# The project item's own `updatedAt` is never consulted: it records when the
	# CARD last changed, is floored at 2024-01-25 by past bulk operations, and is
	# blind to anything older.
	#
	# Writes go through the REST labels endpoints (`POST/DELETE
	# /repos/{o}/{r}/issues/{n}/labels`), which serve issues and PRs alike, paced
	# at one per second with a retry on GitHub's secondary rate limit. Both labels
	# were created on every in-scope repo on 2026-08-27 and are org default labels
	# for repos created since. Because the POST endpoint would silently CREATE a
	# missing label, each (repo, label) pair is confirmed to exist before any add;
	# a repo without it is reported as a per-item failure and the run continues.
	#
	# ==============================================================================
	# Credential
	#
	# Prefers the org-owned GitHub App, which needs ALL of:
	#
	# Repository permissions -> Metadata: Read
	# Repository permissions -> Issues: Read and write
	# Repository permissions -> Pull requests: Read and write
	# Organization permissions -> Members: Read
	# Organization permissions -> Projects: Read and write
	#
	# ...and an installation covering every repository represented on the board.
	# Issues/Pull requests need WRITE for the labels pass; the Status pass alone
	# works with Read. (The PAT fallback needs the `repo` scope for the same
	# reason.) A credential that can read but not write labels completes the
	# Status pass and then aborts on the first label write, so the board's Status
	# is never left behind by a permission gap in the newer pass.
	#
	# The Projects grant alone is not enough: it lets the App read the project and
	# write Status, but the mapping is computed from the issues and pull requests
	# behind the cards. Without Issues/Pull requests read, `content` comes back null
	# for those items, the App passes the probe, and the run aborts on the
	# classification guard below -- which is exactly what the first live dry run
	# did, proposing 13 moves where a full-read credential computes 44.
	#
	# The triage board is an organization project, so repository permissions alone
	# are not enough either. The App token is probed against the project before use; if
	# the probe fails the job falls back to SIMP_PROJECT_TOKEN (a PAT with the
	# project scope). Probing matters because an App token that exists but is
	# refused would otherwise abort the run on its first read, with a usable PAT
	# sitting unused.
	#
	# Note the probe is a read: an App granted only `Projects: Read` passes it
	# and then fails on the first mutation, which is treated as systemic and aborts
	# the run. Read/write is therefore the requirement, not merely read.
	#
	---
	name: 'ORG: Reconcile triage board Status'

	on:
	schedule:
	# Hourly at :23. A handful of API calls, and it keeps the board close enough
	# to live state that the biweekly review starts from something true.
	- cron: '23 * * * *'
	workflow_dispatch:
	inputs:
	dry_run:
	description: 'Dry run (report the moves it would make, change nothing)'
	required: false
	type: boolean
	default: false

	permissions: {}

	env:
	ORG: simp
	PROJECT_NUMBER: '11'

	jobs:
	reconcile:
	name: 'Reconcile Status from live PR state'
	runs-on: ubuntu-latest
	env:
	APP_ID: ${{ secrets.SIMP_APP_ID }}
	steps:
	- name: 'Mint a GitHub App installation token'
	id: app-token
	if: env.APP_ID != ''
	uses: actions/create-github-app-token@v2
	with:
	app-id: ${{ secrets.SIMP_APP_ID }}
	private-key: ${{ secrets.SIMP_APP_PRIVATE_KEY }}
	owner: ${{ github.repository_owner }}

	# Both candidate credentials stay inside this one step on purpose: passing
	# a token through a step output would place it in the run's output data.
	- name: 'Reconcile'
	env:
	APP_TOKEN: ${{ steps.app-token.outputs.token }}
	PAT: ${{ secrets.SIMP_PROJECT_TOKEN }}
	DRY_RUN: ${{ inputs.dry_run }}
	run: \|
	# shellcheck disable=SC2016
	set -euo pipefail

	# shellcheck disable=SC2016
	META_QUERY='
	query($org:String!,$num:Int!){
	organization(login:$org){ projectV2(number:$num){
	id
	field(name:"Status"){ ... on ProjectV2SingleSelectField{
	id options{ id name } } } } } }'

	# ---- pick a credential that actually works ------------------------
	probe() {
	# -f (not -F) for string variables: gh's -F coerces all-digit
	# values to JSON numbers, and single-select option IDs are 8-char
	# hex, so ~2% of them are all digits. $num is genuinely Int!.
	GH_TOKEN="$1" gh api graphql -f query="$META_QUERY" \
	-f org="$ORG" -F num="$PROJECT_NUMBER" > meta.json 2>/dev/null
	}

	GH_TOKEN=''
	CREDENTIAL=''
	app_refused='no'

	if [ -n "${APP_TOKEN:-}" ]; then
	if probe "$APP_TOKEN"; then
	echo 'Credential: GitHub App installation token.'
	GH_TOKEN="$APP_TOKEN"
	CREDENTIAL='app'
	else
	app_refused='yes'
	fi
	fi

	if [ -z "$GH_TOKEN" ] && [ -n "${PAT:-}" ] && probe "$PAT"; then
	if [ "$app_refused" = 'yes' ]; then
	printf '::warning ::App token was refused by the project API; falling '
	printf 'back to SIMP_PROJECT_TOKEN. Grant the App '
	printf '"Organization permissions -> Projects: Read and write".\n'
	else
	printf '::warning ::No App credential; using SIMP_PROJECT_TOKEN.\n'
	fi
	GH_TOKEN="$PAT"
	CREDENTIAL='pat'
	fi

	if [ -z "$GH_TOKEN" ]; then
	printf '::error ::No credential can read the org project. Grant the App '
	printf '"Organization permissions -> Projects: Read and write", or set '
	printf 'SIMP_PROJECT_TOKEN to a PAT with the project scope.\n'
	exit 1
	fi
	export GH_TOKEN CREDENTIAL

	PROJECT_ID="$(jq -r '.data.organization.projectV2.id' meta.json)"
	FIELD_ID="$(jq -r '.data.organization.projectV2.field.id // ""' meta.json)"

	# Guard the field itself, not only its options: if Status is renamed or
	# is no longer a single-select, field(name:"Status") is null and the
	# options iteration below would die with an unexplained jq error.
	if [ -z "$FIELD_ID" ] \|\| [ "$FIELD_ID" = 'null' ]; then
	printf '::error ::The project has no single-select field named "Status". '
	printf 'This job keys every decision off that field, so it refuses to run.\n'
	exit 1
	fi

	jq -r '.data.organization.projectV2.field.options[]\|[.name,.id]\|@tsv' meta.json > options.tsv

	opt() { awk -F'\t' -v n="$1" '$1==n{print $2}' options.tsv; }

	# Destinations must exist, and so must the three names that STOP this
	# job from overriding human placements -- otherwise a column rename
	# silently disarms the guard instead of failing the run.
	# Explicit if/fi rather than a `&&` list: the list form is actually
	# safe under `set -e` (a failing command inside a && list is exempt,
	# so the loop's non-zero status does not exit the shell), but it reads
	# like a bug and would become one if a command were ever appended
	# after it in the loop body.
	missing=''
	for required in 'In Progress' 'In Review' 'Blocked' 'Parked' 'Done'; do
	if [ -z "$(opt "$required")" ]; then
	missing="${missing} '${required}'"
	fi
	done
	if [ -n "$missing" ]; then
	printf '::error ::Status option(s) missing from the board:%s. Renaming a column changes this behaviour, so the job refuses to run.\n' \
	"$missing"
	exit 1
	fi

	# ---- every item, with live PR state --------------------------------
	# Fetched raw rather than through --jq so the response body is on disk
	# when things go wrong. `gh api graphql` exits non-zero whenever the
	# body carries a top-level `errors` array -- even on HTTP 200 with
	# partial `data` -- but it writes the body to stdout before failing,
	# so raw.json still holds the detail we want to report.
	# shellcheck disable=SC2016
	if ! gh api graphql --paginate -f query='
	query($org:String!,$num:Int!,$endCursor:String){
	organization(login:$org){ projectV2(number:$num){
	items(first:100, after:$endCursor){
	pageInfo{ hasNextPage endCursor }
	nodes{
	id
	type
	fieldValueByName(name:"Status"){
	... on ProjectV2ItemFieldSingleSelectValue{ name } }
	content{
	__typename
	... on DraftIssue{ title }
	... on PullRequest{
	number isDraft reviewDecision state createdAt
	author{ login __typename }
	repository{ nameWithOwner }
	labels(first:50){ nodes{ name } }
	timelineItems(last:100, itemTypes:[
	ISSUE_COMMENT, PULL_REQUEST_REVIEW, PULL_REQUEST_COMMIT,
	READY_FOR_REVIEW_EVENT, CONVERT_TO_DRAFT_EVENT, REOPENED_EVENT,
	ASSIGNED_EVENT, REVIEW_REQUESTED_EVENT, RENAMED_TITLE_EVENT]){
	nodes{
	__typename
	... on IssueComment{ createdAt author{ login __typename } }
	... on PullRequestReview{ createdAt author{ login __typename } }
	... on PullRequestCommit{ commit{ committedDate author{ user{ login __typename } } } }
	... on ReadyForReviewEvent{ createdAt actor{ login __typename } }
	... on ConvertToDraftEvent{ createdAt actor{ login __typename } }
	... on ReopenedEvent{ createdAt actor{ login __typename } }
	... on AssignedEvent{ createdAt actor{ login __typename } }
	... on ReviewRequestedEvent{ createdAt actor{ login __typename } }
	... on RenamedTitleEvent{ createdAt actor{ login __typename } } } } }
	... on Issue{
	number state createdAt repository{ nameWithOwner }
	author{ login __typename }
	labels(first:50){ nodes{ name } }
	timelineItems(last:100, itemTypes:[
	ISSUE_COMMENT, REOPENED_EVENT, ASSIGNED_EVENT, RENAMED_TITLE_EVENT]){
	nodes{
	__typename
	... on IssueComment{ createdAt author{ login __typename } }
	... on ReopenedEvent{ createdAt actor{ login __typename } }
	... on AssignedEvent{ createdAt actor{ login __typename } }
	... on RenamedTitleEvent{ createdAt actor{ login __typename } } } }
	closedByPullRequestsReferences(first:20, includeClosedPrs:false){
	nodes{ number state author{ login } } } } } } } } } }' \
	-f org="$ORG" -F num="$PROJECT_NUMBER" > raw.json 2> gh_stderr.txt; then

	gql_errors="$(jq -s '[.[].errors // []] \| flatten \| length' raw.json 2>/dev/null \|\| echo 0)"
	if [ "$gql_errors" -gt 0 ]; then
	printf '::error ::The items query returned %s GraphQL error(s), so the board view is incomplete. Refusing to reconcile a partial view.\n' \
	"$gql_errors"
	jq -rs '[.[].errors // []] \| flatten \| .[0:5][]
	\| "::error :: " + (.message // "unknown error")' raw.json \|\| true
	else
	printf '::error ::Could not read the project items.\n'
	sed -e 's/^/::error :: /' gh_stderr.txt \| head -5 \|\| true
	fi
	exit 1
	fi

	# `// []` guards the rare `data: null` page: without it jq exits 5 with
	# a bare "Cannot iterate over null" and no annotation, unlike every
	# other failure path here.
	jq -s '[ .[] \| (.data.organization.projectV2.items.nodes // [])[] ]' raw.json > items.json
	total="$(jq 'length' items.json)"

	if [ "$total" -eq 0 ]; then
	printf '::error ::The project returned no items. That is never correct for this board, so the response is being treated as unusable rather than as an empty board.\n'
	exit 1
	fi

	# ---- classify every item positively -------------------------------
	# A negative "is it broken?" test leaves anything matching neither
	# branch invisible to both this guard and the mapping below -- the same
	# silent-skip shape this guard exists to close. So every item is put in
	# exactly one bucket, and the buckets are asserted to sum to the total.
	#
	# ProjectV2Item.type is always present (ISSUE, PULL_REQUEST,
	# DRAFT_ISSUE, REDACTED), which makes it a safer discriminator than
	# content, which is null precisely when we cannot see it.
	jq -r '
	[ .[]
	\| { id,
	t: .type,
	cls: (
	if .type == "DRAFT_ISSUE" then "draft"
	elif .type == "REDACTED" then "redacted"
	elif (.type == "ISSUE" or .type == "PULL_REQUEST") then
	(if .content == null then "unreadable" else "readable" end)
	else "unknown" end ) } ]
	\| .[] \| [.cls, .t, .id] \| @tsv
	' items.json > classes.tsv

	count() { awk -F'\t' -v c="$1" '$1==c{n++} END{print n+0}' classes.tsv; }
	drafts="$(count draft)"; gone="$(count gone)"
	readable="$(count readable)"; unreadable="$(count unreadable)"
	unknown="$(count unknown)"

	echo "board items: ${total} (readable ${readable}, draft ${drafts}, unreadable ${unreadable}, gone ${gone}, unknown ${unknown})"

	# ---- resolve org membership authoritatively ------------------------
	# NOT from authorAssociation: that field is credential-dependent. An App
	# installation token reading a public repo gets the same degraded answer
	# as an anonymous client -- verified on simp/pupmod-simp-ssh#229, where a
	# user PAT reports MEMBER while both an App token and an unauthenticated
	# request report CONTRIBUTOR. Trusting it made the mapping treat 29 org
	# members as outside contributors and skip them in silence.
	#
	# Needs Organization permissions -> Members: Read.
	# shellcheck disable=SC2016
	if ! gh api graphql --paginate -f query='
	query($org:String!,$endCursor:String){
	organization(login:$org){
	membersWithRole(first:100, after:$endCursor){
	pageInfo{ hasNextPage endCursor }
	nodes{ login } } } }' \
	-f org="$ORG" > members_raw.json 2> members_err.txt; then
	printf '::error ::Could not read the organization member list, which this job needs in order to tell members from outside contributors. Grant the credential Organization permissions -> "Members: Read".\n'
	sed -e 's/^/::error :: /' members_err.txt \| head -5 \|\| true
	exit 1
	fi

	jq -s '[ .[] \| (.data.organization.membersWithRole.nodes // [])[].login ] \| unique' \
	members_raw.json > members.json
	member_count="$(jq 'length' members.json)"
	echo "org members resolved: ${member_count}"

	# An empty list is the obvious dangerous case: every author would look
	# like an outside contributor, every move would be skipped, and the run
	# would report success having done nothing.
	if [ "$member_count" -eq 0 ]; then
	printf '::error ::The organization member list came back empty. Every author would then look like an outside contributor and be skipped, so this run would do nothing and still report success. Grant the credential Organization permissions -> "Members: Read".\n'
	exit 1
	fi

	# The subtler and more likely case: a partial list. Without
	# Members: Read, membersWithRole still succeeds -- it just returns only
	# the publicly-visible members, with no error and a non-zero count. A
	# count-above-zero check sails straight through it, and the private
	# members' pull requests get silently classified as outside
	# contributions.
	#
	# This is not hypothetical for simp: 12 of 32 members are public, and
	# two of the most prolific board authors are private members. A
	# credential lacking the grant would therefore resolve 12, skip the
	# other 20's work, and report success.
	#
	# public_members is readable without any grant, so comparing the two
	# detects the condition. Assumes the org has at least one private
	# member -- true here (20 of 32), and if that ever stops being true
	# this check aborts a healthy run rather than passing a broken one.
	# Read into a file and count separately, rather than
	# `... \| grep -c . \|\| echo 0`. That idiom fails OPEN: `grep -c` prints
	# 0 and then exits 1 on no match, so `\|\| echo 0` appends a second
	# line, `public_count` becomes "0\n0", the comparison below dies with
	# "integer expression expected" and returns 2 -- which inside an `if`
	# counts as false and is exempt from `set -e`. The guard is skipped and
	# the run proceeds on a partial member list. Same shape when the API
	# call itself fails under `pipefail`.
	if ! gh api "orgs/${ORG}/public_members" --paginate --jq '.[].login' \
	> public_members.txt 2> public_members_err.txt; then
	printf '::error ::Could not read the public organization member list, which this job needs in order to tell a partial member list from a complete one. Refusing to proceed.\n'
	sed -e 's/^/::error :: /' public_members_err.txt \| head -5 \|\| true
	exit 1
	fi
	public_count="$(sort -u public_members.txt \| wc -l \| tr -d ' ')"
	echo "org members: ${member_count} resolved, ${public_count} publicly visible"

	if [ "$member_count" -le "$public_count" ]; then
	printf '::error ::Resolved %s org member(s) but %s are publicly visible, so this credential appears to see only public members. Private members would be misclassified as outside contributors and their work skipped silently. Grant the credential Organization permissions -> "Members: Read".\n' \
	"$member_count" "$public_count"
	{
	echo '## Board Status reconcile'
	echo
	echo "Aborted: the credential cannot see private org members."
	echo
	echo "Resolved ${member_count} members, of which ${public_count} are"
	echo 'public. Without Organization permissions -> Members: Read,'
	echo 'private members are indistinguishable from outside contributors,'
	echo 'and their pull requests would be skipped with no error.'
	} >> "$GITHUB_STEP_SUMMARY"
	exit 1
	fi

	echo "open PR authors, by membership:"
	jq -r --slurpfile M members.json '
	($M[0]) as $members
	\| [ .[] \| select(.content != null)
	\| select(.content.__typename == "PullRequest")
	\| select(.content.state == "OPEN")
	\| .content.author.login as $item_author
	\| (if any($members[]; . == $item_author)
	then "member" else "outside" end) ]
	\| group_by(.) \| map({k: .[0], n: length}) \| .[]
	\| " \(.k): \(.n)"' items.json

	if [ "$((readable + drafts + gone + unreadable + unknown))" -ne "$total" ]; then
	printf '::error ::Item classification does not account for all %s items; refusing to proceed.\n' "$total"
	exit 1
	fi

	if [ "$unknown" -gt 0 ]; then
	printf '::error ::%s item(s) have a ProjectV2Item.type this workflow does not know how to classify. Refusing to reconcile, because an unclassified item would be skipped silently.\n' \
	"$unknown"
	awk -F'\t' '$1=="unknown"{printf "::error :: %s (type=%s)\n", $3, $2}' classes.tsv
	exit 1
	fi

	# An ISSUE/PULL_REQUEST item whose content is null despite a readable
	# type is content that is genuinely gone -- deleted, or transferred
	# somewhere the installation no longer reaches. No permission grant
	# fixes that, so it is reported loudly and skipped: aborting would
	# deadlock the board until a human found and removed the card.
	if [ "$gone" -gt 0 ]; then
	printf '::warning ::%s board item(s) reference content that no longer resolves (deleted, or moved out of reach). They are skipped -- no permission grant will recover them, so remove the cards to silence this.\n' \
	"$gone"
	awk -F'\t' '$1=="gone"{printf "::warning :: %s (type=%s)\n", $3, $2}' classes.tsv \| head -20 \|\| true
	fi

	if [ "$unreadable" -gt 0 ]; then
	printf '::error ::%s of %s board items came back REDACTED, which is how the API reports an item the credential is not allowed to view. The board view is incomplete, so reconciling it would apply some moves and skip the rest silently.\n' \
	"$unreadable" "$total"
	if [ "$CREDENTIAL" = 'app' ]; then
	printf '::error ::Running as the GitHub App: grant it Repository permissions -> "Issues: Read" and "Pull requests: Read", and check the installation covers every repo on the board.\n'
	else
	printf '::error ::Running as SIMP_PROJECT_TOKEN: that PAT needs read access to the repositories behind these items.\n'
	fi
	awk -F'\t' '$1=="unreadable"{printf "::error :: %s\n", $3}' classes.tsv \| head -20 \|\| true
	{
	echo '## Board Status reconcile'
	echo
	echo "Aborted: ${unreadable} of ${total} items could not be read."
	echo
	if [ "$CREDENTIAL" = 'app' ]; then
	echo 'Running as the GitHub App. Grant it **Repository'
	echo 'permissions -> Issues: Read and Pull requests: Read**,'
	echo 'and confirm the installation covers every repo on the board.'
	else
	echo 'Running as SIMP_PROJECT_TOKEN. That PAT needs read access'
	echo 'to the repositories behind these items; granting App'
	echo 'permissions will not change this run.'
	fi
	echo
	echo 'Reconciling a partial view would apply some moves, skip the'
	echo 'rest silently, and report success.'
	} >> "$GITHUB_STEP_SUMMARY"
	exit 1
	fi

	# ---- compute desired Status ---------------------------------------
	# No `\|\| true` here: jq exits non-zero only on real breakage, and
	# swallowing that would leave moves.tsv empty and report "nothing to
	# do" every hour, forever.
	# shellcheck disable=SC2016
	jq -r --slurpfile M members.json '
	($M[0]) as $members
	\| def member($login): ($login != null) and any($members[]; . == $login);

	.[]
	\| . as $item
	\| ((.fieldValueByName.name) // "(none)") as $cur
	\| (.content.__typename) as $t
	\| (if ($t == "PullRequest" or $t == "Issue")
	then "\(.content.repository.nameWithOwner)#\(.content.number)"
	elif ($t == "DraftIssue")
	then "draft: \(.content.title)"
	else "item \($item.id)" end) as $label
	\| (
	# Never override a deliberate human placement.
	if ($cur == "Blocked" or $cur == "Parked" or $cur == "Done") then null

	elif $t == "PullRequest" then
	if (.content.state != "OPEN") then null
	elif ((member(.content.author.login)) \| not) then null
	elif .content.isDraft then "In Progress"
	elif (.content.reviewDecision == "CHANGES_REQUESTED") then "In Progress"
	else "In Review"
	end

	elif $t == "Issue" then
	if (.content.state != "OPEN") then null
	elif ([ .content.closedByPullRequestsReferences.nodes[]?
	\| select(.state == "OPEN")
	\| select(member(.author.login)) ] \| length) > 0
	then "In Progress"
	else null # one-way: never demote. See header.
	end

	else null end
	) as $want
	\| select($want != null and $want != $cur)
	\| [$item.id, $label, $cur, $want] \| @tsv
	' items.json > moves.tsv

	moves="$(wc -l < moves.tsv \| tr -d ' ')"
	echo "moves required: ${moves}"

	{
	echo '## Board Status reconcile'
	[ "${DRY_RUN}" = 'true' ] && echo 'Dry run — nothing changed.'
	echo
	if [ "${moves}" = '0' ]; then
	echo 'Board already matches live PR state; no moves required.'
	else
	echo '\| Item \| From \| To \|'
	echo '\|---\|---\|---\|'
	awk -F'\t' '{printf "\| %s \| %s \| %s \|\n", $2, $3, $4}' moves.tsv
	fi
	} >> "$GITHUB_STEP_SUMMARY"

	# ---- apply Status --------------------------------------------------
	applied=0; failed=0
	: > failures.txt
	if [ "${moves}" = '0' ] \|\| [ "${DRY_RUN}" = 'true' ]; then
	awk -F'\t' '{printf " would move %-46s %s -> %s\n", $2, $3, $4}' moves.tsv
	else
	# Read on fd 3: nothing in this loop currently consumes stdin, but a
	# command that did would silently swallow the remaining moves.
	while IFS=$'\t' read -r item_id label cur want <&3; do
	option_id="$(opt "$want")"
	err=''
	# shellcheck disable=SC2016
	if err="$(gh api graphql -f query='
	mutation($p:ID!,$i:ID!,$f:ID!,$o:String!){
	updateProjectV2ItemFieldValue(input:{
	projectId:$p, itemId:$i, fieldId:$f,
	value:{ singleSelectOptionId:$o } }){
	projectV2Item{ id } } }' \
	-f p="$PROJECT_ID" -f i="$item_id" -f f="$FIELD_ID" -f o="$option_id" \
	2>&1 >/dev/null)"; then
	printf ' moved %-46s %s -> %s\n' "$label" "$cur" "$want"
	applied=$((applied + 1))
	else
	trimmed="$(printf '%s' "$err" \| tr -d '\n' \| cut -c1-160)"
	case "$err" in
	FORBIDDEN\|INSUFFICIENT_SCOPES\|'not have permission'\|'Resource not accessible')
	# Systemic: every remaining move would fail the same way.
	# Abort now rather than making 40-odd more pointless calls.
	printf '::error ::%s: credential cannot write the org project: %s\n' \
	"$label" "$trimmed"
	printf '::error ::Aborting after the first permission failure. Grant the '
	printf 'App "Organization permissions -> Projects: Read and write".\n'
	{ echo
	echo "Aborted before any move was applied — the credential"
	echo "cannot write the org project. None of the moves listed"
	echo "above were made."
	} >> "$GITHUB_STEP_SUMMARY"
	exit 1 ;;
	*)
	printf '::error ::%s: failed to set Status to %s: %s\n' \
	"$label" "$want" "$trimmed"
	printf '%s\t%s\n' "$label" "$trimmed" >> failures.txt
	failed=$((failed + 1)) ;;
	esac
	fi
	done 3< moves.tsv

	echo "applied=${applied} failed=${failed}"
	{
	echo
	echo "applied ${applied} · failed ${failed}"
	if [ "$failed" -gt 0 ]; then
	echo
	echo '### Failures'
	echo
	awk -F'\t' '{printf "- `%s` — %s\n", $1, $2}' failures.txt
	fi
	} >> "$GITHUB_STEP_SUMMARY"
	fi

	# ---- compute label changes ----------------------------------------
	# `community` and `needs-attention`; see the Labels section of the
	# header for the rules. Computed from the same items.json snapshot as
	# the Status pass, with each moved item's Status patched to the value
	# the Status pass just set (or would set, in a dry run). Thresholds are
	# therefore keyed off the column the item is now in, not the one it
	# left -- otherwise a New -> In Review promotion would add the label
	# against New's 7d and remove it against In Review's 14d an hour later.
	#
	# Columns: repo, number, op (add\|remove), label, item, reason.
	jq -Rs 'split("\n") \| map(select(length > 0) \| split("\t") \| {key: .[0], value: .[3]}) \| from_entries' \
	moves.tsv > moves.json
	# shellcheck disable=SC2016
	jq -r --slurpfile M members.json --slurpfile MV moves.json \
	--arg now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" '
	($M[0]) as $members
	\| ($MV[0]) as $moved
	\| def member($login): ($login != null) and any($members[]; . == $login);
	def human($a): ($a != null) and ($a.__typename == "User");
	def days_between($from; $to): ((($to \| fromdateiso8601) - ($from \| fromdateiso8601)) / 86400 \| floor);

	.[]
	\| select(.content != null)
	\| select(.content.__typename == "PullRequest" or .content.__typename == "Issue")
	\| ($moved[.id] // .fieldValueByName.name // "(none)") as $cur
	\| .content as $c
	\| "\($c.repository.nameWithOwner)#\($c.number)" as $ref
	\| ([$c.labels.nodes[]?.name]) as $labels
	\| ($c.state == "OPEN") as $open
	\| (human($c.author) and ((member($c.author.login)) \| not)) as $outside

	# Timeline events, normalised to {at, human}. A commit whose author
	# has no GitHub account counts as human (failing towards "fresh").
	\| ([ $c.timelineItems.nodes[]?
	\| if .__typename == "PullRequestCommit"
	then { at: .commit.committedDate,
	human: ((.commit.author.user == null) or human(.commit.author.user)) }
	else { at: .createdAt, human: human(.author // .actor) }
	end ]) as $events
	\| ([ $events[] \| select(.human) \| .at ]) as $human_at
	# The window holds the NEWEST events of the selected types. If it is
	# full and none of them is human, every human event is older than
	# the whole window: that gives a lower bound on idleness, not the
	# value. If it is not full, it is the complete history, and creation
	# by a human is the floor.
	\| (($events \| length) >= 100) as $saturated
	\| (if ($human_at \| length) > 0 then ($human_at \| max)
	elif $saturated then null
	elif human($c.author) then $c.createdAt
	else null end) as $last_human
	\| (if $last_human != null then days_between($last_human; $now)
	elif $saturated then days_between(($events \| map(.at) \| min); $now)
	else null end) as $idle
	\| ($last_human == null and $idle != null) as $idle_is_lower_bound

	\| (if ($cur == "Todo" or $cur == "Ready" or $cur == "Parked"
	or $cur == "Blocked" or $cur == "Done") then null
	elif ($open \| not) then null
	elif ($outside or ($labels \| index("community") != null)) then 7
	elif ($cur == "New" or $cur == "(none)") then 7
	elif $cur == "In Review" then 14
	elif $cur == "In Progress" then 30
	else null end) as $threshold
	# A lower bound past the threshold is still past the threshold.
	\| ($threshold != null and $idle != null and $idle >= $threshold) as $want_attn
	\| ($labels \| index("needs-attention") != null) as $has_attn
	# Remove only on a positive signal: closed, a column with no
	# threshold (the label is job-owned; see header), or human activity
	# KNOWN to be inside the threshold. A lower bound below the
	# threshold proves nothing, so the label is left alone.
	\| ($has_attn and (($open \| not) or $threshold == null
	or ($idle != null and ($idle_is_lower_bound \| not) and $idle < $threshold))) as $drop_attn

	\| (if ($outside and $open and ($labels \| index("community") == null)) then
	[$c.repository.nameWithOwner, $c.number, "add", "community", $ref,
	"author \($c.author.login) is outside the org"]
	else empty end),
	(if ($want_attn and ($has_attn \| not)) then
	[$c.repository.nameWithOwner, $c.number, "add", "needs-attention", $ref,
	(if $idle_is_lower_bound
	then "no human among the newest 100 events; idle at least \($idle)d (threshold \($threshold)d, Status \($cur))"
	else "no human activity for \($idle)d (threshold \($threshold)d, Status \($cur))" end)]
	elif $drop_attn then
	[$c.repository.nameWithOwner, $c.number, "remove", "needs-attention", $ref,
	(if ($open \| not) then "closed"
	elif $threshold == null then "Status \($cur) has no threshold (label is job-owned)"
	else "human activity \($idle)d ago (threshold \($threshold)d)" end)]
	else empty end)
	\| @tsv
	' items.json > label_ops.tsv

	label_ops="$(wc -l < label_ops.tsv \| tr -d ' ')"
	echo "label changes required: ${label_ops}"

	{
	echo
	echo '### Labels'
	echo
	if [ "${label_ops}" = '0' ]; then
	echo 'Labels already match; no changes required.'
	else
	echo '\| Item \| Change \| Reason \|'
	echo '\|---\|---\|---\|'
	awk -F'\t' '{ sign = ($3 == "add") ? "+" : "-";
	printf "\| %s \| %s`%s` \| %s \|\n", $5, sign, $4, $6 }' label_ops.tsv
	fi
	} >> "$GITHUB_STEP_SUMMARY"

	# ---- apply labels --------------------------------------------------
	lapplied=0; lfailed=0
	: > label_failures.txt
	if [ "${label_ops}" = '0' ] \|\| [ "${DRY_RUN}" = 'true' ]; then
	awk -F'\t' '{ sign = ($3 == "add") ? "+" : "-";
	printf " would label %-46s %s%s (%s)\n", $5, sign, $4, $6 }' label_ops.tsv
	else
	# POST .../issues/N/labels CREATES a label that does not exist in
	# the repo (arbitrary colour, no description). This job must never
	# do that, so each (repo, label) pair is confirmed to exist once per
	# run before any add; a missing one is a per-item failure.
	declare -A LABEL_DEFINED=()
	label_defined() { # $1 repo, $2 label -> 0 if the repo defines it
	local key="$1/$2"
	if [ -z "${LABEL_DEFINED[$key]:-}" ]; then
	if gh api "repos/$1/labels/$2" >/dev/null 2>&1; then
	LABEL_DEFINED[$key]='yes'
	else
	LABEL_DEFINED[$key]='no'
	fi
	fi
	[ "${LABEL_DEFINED[$key]}" = 'yes' ]
	}

	# A mutating call, retried on the secondary rate limit. GitHub
	# reports that limit as HTTP 403 with a body naming it -- the same
	# status as a permission failure -- and a tight loop of 90-odd writes
	# is exactly what trips it. Sets $err; returns gh's status.
	api_write() {
	local attempt rc=1
	for attempt in 1 2 3; do
	err="$(gh api "$@" 2>&1 >/dev/null)" && return 0 \|\| rc=$?
	case "$err" in
	'secondary rate limit'\|'abuse detection'\|'rate limit exceeded')
	printf '::warning ::rate limited by GitHub; waiting 60s before retry %s/3\n' "$attempt"
	sleep 60 ;;
	*) return "$rc" ;;
	esac
	done
	return "$rc"
	}

	while IFS=$'\t' read -r repo number op name item reason <&3; do
	err=''
	if [ "$op" = 'add' ]; then
	if label_defined "$repo" "$name"; then
	api_write -X POST "repos/${repo}/issues/${number}/labels" -f "labels[]=${name}" && rc=0 \|\| rc=$?
	else
	err="label '${name}' is not defined in ${repo}; this job never creates labels"
	rc=1
	fi
	else
	api_write -X DELETE "repos/${repo}/issues/${number}/labels/${name}" && rc=0 \|\| rc=$?
	# Only the 404 that says the label is already absent is the
	# desired state. A bare "Not Found" 404 is how GitHub reports an
	# issue or repo the credential cannot reach (deliberately, for
	# private repos), and that is a failure.
	case "$err" in 'Label does not exist') rc=0 ;; esac
	fi
	# At least one second between mutating requests, per GitHub's
	# guidance on avoiding the secondary rate limit.
	sleep 1
	if [ "$rc" -eq 0 ]; then
	if [ "$op" = 'remove' ]; then sign='-'; else sign='+'; fi
	printf ' labelled %-46s %s%s (%s)\n' "$item" "$sign" "$name" "$reason"
	lapplied=$((lapplied + 1))
	else
	trimmed="$(printf '%s' "$err" \| tr -d '\n' \| cut -c1-160)"
	case "$err" in
	'HTTP 403'\|'Resource not accessible')
	# Systemic: the credential can read but not write labels.
	printf '::error ::%s: credential cannot write labels: %s\n' "$item" "$trimmed"
	printf '::error ::Aborting the labels pass after the first permission failure. '
	printf 'Grant the App Repository permissions -> "Issues: Read and write" and '
	printf '"Pull requests: Read and write" (a PAT needs the repo scope). '
	printf 'The Status pass above completed normally.\n'
	{ echo
	echo 'Labels pass aborted — the credential cannot write labels.'
	echo 'Grant the App Repository permissions -> Issues: Read and write'
	echo 'and Pull requests: Read and write. The Status pass was not'
	echo 'affected.'
	} >> "$GITHUB_STEP_SUMMARY"
	exit 1 ;;
	*)
	printf '::error ::%s: failed to %s %s: %s\n' "$item" "$op" "$name" "$trimmed"
	printf '%s\t%s %s: %s\n' "$item" "$op" "$name" "$trimmed" >> label_failures.txt
	lfailed=$((lfailed + 1)) ;;
	esac
	fi
	done 3< label_ops.tsv

	echo "labels applied=${lapplied} failed=${lfailed}"
	{
	echo
	echo "labels applied ${lapplied} · failed ${lfailed}"
	if [ "$lfailed" -gt 0 ]; then
	echo
	echo '#### Label failures'
	echo
	awk -F'\t' '{printf "- `%s` — %s\n", $1, $2}' label_failures.txt
	fi
	} >> "$GITHUB_STEP_SUMMARY"
	fi

	if [ "$failed" -gt 0 ] \|\| [ "$lfailed" -gt 0 ]; then
	printf '::error ::%s Status update(s) and %s label change(s) failed\n' "$failed" "$lfailed"
	exit 1
	fi

Navigation Menu

Uh oh!

ORG: Reconcile triage board Status #29

Workflow file

ORG: Reconcile triage board Status #29

Uh oh!

Workflow file for this run