Fail fast when OIDC credentials cannot be obtained (#31)

The `Configure AWS credentials` step took 82 minutes to fail on a
cyber-dojo/dashboard run. Each of the twelve default retry attempts sat
on a TCP connect to the STS endpoint that took ~6m45s to time out at the
OS level. `connect ETIMEDOUT` on port 443 means no socket was ever
opened, so nothing was going to succeed — but the runner was held and
billed for 82 minutes, and the caller's `terraform-<name>` concurrency
group was held with it, queueing every subsequent apply for that
environment behind a job that was already doomed.

Bound the step so it dies quickly instead:

action-timeout-s: 45 bounds the action as a whole, retries included
retry-max-attempts: 3 the default of 12 only helps if attempts are fast
timeout-minutes: 2 backstop, independent of the action's behaviour

45 seconds is justified by measurement: across recent successful runs in
kosli-dev and cyber-dojo the step took 0-2 seconds, worst case 6, so the
timeout sits far above the real p99 and will not fail a slow-but-healthy
authentication. `disable-retry` is deliberately not used — STS
throttling is genuinely transient and a couple of quick retries is worth
having now that their total cost is bounded.

Every job also gets a `timeout-minutes` rather than silently inheriting
GitHub's 360-minute default, which an 82-minute hang would never have
tripped. For the plan/apply job the ceiling follows from
`aws_role_duration` rather than from how long Terraform might run: the
credentials are static environment variables that are never refreshed,
so 20 minutes (the 1200s default) after the credentials step every AWS
call fails with ExpiredToken and the work cannot usefully continue.
Setup before that step measures 1-35s, so a job consuming its whole
credential lifetime lands around 22-23 minutes — hence a default of 30,
which is headroom over the real ceiling rather than an arbitrary number.
Housekeeping jobs get 5-10.

Because that ceiling is coupled to `aws_role_duration`, it is exposed as
a `job_timeout_minutes` input on plan.yml, apply.yml and
detect-drift.yml rather than hard-coded. A repository with a
legitimately long apply raises both together, without needing a change
here. Raising only the session duration would let the job be cancelled
part-way through an apply, which can leave the state lock held.

Beyond base.yml and apply.yml named in the ticket, detect-drift.yml has
two `Configure AWS credentials` steps with the same failure mode, so
they are covered here too. The values are recorded in the README as the
standard for other repositories to adopt.

Author: gsavage

.github/workflows/apply.yml

@@ -58,6 +58,10 @@ on:
         description: "When true, pass `--commit <ref>` to the Kosli commands via the COMMIT_ARG environment variable."
         default: false
         type: boolean
+      job_timeout_minutes:
+        description: "Minutes before GitHub cancels the apply job. Raise it together with aws_role_duration when a legitimate apply needs longer; see the README on timeouts."
+        default: 30
+        type: number
     secrets:
       kosli_api_token:
         description: "Kosli API token. Required when kosli_template_file is set."
@@ -96,12 +100,14 @@ jobs:
       kosli_flow: terraform-apply-${{ inputs.environment }}-${{ github.event.repository.name }}
       kosli_cli_version: ${{ inputs.kosli_cli_version }}
       pass_commit_to_kosli_commands: ${{ inputs.pass_commit_to_kosli_commands }}
+      job_timeout_minutes: ${{ inputs.job_timeout_minutes }}
     secrets:
       kosli_api_token: ${{ secrets.kosli_api_token }}
 
   reset-drift-detection:
     needs: apply
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     permissions:
       id-token: write
       contents: read
@@ -121,12 +127,15 @@ jobs:
           ref: ${{ inputs.ref }}
 
       - name: Configure AWS credentials
+        timeout-minutes: 2
         uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
         with:
           role-to-assume: ${{ inputs.aws_role_arn }}
           aws-region: ${{ inputs.aws_region }}
           role-duration-seconds: ${{ inputs.aws_role_duration }}
           role-session-name: ${{ github.event.repository.name }}
+          action-timeout-s: 45
+          retry-max-attempts: 3
 
       - name: Compute state bucket name
         run: |

.github/workflows/base.yml

@@ -21,6 +21,10 @@ on:
       tf_apply:
         default: false
         type: boolean
+      job_timeout_minutes:
+        description: "Minutes before GitHub cancels the plan/apply job. The default follows aws_role_duration: the OIDC credentials are static environment variables that are never refreshed, so AWS calls start failing with ExpiredToken 1200s (20 min) after the credentials step and a longer ceiling would buy nothing. Raise both together if a legitimate apply needs longer."
+        default: 30
+        type: number
       environment:
         required: true
         type: string
@@ -74,6 +78,7 @@ on:
 jobs:
   terraform:
     runs-on: ubuntu-latest
+    timeout-minutes: ${{ inputs.job_timeout_minutes }}
     env:
       AWS_DEFAULT_REGION: ${{ inputs.aws_region }}
       environment: ${{ inputs.environment }}
@@ -121,12 +126,15 @@ jobs:
         run: terraform fmt --recursive -check
 
       - name: Configure AWS credentials
+        timeout-minutes: 2
         uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
         with:
           role-to-assume: ${{ inputs.aws_role_arn }}
           aws-region: ${{ inputs.aws_region }}
           role-duration-seconds: ${{ inputs.aws_role_duration }}
           role-session-name: ${{ github.event.repository.name }}
+          action-timeout-s: 45
+          retry-max-attempts: 3
 
       - name: Setup Kosli CLI
         if: inputs.kosli_template_file != ''

.github/workflows/detect-drift.yml

@@ -25,6 +25,10 @@ on:
         description: "Extra environment variables, one KEY=VALUE per line, exported before the drift plan. To set Terraform variable `foo`, use `TF_VAR_foo=...`. Single-line values only. Note: for per-build values (e.g. an image tag) give the variable a default in variables.tf instead, otherwise drift detection reports false drift."
         default: ""
         type: string
+      job_timeout_minutes:
+        description: "Minutes before GitHub cancels the drift plan job. Raise it together with aws_role_duration when a legitimate plan needs longer; see the README on timeouts."
+        default: 30
+        type: number
 
 concurrency:
   group: detect-drift-${{ github.repository }}-${{ inputs.environment }}
@@ -33,6 +37,7 @@ concurrency:
 jobs:
   fetch-baseline:
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     permissions:
       id-token: write
       contents: read
@@ -41,12 +46,15 @@ jobs:
       drift: ${{ steps.read.outputs.drift }}
     steps:
       - name: Configure AWS credentials
+        timeout-minutes: 2
         uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
         with:
           role-to-assume: ${{ inputs.aws_role_arn }}
           aws-region: ${{ inputs.aws_region }}
           role-duration-seconds: ${{ inputs.aws_role_duration }}
           role-session-name: ${{ github.event.repository.name }}
+          action-timeout-s: 45
+          retry-max-attempts: 3
 
       - name: Compute state bucket name
         run: |
@@ -98,22 +106,27 @@ jobs:
       ref: ${{ needs.fetch-baseline.outputs.sha }}
       tf_apply: false
       tf_vars: ${{ inputs.tf_vars }}
+      job_timeout_minutes: ${{ inputs.job_timeout_minutes }}
 
   flag-drift:
     needs: [fetch-baseline, plan]
     if: needs.plan.outputs.has_changes == 'true'
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     permissions:
       id-token: write
       contents: read
     steps:
       - name: Configure AWS credentials
+        timeout-minutes: 2
         uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
         with:
           role-to-assume: ${{ inputs.aws_role_arn }}
           aws-region: ${{ inputs.aws_region }}
           role-duration-seconds: ${{ inputs.aws_role_duration }}
           role-session-name: ${{ github.event.repository.name }}
+          action-timeout-s: 45
+          retry-max-attempts: 3
 
       - name: Compute state bucket name
         run: |
@@ -142,6 +155,7 @@ jobs:
     needs: [fetch-baseline, plan]
     if: needs.plan.outputs.has_changes == 'false'
     runs-on: ubuntu-latest
+    timeout-minutes: 5
     steps:
       - name: No-drift summary
         run: |

.github/workflows/plan.yml

@@ -46,6 +46,10 @@ on:
       kosli_cli_version:
         default: "latest"
         type: string
+      job_timeout_minutes:
+        description: "Minutes before GitHub cancels the plan job. Raise it together with aws_role_duration when a legitimate plan needs longer; see the README on timeouts."
+        default: 30
+        type: number
     secrets:
       kosli_api_token:
         description: "Kosli API token. Required when kosli_template_file is set."
@@ -80,5 +84,6 @@ jobs:
       kosli_org: ${{ inputs.kosli_org }}
       kosli_flow: terraform-plan-${{ inputs.environment }}-${{ github.event.repository.name }}
       kosli_cli_version: ${{ inputs.kosli_cli_version }}
+      job_timeout_minutes: ${{ inputs.job_timeout_minutes }}
     secrets:
       kosli_api_token: ${{ secrets.kosli_api_token }}

.github/workflows/test.yml

@@ -9,6 +9,7 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
       - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0

README.md

@@ -100,6 +100,7 @@ Both `plan.yml` and `apply.yml` accept the same core inputs:
 | `working_directory` | no | `./` | Directory containing Terraform config |
 | `tf_version` | no | `1.14.6` | Terraform version to install |
 | `tf_vars` | no | `""` | Extra env vars (one `KEY=VALUE` per line) exported before plan/apply; see [Supplying Terraform variables](#supplying-terraform-variables) |
+| `job_timeout_minutes` | no | `30` | Minutes before GitHub cancels the plan/apply job; see [Timeouts](#timeouts) |
 
 Plus, for opting into Kosli attestation (see [Kosli attestation](#kosli-attestation) below):
 
@@ -169,6 +170,60 @@ image tag, which changes every run.
 | `kosli_api_token` | if `kosli_template_file` is set | Kosli API token for the attest steps. |
 | `kosli_github_token` | no (only `apply.yml`) | GitHub token used by `kosli attest pr github` to look up pull requests. When omitted, the pull-request attestation step is skipped. Typically passed as `${{ secrets.GITHUB_TOKEN }}` — in which case the **calling job must also declare `pull-requests: read`** in its `permissions:` block (see example below), otherwise the attestation step will fail with `Resource not accessible by integration`. |
 
+### Timeouts
+
+Every job carries a `timeout-minutes` rather than inheriting GitHub's 360-minute default, and the
+OIDC credential step is bounded so that an unreachable STS endpoint fails in under a minute instead
+of holding a runner — and the environment's `concurrency` group — for over an hour:
+
+```yaml
+      - name: Configure AWS credentials
+        timeout-minutes: 2
+        uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6.1.0
+        with:
+          # ...
+          action-timeout-s: 45
+          retry-max-attempts: 3
+```
+
+These are the standard values for **any** Kosli workflow using
+`aws-actions/configure-aws-credentials`, not just the ones here. The reasoning:
+
+| Setting | Value | Why |
+|---|---|---|
+| `action-timeout-s` | `45` | Bounds the action as a whole, retries included. Across recent successful runs in `kosli-dev` and `cyber-dojo` the step took 0–2s, worst case 6s, so 45s is far above the real p99 and will not fail a slow-but-healthy authentication. |
+| `retry-max-attempts` | `3` | The default is 12. STS throttling is genuinely transient and worth retrying, but 12 attempts is only useful if each attempt is fast — which is exactly what fails to hold when the endpoint is unreachable. |
+| `timeout-minutes` (step) | `2` | A backstop that holds regardless of how the action behaves or what a future version changes. |
+| `timeout-minutes` (job) | `30` plan/apply, `5`–`10` housekeeping | Bounded by `aws_role_duration`, not by how long Terraform might take — see below. |
+
+`disable-retry` is deliberately **not** used: a couple of quick retries is worth having, and
+`action-timeout-s` already bounds the total cost of them.
+
+The plan/apply job's default of 30 minutes is **derived from `aws_role_duration`**, which defaults
+to `1200` (20 minutes). The credentials the OIDC step exports are static environment variables and
+are never refreshed, so 20 minutes after that step every AWS call starts failing with
+`ExpiredToken` — a longer job ceiling would buy nothing, because the work cannot usefully continue.
+Measured against real runs, setup before the credentials step takes 1–35s, so a job that consumes
+its entire credential lifetime lands around 22–23 minutes; 30 leaves headroom without being
+arbitrary.
+
+The two values are coupled, so **raise `job_timeout_minutes` and `aws_role_duration` together** when
+a repository has a legitimately long apply. Raising only the session duration lets the job be
+cancelled part-way through an `apply`, which can leave the state lock held — a worse outcome than a
+slow run. Raising only the job ceiling buys time in which every AWS call fails:
+
+```yaml
+    with:
+      aws_role_duration: "3600"   # 60 min session
+      job_timeout_minutes: 70     # 60 + setup + headroom
+```
+
+The role's own maximum session duration is the hard limit on `aws_role_duration`; if a longer
+session is refused, that maximum needs raising on the IAM role first.
+
+A job that fails in 45 seconds can be re-run for nothing. A job that hangs for 82 minutes blocks
+every apply queued behind it.
+
 ### What it does
 
 **Plan** (`plan.yml`):