gao.ninja logogao.ninja

Plan Once, Apply the Same Plan: A Terraform Pipeline with OIDC and an Approval Gate

There is a quiet bug in many Terraform pipelines. A pull request runs terraform plan, a human reviews it, and later a different job runs terraform apply, which plans again. Between the first plan and the second one, anything can change: another merge, a manual console tweak, a data source returning something new. The human approved plan A, and the pipeline executed plan B.

The infrastructure repository I worked on closed that gap, and combined it with credentials that don't exist until the moment they're needed. This is how the pipeline was put together, and what I think is still open.

Credentials that expire on their own

Pipelines that talk to a cloud provider need credentials. The old way is an access key stored as a CI secret: long-lived, shared, and valuable to anyone who can read your repo settings.

The Terraform pipeline instead used GitHub's OIDC integration with AWS. The workflow asks for a short-lived token by proving which repository and workflow it is, and AWS hands back temporary credentials for a specific role. The workflow declares what it needs:

permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v5
    with:
      role-to-assume: arn:aws:iam::${{ matrix.aws_account_id }}:role/${{ matrix.aws_role }}
      aws-region: eu-west-1

On the AWS side, the role and the OIDC provider are themselves Terraform, using a community module, and the trust is scoped to a single repository:

module "github-oidc" {
  source = "terraform-module/github-oidc-provider/aws"

  create_oidc_provider = true
  create_oidc_role     = true

  role_name                 = "${module.vars.tenant.name}-TerraformFromGithub"
  repositories              = ["org/terraform-aws-inf"]
  oidc_role_attach_policies = [aws_iam_policy.terraform_policy.arn]
}

No secret stored anywhere, no key to rotate, and anything that leaks from a run stops working once the short session expires.

The workflow requests an OIDC token from GitHub, presents it to AWS STS, and AWS checks the trust policy for the repository before returning temporary credentials for the Terraform roleNo stored keys, only a token exchange per run

Now the honest part. The policy attached to that role lists whole services (s3:*, dynamodb:*, eks:*, iam:*, rds:*, and so on), with resource ARNs narrowed by account, region and resource type. That's a reasonable compromise for a role that must be able to create anything in the stack, but iam:* on a CI role is effectively the keys to the account. If I were tightening it, the next steps would be a permissions boundary on roles the pipeline creates, a separate (smaller) role for plan-only runs, and branch or environment conditions in the trust policy so only the protected branch can assume the apply role.

Runners that live where the infrastructure lives

Plans for private environments often need to reach private endpoints (restricted state buckets, clusters whose API isn't public). The repository doesn't say why it chose this, but the plan and apply jobs run on self-hosted runners labelled per tenant and environment, which is the usual way to avoid punching holes for a hosted runner:

runs-on: org-${{ matrix.tenant }}-${{ matrix.environment }}-platform

A matrix of tenant and environment fans the same job out across every environment affected by a change. The repository's secrets are encrypted in git with git-crypt, so the job unlocks them first:

- name: Unlock git-crypt
  run: git-crypt unlock

(The key is supplied from a CI secret, which is a secret-to-unlock-secrets arrangement worth knowing about: protect that one key harder than anything else.)

Detect what changed, run only that

A full plan across every stack in every environment is slow. The workflow starts with a job that diffs the pull request against its base and keeps only the folders under infrastructure/ that contain a Makefile, i.e. the stacks that actually changed. That list feeds a matrix, so a one-line change in a single module produces one plan, not forty.

The core idea: the plan is an artifact

Each plan run writes a plan file with a unique name that includes tenant, environment, module and commit SHA, and uploads it as a build artifact:

UNIQUE_PLAN_FILE="${tenant}_${environment}_${module//\//_}_${GITHUB_SHA}.tfplan"
# make tf/plan produces ${TF_WORKSPACE}.tfplan
mv "${TF_WORKSPACE}.tfplan" "$UNIQUE_PLAN_FILE"

Then the pipeline stops and asks a human. The job that waits is bound to a GitHub environment with protection rules:

approval:
  needs: [detect-changes, prepare-matrix, plan]
  environment:
    name: terraform-approval

An environment's required reviewers are enforced by GitHub, not by convention. The run pauses until an authorised person approves in the UI, and that approval is recorded against the run.

Only after approval does the apply job run. It downloads the artifact from the plan job and restores the exact file before applying:

- uses: actions/download-artifact@v4
  # ...
- name: Apply the saved plan
  run: |
    cp "$PLAN_FILE_PATH" "${TF_WORKSPACE}.tfplan"
    make tf/apply        # applies the saved plan, does not re-plan

Terraform refuses to apply a saved plan if the state has changed since it was created, so a stale plan fails closed instead of doing something nobody reviewed.

A pull request triggers change detection, then one plan per changed stack uploaded as an artifact, then a paused approval job bound to a protected environment, then an apply job that downloads and applies the same plan fileThe reviewed file and the applied file are the same artifact

One detail deserves a test rather than trust. The Makefile's apply target has a conditional around the plan file, and as written it reads backwards: ifeq (,$(wildcard "${TF_WORKSPACE}.tfplan")) runs terraform apply <workspace>.tfplan when the wildcard finds nothing, and falls back to a plain terraform apply (which re-plans) when it finds the file. Because the pattern is quoted, the wildcard most likely never matches, so in practice the first branch always wins and the saved plan is what gets applied. That is the behaviour you want, but it works by accident, and the day someone "fixes" the quotes the pipeline would silently start re-planning at apply time. A small CI check that fails if the apply step ever runs without a plan file would pin the property down.

There's also an escape hatch, a boolean auto_apply input that skips the approval job. It's useful for lower environments and is a conscious choice at dispatch time rather than a default, but it means the safety property depends on who's allowed to trigger the workflow with that input.

Why this design works

The main property is simple: what was approved is what runs. The reviewer reads a plan, and the pipeline applies that same plan. The gate also doesn't rely on memory, because the approval is a platform feature rather than a team habit. Operators get least surprise, since plans are named by what they are, retained as artifacts and summarised in the run. And there are no standing credentials, because everything that touches the cloud is short-lived.

What I'd still change

  • Scope the role per stage. Plan needs read access plus state locking, while apply needs write. Two roles beat one.
  • Gate by branch. The apply role should be assumable only from the protected branch or the approval environment.
  • Expire artifacts quickly. A plan file can contain sensitive values, so retention should be days, not months.
  • Report drift on a schedule. A nightly plan with no apply would flag out-of-band changes before the next real change trips on them.

Wrapping up

Make the plan a first-class artifact, put the human between plan and apply with a mechanism the platform enforces, and make the credentials expire on their own. You can then be strict about the process without making it slow. The remaining work is mostly about narrowing what the role is allowed to do and who is allowed to skip the gate.