Plan Once, Apply the Same Plan: A Terraform Pipeline with OIDC and an Approval Gate
There is a quiet bug in many Terraform pipelines. A pull request runs terraform plan, a human reviews it, and later a different job runs terraform apply, which plans again. Between the first plan and the second one, anything can change: another merge, a manual console tweak, a data source returning something new. The human approved plan A, and the pipeline executed plan B.
The infrastructure repository I worked on closed that gap, and combined it with credentials that don't exist until the moment they're needed. This is how the pipeline was put together, and what I think is still open.
Credentials that expire on their own
Pipelines that talk to a cloud provider need credentials. The old way is an access key stored as a CI secret: long-lived, shared, and valuable to anyone who can read your repo settings.
The Terraform pipeline instead used GitHub's OIDC integration with AWS. The workflow asks for a short-lived token by proving which repository and workflow it is, and AWS hands back temporary credentials for a specific role. The workflow declares what it needs:
permissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v5
with:
role-to-assume: arn:aws:iam::${{ matrix.aws_account_id }}:role/${{ matrix.aws_role }}
aws-region: eu-west-1
On the AWS side, the role and the OIDC provider are themselves Terraform, using a community module, and the trust is scoped to a single repository:
module "github-oidc" {
source = "terraform-module/github-oidc-provider/aws"
create_oidc_provider = true
create_oidc_role = true
role_name = "${module.vars.tenant.name}-TerraformFromGithub"
repositories = ["org/terraform-aws-inf"]
oidc_role_attach_policies = [aws_iam_policy.terraform_policy.arn]
}
No secret stored anywhere, no key to rotate, and anything that leaks from a run stops working once the short session expires.
No stored keys, only a token exchange per run
Now the honest part. The policy attached to that role lists whole services (s3:*, dynamodb:*, eks:*, iam:*, rds:*, and so on), with resource ARNs narrowed by account, region and resource type. That's a reasonable compromise for a role that must be able to create anything in the stack, but iam:* on a CI role is effectively the keys to the account. If I were tightening it, the next steps would be a permissions boundary on roles the pipeline creates, a separate (smaller) role for plan-only runs, and branch or environment conditions in the trust policy so only the protected branch can assume the apply role.
Runners that live where the infrastructure lives
Plans for private environments often need to reach private endpoints (restricted state buckets, clusters whose API isn't public). The repository doesn't say why it chose this, but the plan and apply jobs run on self-hosted runners labelled per tenant and environment, which is the usual way to avoid punching holes for a hosted runner:
runs-on: org-${{ matrix.tenant }}-${{ matrix.environment }}-platform
A matrix of tenant and environment fans the same job out across every environment affected by a change. The repository's secrets are encrypted in git with git-crypt, so the job unlocks them first:
- name: Unlock git-crypt
run: git-crypt unlock
(The key is supplied from a CI secret, which is a secret-to-unlock-secrets arrangement worth knowing about: protect that one key harder than anything else.)
Detect what changed, run only that
A full plan across every stack in every environment is slow. The workflow starts with a job that diffs the pull request against its base and keeps only the folders under infrastructure/ that contain a Makefile, i.e. the stacks that actually changed. That list feeds a matrix, so a one-line change in a single module produces one plan, not forty.
The core idea: the plan is an artifact
Each plan run writes a plan file with a unique name that includes tenant, environment, module and commit SHA, and uploads it as a build artifact:
UNIQUE_PLAN_FILE="${tenant}_${environment}_${module//\//_}_${GITHUB_SHA}.tfplan"
# make tf/plan produces ${TF_WORKSPACE}.tfplan
mv "${TF_WORKSPACE}.tfplan" "$UNIQUE_PLAN_FILE"
Then the pipeline stops and asks a human. The job that waits is bound to a GitHub environment with protection rules:
approval:
needs: [detect-changes, prepare-matrix, plan]
environment:
name: terraform-approval
An environment's required reviewers are enforced by GitHub, not by convention. The run pauses until an authorised person approves in the UI, and that approval is recorded against the run.
Only after approval does the apply job run. It downloads the artifact from the plan job and restores the exact file before applying:
- uses: actions/download-artifact@v4
# ...
- name: Apply the saved plan
run: |
cp "$PLAN_FILE_PATH" "${TF_WORKSPACE}.tfplan"
make tf/apply # applies the saved plan, does not re-plan
Terraform refuses to apply a saved plan if the state has changed since it was created, so a stale plan fails closed instead of doing something nobody reviewed.
The reviewed file and the applied file are the same artifact
One detail deserves a test rather than trust. The Makefile's apply target has a conditional around the plan file, and as written it reads backwards: ifeq (,$(wildcard "${TF_WORKSPACE}.tfplan")) runs terraform apply <workspace>.tfplan when the wildcard finds nothing, and falls back to a plain terraform apply (which re-plans) when it finds the file. Because the pattern is quoted, the wildcard most likely never matches, so in practice the first branch always wins and the saved plan is what gets applied. That is the behaviour you want, but it works by accident, and the day someone "fixes" the quotes the pipeline would silently start re-planning at apply time. A small CI check that fails if the apply step ever runs without a plan file would pin the property down.
There's also an escape hatch, a boolean auto_apply input that skips the approval job. It's useful for lower environments and is a conscious choice at dispatch time rather than a default, but it means the safety property depends on who's allowed to trigger the workflow with that input.
Why this design works
The main property is simple: what was approved is what runs. The reviewer reads a plan, and the pipeline applies that same plan. The gate also doesn't rely on memory, because the approval is a platform feature rather than a team habit. Operators get least surprise, since plans are named by what they are, retained as artifacts and summarised in the run. And there are no standing credentials, because everything that touches the cloud is short-lived.
What I'd still change
- Scope the role per stage. Plan needs read access plus state locking, while apply needs write. Two roles beat one.
- Gate by branch. The apply role should be assumable only from the protected branch or the approval environment.
- Expire artifacts quickly. A plan file can contain sensitive values, so retention should be days, not months.
- Report drift on a schedule. A nightly plan with no apply would flag out-of-band changes before the next real change trips on them.
Wrapping up
Make the plan a first-class artifact, put the human between plan and apply with a mechanism the platform enforces, and make the credentials expire on their own. You can then be strict about the process without making it slow. The remaining work is mostly about narrowing what the role is allowed to do and who is allowed to skip the gate.