gao.ninja logogao.ninja

Bootstrapping Terraform State Without the Chicken-and-Egg Problem

Every Terraform setup eventually hits the same awkward question: where does the state bucket come from? You want remote state in S3 with locking and encryption, but the bucket has to be created by something, and that something can't store its state in a bucket that doesn't exist yet.

On a multi-tenant lending platform I worked on, the infrastructure repository answered this with a very small, very deliberate design. It's worth walking through, because the interesting part isn't the bucket. It's how every other stack finds the bucket without anyone typing its name.

One account, one backend, many workspaces

The platform served several customers (tenants), each with several environments: test, staging, UAT, sandbox, production, plus disaster-recovery variants. Every combination got its own Terraform workspace, named with a simple convention:

<tenant>-<environment>[_<label>]

So acme-prod is the production environment of a tenant, and acme-sbx_dr is the disaster-recovery copy of its sandbox. The convention isn't just cosmetic. The bootstrap module parses the workspace name to derive everything else:

locals {
  tenant      = split("-", terraform.workspace)[0]
  environment = split("_", split("-", terraform.workspace)[1])[0]
  label       = length(regexall("_", terraform.workspace)) > 0 ? split("_", terraform.workspace)[1] : ""

  prefix              = "org-${local.tenant}-${local.environment}"
  s3_bucket_name      = "${local.prefix}-terraform-aws"
  dynamodb_table_name = "${local.prefix}-terraform-aws-state-lock"
}

(I've renamed the prefix here, but the shape is real.) The consequence is that there is no per-environment configuration for the backend at all. Pick a workspace and the bucket, lock table and KMS key names fall out of it.

The bootstrap stack uses a local backend, on purpose

The bootstrap lives in its own init/ folder and its backend is exactly this:

terraform {
  backend "local" {}
}

That single stack creates what every other stack will use: a versioned S3 bucket encrypted with a dedicated KMS key, a DynamoDB table (pay-per-request, LockID as the hash key) for state locking, and, when the environment is flagged for it, a replica of all three in a secondary region with bidirectional replication so the state itself survives a regional failure.

Because init/ uses a local backend, its own state is a file on disk. The repository commits that file, under init/terraform.tfstate.d/<workspace>/terraform.tfstate, and encrypts it transparently with git-crypt. A .gitattributes rule is all it takes:

init/terraform.tfstate.d/** filter=git-crypt diff=git-crypt

State in git is not something I'd reach for by default, and I wouldn't do it for a stack that changes often. But the bootstrap stack is tiny, should rarely change, and exists precisely so that nothing else needs this trick. Committing it, encrypted, makes the whole repository self-describing: clone, unlock, and you can find every backend.

The init stack, with a local backend and state committed encrypted in git, creates the S3 bucket, DynamoDB lock table and KMS key, plus optional replicas. Every other stack then initializes an S3 backend from those outputsThe bootstrap stack runs first and is the only one with local state

How every other stack finds the backend

Here's the part I like most. The other stacks declare an S3 backend, but only partially:

{
  "terraform": {
    "backend": {
      "s3": {
        "encrypt": true,
        "key": "kubernetes/eks/terraform.tfstate",
        "workspace_key_prefix": "tf-states"
      }
    }
  }
}

No bucket, no region, no KMS key, no lock table. Those arrive at terraform init time, from a wrapper script that reads the outputs of the bootstrap stack's state file with jq:

function __get_tf_init_backend_output_from_state() {
   init_tf_state_file="$1/init/terraform.tfstate.d/$init_workspace/terraform.tfstate"

   backend_output_name="backend"
   if [[ "$is_dr" == "true" ]]; then
      backend_output_name="backend_replica"
   fi

   jq empty "$init_tf_state_file" >/dev/null 2>&1 || {
     echo "Invalid state file, did you 'git-crypt unlock'?"; exit 1;
   }

   backend_s3_bucket_id=$(jq -r ".outputs.${backend_output_name}.value.bucket_id" "$init_tf_state_file")
   backend_dynamodb_table=$(jq -r ".outputs.${backend_output_name}.value.dynamodb_table" "$init_tf_state_file")
   backend_kms_key_id=$(jq -r ".outputs.${backend_output_name}.value.kms_key_id" "$init_tf_state_file")
}

Two details make this more than a convenience.

First, the is_dr switch. A disaster-recovery workspace automatically reads the replica backend's outputs, so DR stacks keep their state in the secondary region without a single module being aware of it.

Second, the guard. If the file isn't valid JSON, the most likely reason is that git-crypt hasn't been unlocked and the "state" is ciphertext. Instead of letting Terraform fail with something cryptic, the script says exactly what to do. Small error messages like this one are what separate a repo people can operate at 2 a.m. from one only its author understands.

The final state layout is predictable: tf-states/<workspace>/<module path>/terraform.tfstate. One bucket per account and environment, one object per stack per workspace. That predictability matters for the next problem, which is how stacks read each other's outputs (a topic for another post).

What I'd keep and what I'd change

The part I'd keep without hesitation is derive, don't configure. Everything about the backend comes from the workspace name, so there is nothing to forget to set. Close behind it is a boring bootstrap: one small stack with a local backend that almost nobody ever touches. And I like replication as a flag, not a fork. DR is a property of the environment, and the tooling simply follows it.

If I were starting today, I'd change two things. I'd look hard at whether committing even an encrypted state file is worth it, compared with a documented, one-time manual bootstrap per account. And I'd move the shell-based discovery into something with tests. It works, but it's the one piece of the chain you can break with a typo and only notice at init time.

Wrapping up

The chicken-and-egg problem disappears when the bootstrap is small enough to be boring and the discovery is automatic enough that nobody ever copies a bucket name into a file. A workspace naming convention gives every stack its backend for free, a local-state init/ stack creates that backend once, and a short script hands its outputs to terraform init.