gao.ninja logogao.ninja

Rendering Config at Container Start Instead of Baking It In

Configuration is where twelve-factor ideals meet real systems. The textbook says: put config in the environment. Then you count the settings a lending platform needs per service (database hosts and users, third-party API credentials, queue names, feature toggles per tenant) multiplied by environments multiplied by tenants, and "just use environment variables" turns into a very long Helm values file nobody wants to review.

The platform I worked on took a different route. It kept configuration templates in one repository, rendered them through a small config service, and let every container fetch its own rendered file at start-up. This post is about why that design is attractive, and where it bites.

The moving parts

There are four pieces, and the interesting thing is how little each one knows.

1. A templates repository. For every service there is a folder with a config template in whatever format the service already speaks: JSON, .env, INI or TOML. A template uses a small function syntax to pull values:

"database": {
  "mysql": {
    "platform": {
      "master": {
        "host": "{{ .GetValue "rds.platform.writer.host" }}",
        "password": "{{ .GetValue "rds.platform.users.platform.password" }}"
      }
    }
  }
},
"thirdParty": { "apiKey": "{{ .GetValue "third-party-service.example.api-key" }}" }

Nothing sensitive is in the template. It only contains keys.

2. A config service. A dedicated HTTP service ships with those templates inside its image. When a container asks for its config, the service renders the template by resolving each key from layered sources: Kubernetes ConfigMaps, per-environment override files mounted from the environment's repository, and the cloud parameter store, where secrets live. Override files can point at the store using a simple prefix (ssm:/org/<tenant>/<env>/...), so even overrides never contain the secret value.

3. A rebuild trigger. The templates repository and the config service live in separate repositories, so a template change has to cause a new config-service build. A small workflow on the templates repository does exactly that: on every push to main it writes the commit hash into a file in the config service's repository and commits it.

- name: Add commit hash
  run: echo "$GITHUB_SHA" > config-service/templates.sha
- name: Add and Commit
  uses: EndBug/add-and-commit@v9
  with:
    cwd: config-service
    message: "fix(autocommit): templates changed"
    add: "templates.sha"

The commit produces a new release of the config service, which re-clones the templates. The .sha file also doubles as an audit trail of which template revision each release embeds. (It doesn't pin the clone, though. The build clones the default branch. Worth knowing before you rely on it for reproducibility.)

A push to the templates repository commits a hash file to the config service repository, which triggers a new config service release that clones the templatesA template change takes effect only through a new config service release

4. The container entrypoint. Most services start through the same shell entrypoint, which accepts flags describing the config it needs:

--config=<remote-name>:<local-path>

For each one, before launching the real process, the entrypoint downloads the rendered file and refuses to continue if that fails:

curlstatus=$(curl -s -o "${v}" -w "%{http_code}" -X GET "http://config-service/config/${k}")

if [ "${curlstatus}" != "200" ]; then
    echo "failed downloading configuration: ${k} (http status: ${curlstatus})"
    exit 1
fi

At container start the entrypoint fetches each rendered file from the config service, which resolves keys from ConfigMaps, overrides and the parameter storeConfig is rendered on request at container start, and a failed download stops the container

Why this works well

Config follows the application's format, not Kubernetes'. A PHP service gets a real .ini, a Go service gets JSON, and a Node service gets a .env. No translation layer from YAML.

Secrets never touch the manifests. Helm values and templates carry references, never values. Rotating a secret means changing the parameter store and restarting the pods that read it, not re-releasing (and since config is fetched at start-up, running pods keep the old value until they restart).

One place to answer "what does this service read?" Every key a service depends on is visible in a single template.

Fail fast. Exiting non-zero when the download fails means a missing or broken config shows up as a pod that won't start, which is the loudest and most diagnosable failure Kubernetes has. The alternative, starting with empty or stale config, is how bad data gets written at 3 a.m.

What you're signing up for

A new critical dependency. Every start-up (including every autoscaled pod, every node replacement, every rollout) depends on the config service being reachable. If it's down, nothing new can boot. You need it highly available, and you need to be deliberate about start-up ordering when you rebuild a cluster from scratch.

Failures move to runtime. A typo in a key won't fail in CI. It fails when a pod starts. The mitigation is rendering every template against every environment's values in a pipeline, so mistakes die before a deploy.

Debugging has an extra hop. "Why does this service have the wrong host?" now means checking template, overrides, ConfigMaps and the parameter store, in layered order. A "render this service's config for environment X" endpoint, available to engineers, pays for itself immediately.

Config and code can drift. A service release and a template change are separate events. Without a compatibility story (templates tolerate the old and new keys during a transition), you can ship a service that expects a key the running config service can't yet render.

Would I do it again?

For a platform with many services, several languages and strict secret handling, yes, with two additions: render-all-templates validation in CI, and a clear degradation story for the config service (replicas, a PodDisruptionBudget, and a documented bootstrap order for disaster recovery).

For a handful of services, I'd start with plain environment variables plus a secrets operator and revisit when the pain shows up.

Takeaway: Moving config out of manifests only helps if you also move the validation out of production. Make the template renderer something you can run in a pull request, and let the entrypoint be strict at runtime.