Configuration and secrets in cloud-native applications
Most teams believe they have solved this because nothing is hardcoded and everything comes from an environment variable. That satisfies the letter of the rule and misses two of the three things that matter: whether a secret can be rotated without a redeploy, and whether it leaks into the places nobody thinks to check. Both problems surface at the worst possible time, which is during an incident involving a credential.
What belongs in configuration and what does not?
Anything that legitimately differs between environments, and nothing else. Database hostnames, API endpoints, bucket names, log levels, credentials. If a value is the same in staging and production, it is not configuration, it is a constant, and moving it out of the code buys nothing while making the behaviour harder to find.
Two categories get miscategorised routinely. Feature flags are not configuration: they change during the life of a running process, often several times a day, and they belong in something built for runtime toggling rather than in a variable that requires a restart to change. Business rules such as tax rates and thresholds are not configuration either, because they need history, an audit trail and usually a user interface, which makes them data.
The test that resolves most arguments is to ask who changes the value and how often. Set once per environment by an engineer means configuration. Changed during business hours by someone who is not an engineer means data with a screen attached, and putting it in an environment variable guarantees a deploy every time the business wants a different number.
Environment variables or mounted files?
Environment variables for non-sensitive configuration, mounted files for secrets. The distinction is not stylistic: environment variables are inherited by every child process, visible to anything that can read the process table, and captured by a surprising number of crash reporters and error trackers by default. A mounted file is readable only by processes that open that path.
The second difference is rotation. An environment variable is fixed for the life of the process, so changing it requires a restart of every instance. A mounted secret is updated in place by the platform, which means an application that re-reads the file can pick up a new credential without a deploy.
| Mechanism | Suited to | Leaks through | Rotation |
|---|---|---|---|
| Environment variables | Hostnames, feature toggles, log levels, non-sensitive settings | Child processes, crash dumps, error trackers, debug endpoints | Requires a restart of every instance |
| Mounted secret files | Database passwords, API keys, certificates | Anything that can read the container filesystem | Updated in place; the app must re-read |
| Secret manager, read at runtime | Short-lived credentials and dynamic leases | Application logs, if the response is logged | Native, including automatic expiry |
| Config maps or plain config objects | Non-sensitive structured settings | Anyone with read access to the namespace | Updated in place, same re-read requirement |
| Baked into the image | Nothing | Every image layer, forever, including the registry | Rebuild and redeploy |
| CI/CD pipeline variables | Deploy-time credentials only | Build logs, and any fork or pull request build | Manual, and easily forgotten |
Why do secrets leak from environment variables?
Through five specific routes, none of which involve anybody doing anything obviously wrong. On Linux, the environment of a process is readable from the proc filesystem by anything running with sufficient privilege in the same container. Child processes inherit the whole environment, so a shell-out to a command line tool passes every credential along with it. Crash handlers and error tracking libraries commonly attach environment state to reports, sending credentials to a third party. Many frameworks expose a debug or diagnostics page listing the environment. And startup logging that prints configuration for troubleshooting is the direct route into your log aggregator, where it is now retained and searchable.
The consequence of each is the same and worth naming: the credential is now in a system with different access controls and a different retention period from the secret store it came from. Revoking it later does not remove it from six months of log archives.
Mitigations are straightforward once you know the routes. Scrub known secret keys in the error tracker configuration rather than trusting the defaults, never log the whole configuration object, disable framework diagnostics pages in production, and prefer file-based delivery for anything that would matter if it appeared in a log line.
What does rotation actually require of the application?
That it can acquire a new credential without being restarted, and that its existing connections do something sensible when the old one stops working. The second part is what gets missed. A connection pool authenticates when a connection opens, so pooled connections keep working with a revoked password until they are recycled, which makes the rotation appear successful and produces failures hours later when the pool refreshes.
The workable pattern is to read credentials through an accessor rather than capturing them at startup, so that every new connection uses the current value. Then set a maximum connection lifetime in the pool so old connections retire on a known schedule rather than living indefinitely. Rotation then completes within that lifetime without a deploy and without an outage.
The version to avoid is rotation by rolling restart. It works, it is what most teams do, and it means every credential change is a deployment event that has to be scheduled, which is precisely why credentials end up not being rotated. If the only way to change a password is to redeploy production, the password will not be changed.
How should local development get its configuration?
From a checked-in example file listing every required variable with placeholder values, plus a local file that is ignored by version control. The example file is the part that gets skipped and the part that matters, because it is the only machine-readable record of what the application actually needs, and it turns onboarding from a conversation into a command.
Give developers credentials that only work locally. The pattern where a new engineer is handed production read credentials for convenience is how those credentials end up on unencrypted laptops and in support chats, and it is entirely avoidable when the local environment starts real backing services in containers with throwaway passwords.
Make the application fail loudly when a required value is absent. Configuration that silently falls back to a default is the mechanism behind a specific class of incident: a variable is renamed, the deploy succeeds, the application starts, and it quietly connects to the wrong place or runs with authentication disabled. Validate the full set at startup and refuse to start if anything required is missing.
What can you check this afternoon?
Four things. Run the image history for your production container and look at the build layers for anything resembling a credential, since a secret passed as a build argument or copied and later deleted still exists in an earlier layer. Grep your log aggregator for the name of a known configuration key and see what comes back. Check your error tracker for a captured environment on any recent exception. And take a non-production database password, rotate it in the secret store, and time how long until the application uses the new one without help.
The fourth is the informative one. If the answer is that it never does until someone redeploys, you have discovered that your rotation procedure is a deployment procedure, which tells you what will happen the day a credential has to be replaced urgently.
None of these need a project to run. They need an afternoon and someone with access to the pipeline, and each has a binary answer that either closes the question or defines the next piece of work.
Common questions
- Should secrets be stored in environment variables?
- Preferably not. Environment variables are inherited by child processes, readable from the process table, commonly captured by error trackers and crash handlers, and often exposed on framework diagnostics pages. They also cannot be changed without restarting the process. Mounted secret files or a secret manager read at runtime avoid both problems, and allow rotation without a deploy.
- What counts as configuration and what does not?
- Configuration is anything that legitimately differs between environments: hostnames, endpoints, bucket names, log levels, credentials. A value identical in staging and production is a constant, not configuration. Feature flags belong in a runtime toggling system because they change during the life of a process, and business rules such as rates and thresholds are data, because they need history and an audit trail.
- How do you rotate a database password without downtime?
- Read the credential through an accessor rather than capturing it at startup, so every new connection uses the current value, and set a maximum connection lifetime in the pool so existing connections retire on a known schedule. Rotation then completes within that lifetime. Without this, pooled connections keep working with a revoked password and fail hours later when the pool refreshes.
- Why should an application fail to start when configuration is missing?
- Because a silent fallback to a default causes a specific class of incident: a variable is renamed, the deploy succeeds, the process starts, and it quietly connects to the wrong system or runs with authentication disabled. Validating the complete required set at startup and refusing to start turns a subtle production problem into an obvious deployment failure.
- Can secrets hide inside a container image?
- Yes. A secret passed as a build argument, or copied in and deleted in a later step, still exists in the earlier layer and travels with the image into the registry. Inspecting the build history of a production image will show it. This is why secrets should be delivered at runtime rather than at build time, and why registry access should be treated as sensitive.
