Cloud secrets and rotation that people actually do
A ninety-day rotation policy that nobody follows is worse than no policy, because it produces an audit answer that is false and a team that has learned to ignore the control. Most secrets in a modern cloud estate should not be rotated at all; they should stop existing, replaced by identities the platform issues and expires on its own. What is left after that elimination is a short list, and a short list is the only thing anybody rotates on schedule.
Which secrets should you not be rotating at all?
The ones that can be removed. A static cloud access key in a pipeline can be replaced by workload identity federation, where the pipeline presents a signed token proving which repository and which workflow it is and receives a credential valid for that job. A database password held by an application can often be replaced by the provider's own database authentication, where the connection is authorised by the workload's cloud identity and the token lives minutes. In both cases the rotation question disappears with the secret.
The same applies to service account key files, which are the single most commonly leaked cloud credential type because they are portable, permanent and easy to email. Where a provider offers attached identity for a workload, whether an instance profile, a managed identity or a workload identity binding, the key file exists only because somebody built it that way before the alternative was available or convenient.
So the first pass is subtraction. Inventory the secrets you hold, mark each as eliminable or not, and eliminate before you design rotation. Teams that skip this step end up building rotation automation for credentials that could simply have gone away, which is expensive engineering in service of a control that was avoidable.
Why do rotation policies not get followed?
Because rotation is a two-credential problem and most systems are built for one. To change a credential without downtime you need a window in which both the old and the new value are accepted, so consumers can pick up the new one at their own pace. When the target supports only a single password or key, rotation means a simultaneous, coordinated change across every consumer, which is an outage risk, so it gets deferred and then deferred again.
The second obstacle is that nobody knows who consumes a given secret. A key created three years ago for one integration is now used by two scheduled jobs, a partner, and a script on somebody's machine, none of which are documented. Rotating it is a gamble on discovering the consumers through the incidents that follow, and that experience once is enough to end a rotation programme.
Both are solvable with mechanism rather than discipline. Prefer targets that support two active credentials, so rotation becomes create, deploy, verify, then revoke, with a rollback at every step. Use last-used metadata and access logs to enumerate consumers before you touch anything. And rotate on a cadence frequent enough that the process stays exercised, because the risk of rotation is highest when it has not been done for a year.
| Secret type | Can it be eliminated? | Rotation mechanism | What breaks if you get it wrong |
|---|---|---|---|
| Cloud access key in a pipeline | Yes, use OIDC federation | Not needed once removed | Nothing, this is the safest change on the list |
| Service account key file | Usually, use attached workload identity | Two active keys, then delete the old | Any consumer you did not know about |
| Database password | Often, use provider database authentication | Managed rotation with two users, or dual passwords | Long-lived connections and pooled clients |
| Third-party API key | Rarely, the vendor decides | Depends on the vendor supporting two keys | Integrations mid-request, and webhook delivery |
| Webhook signing secret | No | Accept both old and new during overlap | Silent rejection of valid inbound events |
| TLS private key | No, but automate issuance | Automated renewal well before expiry | Everything, loudly, at the worst moment |
| Encryption key in a key service | No | New version, old versions kept for decrypt | Data encrypted under a version you destroyed |
Where do secrets actually leak from?
Rarely from the secrets store, and usually from the places a secret passes through on its way to being used. Infrastructure-as-code state files are the most underrated: several tools record resolved values in state, in plain text, so a database password ends up in a file stored in a bucket that a wider group can read than anyone intended. Treat state as a secret store in its own right, encrypted, versioned, and access-controlled accordingly.
Then continuous integration logs, where a debugging line prints the environment, and job output is retained for months and often readable by anyone in the organisation. Then container image layers and machine images, where a credential added in one layer remains recoverable even if a later layer deletes it. Then version control history, where the fix was a commit that removed the file rather than a rotation. Then error tracking, which faithfully captures request headers including authorisation.
Finally the human channels: chat messages, tickets, shared documents, screenshots and the occasional pasted snippet into an external tool. None of this is preventable by policy alone, which is the argument for short-lived credentials over careful handling of long-lived ones. A token that expires in an hour is not much of a prize wherever it ends up.
What does a managed secrets service actually buy you?
Three things: an access record, per-consumer permissions, and the plumbing for versioned rotation. The access record is the one people undervalue. Knowing which identity read which secret and when converts an incident from guesswork into an investigation, and it is not available when the secret lives in a configuration file or an environment variable set at deploy time.
It does not make the secret harder to read for anyone with permission to read it, and this is the distinction worth being clear about internally. A compromised workload with permission to retrieve a secret retrieves it, store or no store. What the store gives you is the ability to scope that permission narrowly per secret rather than per environment, to see the retrieval, and to change the value in one place with old versions retained.
It is also worth separating the key service from the secrets service in your thinking. A key management service holds key material that never leaves it and performs cryptographic operations on request, which is what makes envelope encryption possible and what makes key policy a meaningful control. A secrets service returns a value to the caller. Different guarantees, different threat models, and the key policy is the one worth reviewing carefully because it can be written to allow more than intended.
What should happen in the hour after a leak?
Rotate first, investigate second. The instinct to determine the scope of exposure before invalidating the credential is understandable and wrong, because every minute spent scoping is a minute the credential still works. Revoke or rotate, confirm the old value no longer authenticates, and only then work out what was reachable and whether anything was accessed.
Assume the credential was used, then check. Last-used metadata on cloud credentials and access logs on the secrets service are usually enough to establish whether there was activity you cannot attribute, and the absence of data plane logs is itself a finding to record. Look specifically for actions that establish persistence, such as new credentials, new trust relationships or new roles, because those outlast the rotation you just performed.
Then close the route rather than the instance. A key in a repository means your pipeline had no secret scanning; a key in a log means somebody printed the environment; a key in an image means it was passed at build time. Rotating without fixing the route means the next occurrence is a matter of time, and history rewriting in version control is not a remediation, because copies exist.
What can you test this week?
Rotate one production secret on a Tuesday afternoon. Not a rehearsal in a test environment, and not the least important credential you own. If the prospect is frightening, you do not have rotation, you have a policy, and the fear itself is the finding: it means the consumers are unknown or the target supports only one credential at a time. Both are fixable, and neither gets fixed while rotation stays theoretical.
Alongside it, run a history scan across your repositories, including branches and old commits, and a search of your continuous integration logs for the string patterns your providers use for key material. Both are quick, both routinely find something, and the results tell you which route to close first.
Then produce the inventory that makes all of this manageable: every long-lived credential, its owner, its consumers, its age and whether an identity-based replacement exists. The list is usually shorter than expected once federation removes the pipeline keys, and a short list of owned secrets is the only version of this control that survives a year.
Common questions
- How often should cloud secrets be rotated?
- Frequently enough that the process stays exercised, which for most estates means quarterly at minimum for the credentials that remain. The more useful question is which secrets should exist at all. Pipeline access keys and service account key files can usually be replaced by workload identity federation, and database passwords by the provider's own database authentication, which removes the rotation requirement rather than scheduling it.
- Why does secret rotation keep getting postponed?
- Because it needs two valid credentials at once and many systems accept only one, so rotation becomes a simultaneous change across every consumer with no rollback. The second reason is that consumers are undocumented, so rotating a three-year-old key means discovering who used it through the failures that follow. Prefer targets supporting two active credentials, and enumerate consumers from last-used metadata and access logs first.
- Where do cloud credentials most often leak from?
- Not usually the secrets store. Infrastructure-as-code state files record resolved values in plain text and often sit in a widely readable bucket. Continuous integration logs capture printed environments and are retained for months. Container image layers keep credentials even after a later layer deletes them. Version control history keeps them after the file is removed, and error tracking captures authorisation headers with request data.
- What is the difference between a key management service and a secrets manager?
- A key management service holds key material that never leaves it and performs cryptographic operations on request, which is what enables envelope encryption and makes key policy a real control. A secrets manager stores a value and returns it to an authorised caller. The guarantees differ: with a key service the secret cannot be extracted, with a secrets manager it can, by anyone holding read permission.
- What should you do first when a secret is exposed?
- Rotate or revoke immediately, then investigate. Scoping the exposure first feels responsible but leaves the credential working while you work. After invalidating it, check last-used metadata and access logs for activity you cannot attribute, and look specifically for persistence such as new credentials, roles or trust relationships, since those survive the rotation. Then close the route that leaked it.
- Does a secrets manager stop a compromised application reading secrets?
- No. Any workload with permission to retrieve a secret will retrieve it whether the value comes from a managed service or a file. What the service adds is narrow per-secret permissions instead of per-environment ones, a record of which identity read what and when, and versioning that makes rotation possible without a coordinated outage. The audit record is the part most teams undervalue.
