Secrets in source control, and what to do after one leaks

Why is rewriting history the wrong first move?

Because removal from your branch does not remove the value from circulation. A commit that has been pushed has been fetched, mirrored and cached. Forks retain it independently. Continuous integration workspaces, developer laptops, artefact caches and repository backups hold it. On several hosting platforms an unreachable commit can still be retrievable by its identifier until the provider prunes it, which is a support request rather than a git command. Automated scanners watching public repositories index within minutes.

It is also expensive in a way that competes with the actual response. Rewriting history changes every subsequent commit identifier, invalidates open pull requests, requires every collaborator to re-clone or reset, and breaks any external reference to a commit. Doing that while a live credential is still valid means the team is coordinating a repository migration during an active exposure.

So the decision to clean history is a separate, later one, and it is genuinely worth doing in some cases: a public repository, a secret that cannot be rotated, or a compliance obligation to demonstrate removal. Make it after rotation, deliberately, with the platform's support process included where unreachable objects need pruning.

What is the correct order of operations?

Establish what the credential can do, rotate or revoke it, check whether it was used, then decide about history, then fix the mechanism that allowed it. The first step takes minutes and determines everything after it: a read-only key scoped to one bucket is a different incident from a cloud access key attached to an administrative role, and treating them identically either wastes the day or understates the damage.

Rotation means the old value stops working, which is not the same as issuing a new one. Systems that accept multiple valid keys will happily continue honouring the leaked one after a new key is deployed, and that is a common way an incident is closed while remaining open. Confirm invalidation by attempting to use the old credential and observing a rejection, and write that observation down.

Where rotation would cause an outage, sequence it: issue the new credential, deploy consumers, verify traffic on the new one, then revoke the old. Plan that sequence before starting, because the middle state is the risky one. If a credential cannot be rotated at all, escalate immediately rather than working around it, since the response then becomes containment through network controls, monitoring and compensating restrictions.

How do you tell whether the secret was used?

Query the provider's own audit trail for the credential identifier over the whole window from first commit to revocation, not from discovery. Cloud providers, payment processors, email services and code hosts all record authenticated calls with a key or token identifier, and that log is the only source that answers the question. Absence of unexpected activity is meaningful evidence; absence of logging is not.

Look for three signals specifically: use from an address range or region your systems do not run in, calls to operations your application never makes, and enumeration patterns such as listing buckets, describing instances or reading identity metadata. Those are what an opportunistic finder does first, because they are establishing what the key is worth before doing anything with it.

Then set the retention question aside early. Many providers keep detailed logs for a limited period, so if the commit is older than that window you cannot answer the question and should say so plainly in the write-up rather than implying no misuse occurred. That distinction matters if the incident is later reviewed by a regulator or a customer, and where notification duties apply they depend on jurisdiction and on what data was reachable, which is a question for counsel rather than for engineering.

Which secrets are hardest to deal with?

The ones with many consumers or no rotation path. Difficulty here is not about sensitivity, it is about how many things break when the value changes, and that determines how the incident is run.

Where do secrets hide besides commits?

Container image layers, and this is the copy teams most often miss. A layer is immutable, so a file added in one instruction and deleted in a later one remains fully present in the image and extractable by anyone who can pull it. Values passed as build arguments and values set with an environment instruction are readable in the image configuration without even unpacking it. Rebuilding without the value does not clean the images already published to a registry, so those tags need deleting as well.

Then the orchestration layer. Kubernetes secret objects are encoded, not encrypted, so anyone able to read the object in that namespace reads the value, and describing a workload can print environment configuration. Whether the data is encrypted at rest depends on cluster configuration rather than being a default property. Environment variables are also visible to anything that can read process information inside the container, which includes a compromised sidecar.

Finally, the places nobody scans: build logs where a variable was echoed during debugging, ticketing and chat systems where credentials get pasted for convenience, spreadsheets and password-protected documents, screenshots attached to support tickets, and database backups taken before the configuration changed. Scanning source control while ignoring these produces a false sense of the exposure surface.

How do you stop it happening again?

Remove the long-lived credential rather than protecting it better. Workload identity federation lets a build pipeline present a short-lived identity token and exchange it for temporary cloud credentials, which means there is no static key to leak, no rotation to schedule and nothing useful in a stolen environment file. This is the single change with the largest effect, and it is now standard capability across the major providers and code hosts.

Layer detection so it operates at push time, not review time. Server-side push protection that rejects a commit containing a recognised credential pattern stops the exposure before it becomes a public fact; a pre-commit hook helps the individual developer but is optional and bypassable, so it is a convenience rather than a control. Prefer detection that verifies candidates against the provider, because a verified live key is an unambiguous incident while an unverified match is noise.

Then do the one-off work: scan full history rather than the current tip, scan published images as well as repositories, and write a one-page playbook naming who can revoke each credential type out of hours. The playbook is the part that determines how the next incident goes, because the technical steps are simple and the delay is always in finding the person with permission.