How do you stop config drift between on-prem and cloud?
You do not stop it, and aiming to is how teams waste a year. A firewall rule and a cloud security group are not the same object, storage tiers do not correspond, and the resilience model on each side is built from different parts, so demanding identical configuration means constantly reconciling things that were never comparable. The achievable goal is different: make the intent match, make the divergence visible within a day, and make sure a change on one side cannot silently break the other.
What is configuration drift?
Two different things that share a name, and separating them clarifies most arguments about it. The first is drift between declared and actual state: something is defined in code or a build document, and the running system no longer matches because a change was made directly. The second is drift between two environments that were meant to behave alike, where each is internally consistent and they no longer agree with each other.
Hybrid estates suffer from both, and the second is the one that produces the memorable incidents. The cloud side and the on-premise side each work perfectly in isolation, then a request that traverses both hits a limit configured on one and not the other, and the symptom appears in the environment that is not at fault.
The reason it matters more here than in a single environment is that the two sides usually have different change processes. On-premise changes go through a review board and a maintenance window; cloud changes are self-service and immediate. The same organisation, applying its own rules honestly, ends up with one side changing weekly and the other quarterly.
Why does hybrid drift faster than a single estate?
Different primitives, different tooling, different teams, and different urgency. Each of the four contributes, and the combination is why hybrid drift outpaces what either environment would produce alone.
The primitive mismatch is structural. A network access rule on-premise may be expressed as a source, destination, port and an appliance policy; in the cloud it is a security group referencing another security group. Translating intent between them requires a person to decide what the equivalent is, and two people will decide differently. Nothing has gone wrong at this point, and the environments already differ.
The urgency difference is what makes it compound. Emergency changes are made by hand, at unsociable hours, by whoever is on call, and the ticket to fold that change back into code is written with genuine intent and then not done. Any estate that has had a few incidents has a set of manual changes that exist only in the running system, and hybrid estates have two such sets.
Does infrastructure as code solve it?
It solves half of it, and the half it does not solve is the half that causes outages. Defining infrastructure in code makes the intended state explicit and repeatable, which is genuinely valuable. It does nothing to prevent someone changing the running system directly, and by default nothing notices when they do.
The missing piece is a scheduled comparison that runs whether or not anyone is deploying. Running the tool in a mode that reports differences without applying them, on a timer, and raising an alert when the running state has moved away from the declared state, is what converts infrastructure as code from documentation into a control. Most teams have the first and not the second, which is why they are surprised by drift despite having the code.
There is a caveat specific to hybrid. Cloud resources are usually fully described by the tool, while on-premise infrastructure is often only partly covered, with the physical network, appliances and hypervisor settings managed elsewhere. Drift detection that covers only the cloud side gives a false sense of coverage precisely where the interfaces between the environments are configured.
Which kinds of drift actually cause outages?
A small and predictable set, which is useful because it means detection can be targeted rather than exhaustive. The pattern shared by all of them is that the drifted setting is not exercised until the moment it matters.
Certificate expiry deserves particular mention because it is not drift in the usual sense and behaves exactly like it. A certificate rotated on one side and not the other, or an internal root distributed to on-premise systems and not to cloud workloads, produces an integration that stops at a precise moment for a reason nobody changed that day.
| Drift type | How it happens | How to detect it | What it breaks |
|---|---|---|---|
| Firewall and security rules | An emergency allow rule added by hand and never removed | Scheduled diff of running rules against declared policy | Security posture, and traffic that should not flow |
| TLS certificates and trust stores | Rotated on one side, or a root not distributed to the other | Expiry monitoring covering both estates in one report | The integration between environments, at an arbitrary hour |
| Timeouts and connection limits | Tuned during an incident on the side that was struggling | Compare against declared values, not against each other | Requests that traverse both, failing on the healthy side |
| Software and OS versions | One side patched on a cycle, the other on a change window | Inventory both estates into one list | Protocol and cipher mismatches that appear as odd errors |
| Environment variables and secrets | Copied by hand into the second environment | Compare key names, never values, across both | Behaviour that differs only in one environment |
| Quotas and service limits | A cloud limit nobody set and nobody read | Alert on approaching limits, not on breaching them | Scaling that silently stops during the busiest hour |
What should be identical and what should not?
Intent should be identical. Implementation should not, and forcing it to be produces worse systems on both sides. The useful line is between the contract and the mechanism: the contract is what the environment promises about behaviour, and the mechanism is whatever each platform provides to deliver it.
Things worth holding identical: the version of your own software, the names of configuration keys and secrets, the policy intent for who may reach what, timeout and retry values on anything that crosses the boundary, and the observable behaviour of health checks. Each of these produces a real defect when it differs, and none of them is expensive to keep aligned.
Things not worth holding identical: instance sizes, the resilience topology, storage classes, backup mechanisms and the network implementation. Insisting a cloud deployment mirror the on-premise topology usually means paying cloud prices for a design shaped by a hardware purchasing cycle, which is the most common way hybrid estates end up more expensive than either option alone.
How do you find the drift you already have?
Pick the three settings most likely to hurt and check them by hand this afternoon: the certificate expiry dates on both sides of every integration, the timeout values configured at each hop of one important request path, and the firewall or security rules governing traffic between the environments compared against what the documentation claims.
That exercise takes a few hours and reliably produces findings, because these are the settings that are changed under pressure and documented afterwards. It also gives you the honest starting point that a tooling decision needs, since the answer to whether you need drift detection is different when you already have a handful of undocumented differences.
After that, the sequencing is straightforward. Put the intent in code where it is not already, add a scheduled comparison that alerts rather than applies, and give the boundary between the environments its own review whenever either side changes. The last one matters most: most hybrid drift incidents are a change made correctly on one side by someone with no visibility of the other.
Common questions
- What is configuration drift?
- Two distinct problems sharing a name. The first is divergence between declared and actual state, where infrastructure is defined in code or documentation and the running system no longer matches because a change was made directly. The second is divergence between two environments meant to behave alike, where each is internally consistent but they no longer agree with each other. Hybrid estates experience both, and the second causes the more confusing incidents.
- Why do hybrid cloud environments drift apart faster?
- Four reasons compound. The platforms offer different primitives, so translating a rule from one to the other requires a human judgement that two people would make differently. The tooling differs. The teams differ. And the change processes differ, with cloud changes typically self-service and immediate while on-premise changes wait for a review board and a maintenance window, so one side changes weekly and the other quarterly.
- Does infrastructure as code prevent configuration drift?
- It solves half the problem. Defining infrastructure in code makes the intended state explicit and repeatable, but it does not prevent direct changes to the running system and by default nothing detects them. The missing control is a scheduled comparison that runs on a timer, reports differences without applying them, and raises an alert when running state has moved away from declared state.
- Should on-premise and cloud environments be configured identically?
- No. Intent should match and implementation should not, because the primitives genuinely differ. Hold identical: your own software versions, configuration key and secret names, policy intent for access, timeout and retry values on anything crossing the boundary, and health check behaviour. Do not hold identical: instance sizes, resilience topology, storage classes, backup mechanisms and network implementation, since mirroring an on-premise topology usually means paying cloud prices for a design shaped by a hardware purchasing cycle.
- Which configuration differences cause the most hybrid cloud outages?
- Certificates rotated on one side and not the other, timeout values tuned during an incident on one environment only, firewall rules added by hand in an emergency, and cloud service quotas nobody set or read. They share a trait: the drifted setting is not exercised until the moment it matters, so nothing surfaces the difference in advance and the failure often appears in the environment that is not at fault.
- How do you audit configuration drift between environments?
- Start manually with three checks that reliably find something: certificate expiry dates on both sides of every integration, the timeout values at each hop of one important request path, and the rules governing traffic between the environments compared against documentation. These are the settings changed under pressure and documented afterwards. Follow with scheduled automated comparison, and give the boundary its own review whenever either side changes.
