Building an incident response runbook people will follow at 3am
A forty-page incident response policy is not a runbook, and the two are confused because the same audit question accepts either. At 3am the reader is one tired person, working alone, on a phone, possibly on a network they have just been told not to trust. What they need is a page: the first few actions, the things not to do, a name and a number for whoever can authorise impact, and how to preserve evidence while they work. Everything else belongs in an appendix nobody will open.
Why do most runbooks fail when they are needed?
Because they were written for an assessor rather than for a responder. The document explains the phases of incident management, names committees, and defines severity in abstract terms, none of which help someone deciding whether to isolate a host in the next four minutes. The test is simple: hand it to an engineer who was asleep ten minutes ago and see whether they can act from it without reading past the first page.
The second failure is availability. Runbooks stored in the corporate wiki, behind the identity provider that has just been compromised, or on a file share encrypted by the thing you are responding to, are unavailable exactly when they are required. A current copy has to exist somewhere that does not depend on the systems in scope, which usually means printed, in an offline note store, or in a separately administered location.
The third is authority. Most runbooks describe technical steps and go silent on permission, so the responder reaches a step that would interrupt production and stops to find someone. That pause is often the longest interval in the whole incident and it is entirely preventable in advance, with a written statement of who may authorise what and which actions need no authorisation at all.
What must one page contain?
Six elements and no more. The first three or four actions in order, phrased as instructions rather than principles. An explicit do-not list, because the harmful actions at 3am are usually enthusiastic rather than negligent. The name, role and mobile number of whoever can authorise business impact, plus a deputy. How to preserve evidence for this scenario specifically. The first external notification consideration, if any. And where the full detail lives for whoever picks it up in the morning.
Write per scenario, not per phase. A responder at 3am does not need generic guidance, they need the ransomware page or the business email compromise page or the compromised cloud credential page. Six scenario pages covering what plausibly happens in your estate will be used; a single comprehensive document covering everything will not.
Keep it in the same repository as the platform configuration if your team lives there, and print it as well. The version that gets followed is the one the responder can reach without thinking, and duplicating it into two independent locations is cheaper than any of the alternatives to having it.
What do the first actions look like by scenario?
Specific to the scenario and short enough to memorise. The pattern below is illustrative rather than prescriptive, because the right first action depends on your architecture, but the shape, first action, prohibition, and who to wake, is what each page needs.
| Scenario | First action | Do not | Who to wake |
|---|---|---|---|
| Ransomware precursor on a server | Isolate the host through the endpoint tool, keep it powered on | Power off, reimage, or start restoring from backup yet | Infrastructure lead and the executive with authority to stop production |
| Business email compromise | Revoke active sessions and refresh tokens, then reset credentials | Rely on a password reset alone, since existing sessions survive it | Identity administrator, and finance if payment instructions are involved |
| Cloud credential or API key in use by an unknown party | Restrict the credential, then review what it has already done in audit logs | Delete the identity before its activity has been reviewed | Cloud platform owner, plus whoever owns the affected accounts |
| Compromise inside a container workload | Cordon the node and remove the pod from service without deleting it | Delete the pod or let autoscaling replace it, which destroys the evidence | Platform engineer on call, and the service owner |
| Laptop with a confirmed endpoint detection | Isolate through the endpoint tool and contact the user by phone | Email the user, since the mailbox may be in scope | Nobody until scope is known, unless the user is privileged |
| Third party reports your data is for sale | Preserve the claim, start a formal incident, extend log retention immediately | Engage publicly, or contact the claimant without guidance | Legal counsel and the insurer notification contact |
Who has authority to disconnect production at 3am?
Somebody named, with a phone number, and a deputy behind them, and this is the part of the exercise that requires an executive rather than a security team. The useful mechanism is standing pre-authorisation: a written statement that specified containment actions may be taken immediately without seeking approval, and that the responder will be supported for taking them, with a defined list of assets where that does not apply and approval must be sought.
That list of exceptions is the real work. Systems where isolation carries safety, clinical, regulatory or contractual consequences need to be identified in advance, along with who can weigh those consequences at night. Producing it forces a conversation about which systems the business will accept downtime on, which is precisely the conversation that cannot be held during an incident.
Include the insurer and counsel path in the same place. Many cyber policies require notification within a stated period and require that forensic providers and legal support be engaged from an approved panel, and engaging your usual supplier first can affect cover. In some jurisdictions, instructing forensic work through counsel affects whether the resulting report attracts privilege, and the position varies, so the runbook should record who to call rather than attempt to summarise the law.
How do you preserve evidence while containing?
Contain in the way that keeps the machine, not the way that removes it. Network isolation through an endpoint tool preserves memory, running processes and the filesystem while cutting the intruder's access, and it is almost always preferable to shutting down. Memory is the most volatile evidence and the most valuable in a modern intrusion, so anything that reboots or reimages a host destroys the record of what actually ran.
In cloud and container environments the equivalent moves are different and need to be written down explicitly. Snapshot volumes before terminating an instance. Suspend the autoscaling group so a replacement does not obscure the timeline. Cordon the node and remove the pod from its service rather than deleting it, since a deleted pod is gone in a way a quarantined server is not. And extend log retention as a first action, because default windows continue expiring while you investigate.
Write down the order once, in the scenario page, in the imperative. Preservation decisions are made in the first ten minutes by whoever is present, and by the time a specialist joins the call the opportunity has usually passed. This is the part of a runbook with the shortest window and the longest consequences.
How do you keep it true?
Treat the runbook as something that decays, because it does. Phone numbers change, the endpoint tool's isolation feature moves in the interface, the person named as authoriser leaves, and the cloud account structure is reorganised. A runbook that has not been touched in a year is a document about an estate that no longer exists, and it will be discovered to be wrong at the least convenient moment.
Tie updates to two triggers. Every real incident produces corrections, and the review that follows should end with edits to the specific pages used, not with a report. Every exercise produces the same, and the pages that nobody could follow during the exercise are the ones to rewrite first.
Test the parts that are not technical, since those fail most quietly. Once a quarter, dial the out-of-hours numbers in the runbook and see who answers. Confirm that the offline copy exists and is current. Ask the named authoriser whether they know they are named. That takes under an hour and finds more genuine problems than a rewrite of the document ever does.
Common questions
- What should an incident response runbook contain?
- One page per scenario with six elements: the first three or four actions in order, an explicit do-not list, the name and mobile number of whoever can authorise business impact plus a deputy, how to preserve evidence for that specific scenario, the first external notification consideration, and where the full detail lives for the morning. Anything longer will not be read by someone who was asleep ten minutes earlier.
- Why do incident response plans fail during real incidents?
- Three reasons. They were written for an assessor, so they describe phases and committees rather than actions. They are stored somewhere the incident has made unavailable, such as a wiki behind a compromised identity provider or a file share that has just been encrypted. And they are silent on authority, so the responder reaches a step that would interrupt production and stops to find someone, which is often the longest delay in the whole response.
- Should you shut down a compromised machine?
- Generally no. Isolate it at the network level through your endpoint tool and leave it powered on. That cuts the intruder's access while preserving memory, running processes and the filesystem, and memory is both the most volatile evidence and the most valuable in intrusions that never write malware to disk. Shutting down, rebooting or reimaging destroys the record of what actually ran, which the investigation then cannot reconstruct.
- How do you preserve evidence in a container environment?
- Cordon the node and remove the pod from its service rather than deleting it, because a deleted pod takes its filesystem with it. Suspend autoscaling so a replacement does not obscure the timeline, snapshot volumes before terminating any instance, and extend log retention as an early action since default windows keep expiring while you investigate. These steps differ enough from server handling to be written explicitly in the runbook.
- Who should be allowed to isolate production systems out of hours?
- A named person with a deputy, backed by written standing authority that lists which containment actions may be taken immediately without approval and which assets are excluded. Producing the exclusion list is the real work, because it requires the business to state where isolation carries safety, clinical, regulatory or contractual consequences, and who can weigh those at night. That conversation cannot be held during an incident.
- How often should an incident response runbook be reviewed?
- After every real incident and every exercise, editing the specific pages that were used or that nobody could follow, rather than producing a report. Separately, test the non-technical parts quarterly: dial the out-of-hours numbers and see who answers, confirm the offline copy exists and is current, and check that the named authoriser knows they are named. That hour finds more genuine problems than rewriting the document.
