# IT Runbook / SOP: [RUNBOOK NAME, e.g. Production Incident Response — Web Application Outage]

## Document Control

| Field | Entry |
|---|---|
| SOP ID | [RB-IT-001] |
| Version | [1.0] |
| Effective Date | [YYYY-MM-DD] |
| Owner | [SRE LEAD / IT OPS MANAGER NAME] |
| Approver | [HEAD OF ENGINEERING / CTO NAME] |
| Next Review Date | [YYYY-MM-DD — review quarterly; runbooks rot faster than policy SOPs] |
| Systems Covered | [SERVICE NAMES, ENVIRONMENTS, e.g. api.example.com, prod only] |

## 1. Purpose

Give any on-call responder — including someone who has never touched this service — a tested path to detect, triage, mitigate, and close [INCIDENT TYPE, e.g. a production web-application outage], meeting the response targets in Section 7.

## 2. Scope

- **Applies to:** [SEV1/SEV2] incidents on [SERVICES] in [ENVIRONMENT].
- **Does not apply to:** [e.g. "security incidents — page the security on-call and switch to RB-SEC-001; data-loss events — RB-IT-004"].

## 3. Definitions

| Term | Definition |
|---|---|
| SEV1 | Full outage or data integrity risk; all-hands response; comms every [30 MIN] |
| SEV2 | Degraded service with workaround; response within [30 MIN]; comms every [2 H] |
| Incident Commander (IC) | Single owner of coordination and comms; does **not** debug hands-on |
| MTTR / MTTA | Mean time to restore / acknowledge — tracked in [TOOL] |
| Mitigation vs fix | Mitigation restores service (rollback, failover); the root-cause fix can come later |

## 4. Responsibilities

| Role | Responsibility |
|---|---|
| On-call Engineer | Acknowledges the page within [5 MIN]; runs this runbook; becomes IC if no one else does |
| Incident Commander | Declares severity, coordinates responders, owns stakeholder comms, calls the end of the incident |
| Service Owner [TEAM] | Provides deep expertise; approves risky mitigations ([DB FAILOVER, CACHE FLUSH]) |
| Comms Lead [IF SEV1] | Posts status-page and internal updates on the cadence above |

## 5. Required Access and Tools

Verify these **during onboarding, not during an incident**:

- On-call access: [PAGERDUTY/OPSGENIE] schedule, [SLACK] #incident channel creation rights
- Dashboards: [GRAFANA/DATADOG URL — pin the "SERVICE overview" dashboard]
- Logs: [LOG SYSTEM + SAVED QUERY LINK]
- Deploy/rollback rights in [CI/CD TOOL]; break-glass credentials via [VAULT PATH] (use is audited)
- Status page admin: [STATUSPAGE URL]

## 6. Procedure

Example flow below is a web-app outage triggered by a bad deploy; adapt triggers and commands to your stack. Record timestamps for every action in the incident channel — the timeline writes your postmortem.

1. **Acknowledge the page** in [TOOL] within [5 MIN]. Open a dedicated channel: `#inc-[YYYYMMDD]-[short-name]`. Post: what alerted, when, and that you are investigating.
2. **Confirm real user impact.** Check [DASHBOARD]: error rate above [1%]? p95 latency above [THRESHOLD]? Synthetic checks failing from multiple regions? A single flapping probe is not an incident — see [ALERT TUNING DOC].
3. **Declare severity and IC.** If impact is confirmed, declare [SEV LEVEL] in the channel ("I am IC"). For SEV1, page [SERVICE OWNER TEAM] and assign a Comms Lead.
4. **Check the last change first.** Most outages follow a change. Run [DEPLOY HISTORY COMMAND / URL]. If a deploy, config push, or feature-flag change landed within [60 MIN] of impact start → go to step 5. Otherwise → step 6.
5. **Mitigate by rollback.** Roll back with [ROLLBACK COMMAND, e.g. `deployctl rollback api --to previous`] or disable the flag in [FLAG TOOL]. Watch [DASHBOARD] for [10 MIN]: error rate back under [0.1%] = mitigated → step 8. Rollback needs no approval; it is always the default first move.
6. **Triage by layer if no recent change.** In order: (a) upstream dependency status pages [LINKS]; (b) infrastructure — node/pod health via [COMMAND]; (c) database — connections, replication lag via [QUERY/DASHBOARD]; (d) traffic anomaly — possible attack, engage [SECURITY]. Post findings as you go; silence in the channel reads as "no one is working on it."
7. **Apply the matching mitigation.** [DEPENDENCY DOWN → enable degraded mode via FLAG]; [NODE FAILURE → cordon and reschedule]; [DB SATURATION → kill long-running queries per RB-DB-002, then failover with Service Owner approval]. One change at a time; verify after each.
8. **Communicate resolution.** IC confirms [15 MIN] of clean metrics, posts "mitigated" internally, updates the status page, and downgrades or closes the incident. Note anything left in a degraded/manual state.
9. **Close out within [48 H].** File the postmortem from [TEMPLATE LINK]: timeline, impact (duration, users/requests affected), root cause, action items with owners and due dates. Blameless — name systems and gaps, not people.

## 7. Response Targets

| Metric | SEV1 | SEV2 |
|---|---|---|
| Acknowledge | [5 min] | [15 min] |
| Status-page update | [30 min], then every [30 min] | [2 h] |
| Mitigation target | [1 h] | [4 h] |
| Postmortem due | [48 h] | [5 business days] |

## 8. Safety and Compliance Notes

- Break-glass credential use is logged and reviewed within [24 H]; never share them in chat.
- If customer data exposure is suspected at any point, stop and page [SECURITY ON-CALL] — breach-notification clocks ([GDPR 72-HOUR / CONTRACTUAL SLA]) may apply.
- Do not run destructive commands ([DATA DELETION, IRREVERSIBLE MIGRATIONS]) during an incident without a second responder confirming the command in the channel.
- Incident records retained [PERIOD] in [TOOL] for audit and SLA reporting.

## 9. Related Documents

- [RB-SEC-001 — Security Incident Response]
- [RB-DB-002 — Database Failover]
- [DOC — Alert Tuning Guidelines] / [TEMPLATE — Postmortem]

## 10. Revision History

| Version | Date | Author | Summary of Changes | Approved By |
|---|---|---|---|---|
| [1.0] | [YYYY-MM-DD] | [NAME] | Initial release | [NAME] |
| [ ] | [ ] | [ ] | [ ] | [ ] |

---
Template by Pellucida (pellucida.ai) — turn this SOP into a training video at pellucida.ai/solutions/sop-to-training-video