A strong incident response process for startup teams should be simple, role-driven, and executable under stress. Enterprise incident management frameworks with 50-page runbooks and six layers of escalation do not work for teams of 3-15 engineers. What works is a lightweight process that everyone has practiced, with clear roles, communication cadence, and blameless postmortems that actually produce improvements.
Research consistently shows that teams with mature, practiced incident response processes recover from failures dramatically faster than teams without one. The difference is not tooling or team size. It is process clarity and follow-through.
---
Core Response Principles
Every decision during an incident should pass through these four principles:
These principles matter most when things are chaotic. Print them on a card, pin them in Slack, and reference them at the start of every incident.
Incident Roles
Even a 5-person team should assign explicit roles during an incident. One person can hold multiple roles on a small team, but the responsibilities should be clearly separated.
| Role | Responsibilities | Who |
| Incident Commander (IC) | Coordinates response, makes decisions, runs timeline | On-call lead or engineering manager |
| Technical Lead (TL) | Diagnoses issue, implements fix, validates resolution | Senior engineer on affected system |
| Communications Lead | Updates stakeholders, customers, status page | Product manager or designated engineer |
| Scribe | Logs timeline, decisions, actions in incident channel | Any available team member |
Incident Commander Checklist
<h2>IC Checklist (copy-paste at incident start)</h2>
<ul><li>[ ] Declare incident in #incidents channel</li>
<li>[ ] Assign IC, TL, Comms, Scribe roles</li>
<li>[ ] Assess severity (SEV-1/2/3)</li>
<li>[ ] Set communication cadence</li>
<li>[ ] Create incident document/thread</li>
<li>[ ] Coordinate mitigation (TL owns execution)</li>
<li>[ ] Post status updates on cadence</li>
<li>[ ] Declare resolution when confirmed</li>
<li>[ ] Schedule postmortem within 48 hours</li>
</ul>
Severity Matrix
Define severity levels before you need them. During an incident is the wrong time to debate whether something is SEV-1 or SEV-2.
| Severity | Definition | Examples | Update Cadence | Escalation |
| SEV-1 | Complete outage or data breach | Site down, data leak, payment system broken | Every 15 minutes | CEO/CTO notified immediately |
| SEV-2 | Major feature degraded | Search broken, slow API (>5s), login failing for subset | Every 30 minutes | Engineering leadership notified |
| SEV-3 | Minor feature issue | UI glitch, non-critical endpoint slow | Every 2 hours | Team lead notified |
| SEV-4 | Cosmetic or low-impact | Typo in UI, minor logging error | Next business day | Normal issue tracking |
Severity Decision Tree
Is customer data exposed or at risk?
YES -> SEV-1 (security incident)
NO ->
Is the primary service unavailable?
YES -> SEV-1
NO ->
Is a major feature degraded for >50% of users?
YES -> SEV-2
NO ->
Is a feature partially broken for some users?
YES -> SEV-3
NO -> SEV-4
Incident Lifecycle Runbook
Phase 1: Detection and Declaration (0-5 minutes)
<h1>Automated detection via monitoring</h1>
<h1>Alert fires -> on-call engineer receives page</h1>
<h1>On-call engineer verifies alert is real</h1>
curl -s https://api.example.com/health | jq .
<h1>Check dashboard for error spike</h1>
<h1>Grafana: https://grafana.internal/d/api-overview</h1>
<h1>If confirmed, declare in Slack:</h1>
<h1>@incident-team SEV-2 INCIDENT: API response times >5s.</h1>
<h1>I am IC. TL: @engineer. Comms: @pm. Scribe: @dev2.</h1>
<h1>Cadence: updates every 30 minutes.</h1>
Phase 2: Assessment (5-15 minutes)
The Technical Lead determines blast radius while the IC coordinates:
- What services are affected?
- How many users are impacted?
- When did the issue start?
- What changed recently? (deploys, config changes, traffic spike)
<h1>Check recent deployments</h1>
kubectl rollout history deployment/api-server -n production
<h1>Check error rates by endpoint</h1>
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~5..}[5m])" | jq .
<h1>Check resource saturation</h1>
kubectl top pods -n production
Phase 3: Mitigation (15-60 minutes)
The goal is to restore service, not to find root cause. Common mitigation actions:
| Mitigation | When to Use | Risk |
| Rollback deployment | Issue started after a deploy | Low if rollback is tested |
| Restart pods | Memory leak or stuck process | Low, may recur |
| Scale up | Traffic spike causing saturation | Medium, increases cost |
| Feature flag disable | New feature causing errors | Low if flags are in place |
| Database failover | Primary DB unresponsive | Medium, requires validation |
| DNS redirect | Regional outage | High, needs pre-configuration |
<h1>Rollback to previous deployment</h1>
kubectl rollout undo deployment/api-server -n production
<h1>Verify rollback</h1>
kubectl rollout status deployment/api-server -n production
<h1>Check health after rollback</h1>
curl -s https://api.example.com/health | jq .
Phase 4: Communication
Use pre-written templates to communicate quickly and consistently.
#### Internal Incident Declaration Template
INCIDENT DECLARED - [SEV-X]
What: [Brief description of user impact]
When: [Start time UTC]
Impact: [Number of users / services affected]
IC: [Name]
TL: [Name]
Comms: [Name]
Status: Investigating
Next update: [Time UTC]
#### Customer-Facing Status Update Template
We are aware of [issue description] affecting [scope].
Our engineering team is actively working on resolution.
We will provide an update by [time UTC].
#### Resolution Announcement Template
RESOLVED - [SEV-X] [Brief description]
Duration: [Start] to [End] ([total minutes] minutes)
Impact: [What users experienced]
Resolution: [What fixed it]
Root cause: [Brief, if known]
Postmortem: Scheduled for [date]
Phase 5: Resolution and Validation
An incident is not resolved until:
- Error rates are back to baseline
- Performance metrics are within SLO
- No recurring alerts
- Customer-facing communications are sent
- Incident timeline is documented
Phase 6: Postmortem (within 48 hours)
<h2>Postmortem: [Incident Title]</h2>
Date: [Date]
Duration: [Start - End UTC]
Severity: [SEV-X]
IC: [Name]
<h3>Summary</h3>
[2-3 sentences of what happened and user impact]
<h3>Timeline</h3>
<ul><li>HH:MM UTC - Alert fired</li>
<li>HH:MM UTC - IC declared incident</li>
<li>HH:MM UTC - Root cause identified</li>
<li>HH:MM UTC - Fix deployed</li>
<li>HH:MM UTC - Resolved confirmed</li>
</ul>
<h3>Root Cause</h3>
[What actually broke and why]
<h3>What Went Well</h3>
<ul><li>[List things that worked]</li>
</ul>
<h3>What Went Poorly</h3>
<ul><li>[List things that did not work]</li>
</ul>
<h3>Action Items</h3>
Action Owner Due Date Priority
[Action] [Name] [Date] P1/P2/P3
On-Call Setup for Small Teams
| Team Size | On-Call Model | Rotation |
| 3-5 engineers | Primary only, escalation to founder/CTO | Weekly |
| 6-10 engineers | Primary + secondary | Weekly, offset by 3-4 days |
| 11-15 engineers | Primary + secondary + team lead escalation | Weekly per team |
<h1>PagerDuty / Opsgenie schedule example</h1>
schedule:
name: production-on-call
timezone: UTC
layers:
- name: primary
rotation: weekly
start: "09:00"
members:
- engineer-a
- engineer-b
- engineer-c
- name: escalation
rotation: monthly
members:
- tech-lead
- cto
escalation_policy:
- wait: 5m
target: primary
- wait: 15m
target: escalation
Use the Documentation section for runbook storage and the Services Overview for service ownership mapping. Need tailored support? Visit Contact.
Frequently Asked Questions
How do we run postmortems without blame?
Focus the language on systems and processes rather than individuals. Instead of "John deployed bad code," write "The deployment pipeline did not catch the regression because integration tests do not cover the affected endpoint." Action items should improve systems (add tests, improve monitoring, update runbooks) rather than prescribe individual behavior.
Should SEV-4 incidents get postmortems?
No. Postmortems are expensive in team time and attention. Reserve them for SEV-1 and SEV-2 incidents. For SEV-3, write a brief summary in the issue tracker with action items. For SEV-4, normal bug tracking is sufficient.
How do we prevent alert fatigue?
Route only actionable alerts to the on-call pager. If an alert fires and the on-call engineer does not need to take action, it should be a notification, not a page. Review alert volume weekly and aggressively tune or remove alerts with a low signal-to-noise ratio. Target fewer than 5 pages per on-call shift.
When should we page someone outside business hours?
Only for SEV-1 and SEV-2 incidents. If users are not actively impacted and the issue can wait until morning, it should wait. Chronic after-hours paging burns out engineers and degrades incident response quality over time.
How do we track action items from postmortems?
Create tickets in your issue tracker with the label "postmortem" and assign owners with due dates. Review open postmortem action items in weekly engineering meetings. If action items consistently go unfinished, either they are not important enough to be action items, or the team needs dedicated time to address reliability debt.
Final Takeaway
Startup incident response maturity comes from cadence and follow-through, not process complexity. Define roles and severity before you need them. Practice the process with game days. Run blameless postmortems and actually complete the action items. The goal is not to prevent every incident but to detect fast, mitigate fast, and learn from every failure.