Build · Create · Engineering
Incident response runbook
A severity-tiered response playbook (roles, response targets, escalation, comms) so responders follow a plan instead of debating process mid-outage.
You receive: A structured playbook (JSON) that passes the severity-coverage, ordered-target, role, escalation, and comms checks.
Part of Keep App Running
Example
A sample of what this play produces. Your result is generated for your inputs.
Service
stud.com
Context
Production is the Next.js app on Vercel (main branch) backed by Neon Postgres, with the process-isolated judge settling every run. Soft launch: paid fulfillment is gated, so the blast radius today is the live catalog, search, waitlist, and dogfood runs. One founder-operator (Dan Schmitz) wears every responder hat; the roles below are functional hats, not headcount, which is exactly why a written playbook matters: mid-outage there is nobody to debate process with.
Severities
- Level
- SEV1
- Definition
- stud.com is down or serving 500s on the homepage, play pages, or search: the whole front stage is dark. Example: a production deploy ships a schema/code mismatch (the class of failure behind the no-op-schema 500).
- Response Target Minutes
- 15
- Incident Commander Role
- eng-lead
- Steps
Order Role Action 1 on-call Confirm scope: curl https://stud.com/ and a play page; check the latest production deployment state via the GitHub deployments API. Declare SEV1 and note the last deployed SHA. 2 ic Roll back first, diagnose second: revert the offending commit on main so Vercel redeploys the last known-good build (the deploy pipeline auto-runs migrations, so prefer revert over manual DB surgery). If the Next.js app itself cannot build, fail over to the preview/ Vite SPA, the designated rollback path. 3 on-call Verify recovery: deployment state success plus a 200 on /, a play page, and search. Confirm Neon connectivity from the runtime logs. 4 ic Post the outage note and the all-clear on the comms channels, then open the RCA issue with the timeline while it is fresh. - Escalation Path
- on-call pages ic at declaration
- ic pulls in eng-lead if rollback does not restore service within 30 minutes
- eng-lead decides preview-SPA failover and any Neon support escalation
- Comms Channels
- stud-incidents (internal channel)
- GitHub incident issue (public timeline)
- Level
- SEV2
- Definition
- A core surface is degraded but the site is up: the judge worker rejects or times out on all runs, search returns empty, or the waitlist form fails. Buyers can browse but cannot get work settled.
- Response Target Minutes
- 60
- Incident Commander Role
- ic
- Steps
Order Role Action 1 on-call Reproduce and bound the blast radius: which plays or surfaces fail, since which deploy or dependency change. Check whether the failure is the known Turbopack/judge interaction class before assuming new breakage. 2 ic Mitigate: revert the implicated commit, or feature-gate the broken surface so it degrades to 'opens soon' rather than erroring (never a 500 on a paid action). 3 eng-lead Run the full local gate on the fix branch before re-deploying; a SEV2 fix that skips the gate is how a SEV2 becomes a SEV1. 4 ic Confirm settled runs resume end to end with one dogfood play, then log the incident and file follow-up issues. - Escalation Path
- on-call escalates to ic after 60 minutes without mitigation
- ic escalates to eng-lead for anything touching money data or the judge sandbox
- Comms Channels
- stud-incidents (internal channel)
- GitHub incident issue
- Level
- SEV3
- Definition
- Cosmetic or single-play defects with a workaround: one play's validator misfires, a render glitch on mobile, a stale sample on a play page. No settling path is blocked.
- Response Target Minutes
- 240
- Incident Commander Role
- on-call
- Steps
Order Role Action 1 on-call Capture the repro (URL, play slug, viewport) in a GitHub issue with the sev3 label; quantify extent before scoping the fix (grep the catalog for siblings of the same defect). 2 on-call If a buyer-visible surface is wrong, patch or hide the specific surface the same day; a misleading sample is worse than a missing one. 3 eng-lead Batch the durable fix into the normal PR flow through the gate; close the issue only when the fix is verified on production. - Escalation Path
- on-call promotes to SEV2 if the defect turns out to block settling or touches money display
- Comms Channels
- GitHub incident issue
Opens soon
STUD opens runs for everyone soon, on every play at once.
Scope of the check
What's verified: STUD verifies the submitted playbook (JSON) covers exactly your required severity levels, each with a positive response target that is monotonic by severity (more severe gets an equal-or-tighter target), the minimum steps per severity, roles drawn from your allowed set, uniquely ordered steps, an escalation path where required, and comms channels. STUD does NOT judge whether the targets are achievable or whether the plan will actually resolve an incident; it checks the playbook's structure and consistency.
Get early access to STUD the day it goes live.