OnCallr

The Challenge

Incident response must remain dependable even when responders do not acknowledge immediately, browser sessions disconnect, or services restart. OnCallr needed durable escalation timers, schedule-aware assignment, secure service-specific webhook ingestion, real-time operator feedback, a complete event history, role-aware actions, and useful post-incident analysis without fragmenting the experience across separate tools.

The Impact

The platform centralizes on-call operations into one product, reduces manual responder coordination, preserves a traceable incident history, helps teams understand MTTA and MTTR, and turns resolved incidents into editable AI-assisted learning documents.

Incident Operations Dashboard

Escalation & Scheduling

• Create escalation policies with ordered user or schedule-based steps.
• Build schedules from rotation members and generate future coverage shifts.
• Support shift overrides and swaps for real operational changes.
• Enqueue BullMQ escalation jobs when incidents are assigned.
• Reassign unacknowledged incidents when timeout windows expire.
• Keep escalation timers durable in Redis so they survive API process restarts.

Incident Operations

• Track triggered, acknowledged, and resolved incident states.
• Give engineers a focused My Incidents queue for assigned response work.
• Maintain a full incident event timeline from trigger through resolution.
• Support comments, acknowledgement, resolution, and administrator force-escalation actions.
• Push state changes into active dashboards and toast notifications in real time.
• Preserve operational context for later postmortem generation and analytics.

Postmortems & Analytics

• List resolved incidents that are ready for post-incident review.
• Generate structured postmortem drafts with the OpenAI Responses API.
• Allow teams to edit AI-generated content before saving the final record.
• Calculate MTTA, MTTR, incident totals, service volume, severity mix, and time-series trends.
• Measure busiest on-call coverage to make response load more visible.
• Filter analytics by date range and service for focused operational review.

Webhook Incident Pipeline

• Each service owns a unique webhook token that can be rotated by an administrator.
• Incoming webhook payloads include incident title, description, and severity.
• The Fastify route validates the service token before accepting the trigger.
• The backend creates the incident and its initial timeline event in PostgreSQL.
• Schedule and escalation policy data determine the first responder assignment.
• Socket.io notifies the responder while BullMQ schedules the acknowledgement timeout.

Durable Escalation Architecture

• BullMQ stores delayed escalation work in Redis instead of relying on in-process timers.
• Queue workers evaluate whether the incident remains unacknowledged when a timeout expires.
• The escalation service advances through ordered policy steps and records reassignment events.
• User-based steps assign a specific engineer while schedule-based steps resolve current coverage dynamically.
• Durable jobs allow escalation behavior to continue correctly after backend restarts.

Schedules & Coverage

• Administrators create schedules and define ordered rotation membership.
• Future shifts are generated so responder coverage can be resolved ahead of time.
• Overrides provide temporary replacement coverage without destroying the base rotation.
• Shift swaps allow engineers to exchange assigned coverage windows.
• Coverage data feeds both incident assignment and on-call workload analytics.

Authentication & Permissions

• Supabase magic links provide the initial frontend authentication experience.
• The application exchanges authenticated identity for a Fastify cookie-based JWT session.
• Protected dashboard routes prevent unauthenticated access to operational data.
• Engineer permissions focus on assigned incidents and response actions.
• Administrator permissions add service, schedule, policy, analytics, and force-escalation management.

AI-Assisted Postmortems

• Resolved incidents provide the source timeline and operational context for draft generation.
• The backend calls the OpenAI Responses API to produce structured postmortem content.
• Generated drafts remain editable so engineers can correct context and add human judgement.
• Final postmortem records are saved against the originating incident.
• The workflow accelerates documentation without treating AI output as an unreviewed final report.

Analytics & Operational Learning

• MTTA measures the delay between incident trigger and responder acknowledgement.
• MTTR measures the time required to move an incident from trigger to resolution.
• Status, service, severity, and time-based summaries expose incident patterns.
• On-call load analysis shows which responders or coverage windows carry the most activity.
• Date-range and service filters let administrators isolate meaningful operational slices.

Monorepo & Local Development

• api contains the Fastify server, Prisma schema, BullMQ jobs, and Socket.io delivery layer.
• web contains the Next.js dashboard and frontend integrations.
• docs contains deployment and demo guidance while features.md records product behavior.
• PostgreSQL and Redis provide the required local persistence and queue infrastructure.
• Prisma migrations and seed scripts create demo services, schedules, admins, and engineers.
• Separate frontend and backend development servers run on ports 3000 and 5000.

Quality & Deployment

• Frontend and backend packages each provide lint and production build checks.
• The Next.js dashboard is suited for deployment on Vercel.
• The Fastify API can run on Railway, Render, or Fly.io.
• PostgreSQL can use Supabase or Railway while Redis can use Upstash or Railway Redis.
• WEB_ORIGIN and public API or Socket URLs connect deployed frontend and backend services safely.
• SMTP and OpenAI credentials enable email fallback and AI postmortem generation outside local development.