Incidents

Incidents turns alerts from Prometheus Alertmanager or Grafana into incidents, posts them into a team room and pages whoever is on call. If nobody acknowledges in time, the page escalates to the next person. When the alert clears, the incident resolves itself.

Where to find itCopy Anchor Link

Open Incidents from the sidebar's app row or All apps (/incidents). Its sidebar:

An incident opens at /incidents/<id>; its breadcrumb leads back to the list. The settings page has the same breadcrumb. In the search palette (⌘K) the settings page is called Incident rotations and endpoints.

Roles and permissionsCopy Anchor Link

Roles: Responder, Manager, Administrator, granted at System console → Access → Roles. Responders see, acknowledge and resolve incidents. Creating and changing rotations and endpoints needs the permission to manage incidents; everybody else sees the settings page read-only, and the page says so.

Main workflowsCopy Anchor Link

Respond to a pageCopy Anchor Link

1. The page arrives as a time-sensitive notification and as a post in the endpoint's room.
2. Open the incident and press Acknowledge — this stops the escalation.
3. Add a note with what was found, tried or decided.
4. Press Resolve when it is fixed (or let it resolve when the alert clears).
5. Reassign to… hands it to a colleague.
6. When the endpoint has a Follow-up board, resolving the incident leaves an item on that board; the incident then shows it under Follow-up.

12345

1. 1AcknowledgeTells the team you are on it and stops the escalation.
2. 2EscalatesUntil acknowledged, the page moves on to the next person on call.
3. 3Add a noteWhat was found, tried or decided; notes land on the timeline below.
4. 4ResolveCloses the incident once it is fixed.
5. 5Reassign to…Hands the incident to a colleague.

Find an incidentCopy Anchor Link

1. Open Incidents.
2. Which incidents: Open, Triggered, Acknowledged, Resolved or All; pick a Severity; search title or summary; limit Opened between (today, last 7 days, last 30 days).
3. The header shows the Median time to acknowledge for what is listed; Median time to resolve joins it once the list holds resolved incidents (Resolved or All). Neither appears while more incidents are still to load.
4. When nothing matches, Clear filters goes back to the open incidents. For whoever may manage incidents, a list with no incidents at all links to Rotations and endpoints, where alerts are wired up.

12345

1. 1Which incidentsOpen, Triggered, Acknowledged, Resolved or All.
2. 2SeverityShows only incidents of one severity, here Critical.
3. 3Opened betweenLimits the list to today, the last 7 days or the last 30 days.
4. 4Search title or summaryFinds an incident by its words.
5. 5Median time to acknowledgeFor what is listed; the median time to resolve joins it once incidents are resolved.

Open an incident by handCopy Anchor Link

For something no alert caught: Open incident on the list, choose the endpoint (it decides the room and the rotation), give it a title and Open and page.

Set up on-callCopy Anchor Link

1. Open Rotations and endpoints.
2. New rotation: name, daily or weekly handover, first day, handover time and time zone, how many minutes to wait before escalating, and the people in order.
3. When somebody swaps a shift, press Edit on the rotation and, under Overrides, pick the Person, From and Until, then Add override. The overrides are listed there, not on the rotation itself. People with an approved absence in Time are skipped automatically.

1234

1. 1On-call rotationsWho is paged first and how the page escalates; people away in Time are skipped.
2. 2People, in orderThe rotation’s people in the order they take over, with who is on call now.
3. 3New rotationName, daily or weekly handover, time zone, escalation delay and people.
4. 4New endpointConnects an alert source: name, source, room and the rotation to page.

Connect an alert sourceCopy Anchor Link

1. New endpoint: name, source (Prometheus Alertmanager or Grafana), the room to post in (not end-to-end encrypted) and the rotation to page, then Create endpoint. Its Follow-up board, set on the endpoint in the list, is optional.
2. Copy the URL and the secret — the secret is shown only once — into the alert source. The operator runbook docs/runbooks/wiring-alertmanager-and-grafana-to-incidents.md has the exact configuration.
3. Rotate secret replaces it; the old one stops working at once.

1234

1. 1The secretShown only now: copy it into the alert source before you leave.
2. 2Copy URLThe address the alert source posts to.
3. 3Follow-up boardOptionally, the board that follow-up items go to.
4. 4Rotate secretReplaces the secret; the old one stops working at once.

Settings and adminCopy Anchor Link

• Rotations and endpoints (/incidents/settings).
• Installing the module: System console → Marketplace → Incidents.
• Roles: System console → Access → Roles.

TipsCopy Anchor Link

• Keep the escalation delay short for production alerts and longer for business-hours services.
• Deleting an endpoint deletes its incidents; prefer rotating its secret if it leaked.