Notification policies, escalation and maintenance windows

A policy decides who is told about which ticket, and how. It has three parts: what it matches, whom it tells, and how it repeats and escalates. Policies belong to an organisation and are managed by its administrators under Settings → Notifications → Policies.

What a policy matches

• From severity: warning, serious (the default) or critical.
• Categories: none ticked means all. The categories are Not responding, Slowdown, Errors, Resources, Network, Visitor experience, Security and Scanning.
• Scope: chosen machines, applications or IoT devices, or "a team's things", meaning everything a team was granted. Nothing ticked means everything the organisation owns.

Whom it tells

A policy can name people, teams and channels in any mix. People who switched their alert mail off, whose accounts are blocked, or who left the organisation are skipped. When several policies match one ticket, each recipient still gets one notice per event.

How

• Do not repeat the same thing within (minutes): 360 by default, up to a week.
• Also say when it resolves: off by default.
• Escalate after (minutes still open and not acknowledged), with an Escalation ring. Leave it empty to never escalate.

Escalation

Every minute, open tickets are checked against their policies. If a ticket is still open after the escalation time and nobody has acknowledged it, the escalation ring is told once: "Still open after N min, not acknowledged". If nobody in the ring can be reached, that failure is recorded in the delivery log. Pressing Acknowledge on the ticket stands the ring down. See Acknowledging and closing.

This ring is told once. For an order of several steps, with whoever is on call, use an escalation policy (below).

On-call schedules

A schedule says who answers for the organisation at a given moment. Find it under Settings → On-call. Everybody in the organisation can see who is on call now and who is next. Its owners and admins create and change schedules.

• Rotation: the people in order. Each holds the schedule for a day or a week, then hands it to the next.
• Handover: every day, or every week on a chosen weekday, at a time of day in the schedule's own time zone. The time stays the same on the wall clock across a daylight-saving change, so that shift is an hour shorter or longer. A handover time the clock skips happens that many minutes later; one the clock passes twice happens the first time.
• First shift from: the date the first person in the list starts.
• Cover: under Shifts, somebody else can be put on call for a period of up to 31 days, with an optional reason. Where two covers overlap, the newer one wins.

Shifts shows the next 14 days as one line per person, with cover marked.

Escalation policies

An escalation policy is an ordered list of steps, also under Settings → On-call. Each step names whom to tell, in any mix: whoever is on call for a schedule at that moment, people, teams and channels. Each step also has a wait: if nobody acknowledges the ticket within that many minutes, the next step is told. After the last step it can start again from step 1, up to ten more times (Then repeat).

A notification policy hands its tickets to one: in the policy's Escalation section, choose it under Hand the ticket to an escalation policy. Step 1 is told together with the policy's own message, and a person both name gets one letter. The later steps say "Still open after N min, not acknowledged (step 2 of 3)".

• It stops when somebody acknowledges the ticket, on the ticket page, through the API or from a link in a message, and when the ticket closes.
• It waits during the install's maintenance window, the machine's own maintenance window, while the ticket is muted by name, and while the AI filter holds the ticket. When the hold ends, the waiting step is told.
• Snooze 1 h makes it wait an hour. Then whoever was told last is told again, and it carries on from there.
• People who switched their alert mail off, or who left the organisation, are skipped, and the ticket's history says so.
• The same condition opening again inside the policy's repeat window, at the same or a lower severity, does not start the escalation again.

Its progress is kept in the database, so a restart or an update of the server carries on where it was. Each step is told once: after a long outage only the step that was due is told, and the next one is timed from then. A letter that was being sent at the moment the server stopped is recorded as unknown and is not sent a second time.

The ticket page's Response card shows who acknowledged and when, a snooze, and every step: who was told, by which route (for example "on call for Primary"), who was skipped and why, and when it stopped.

Acknowledge, resolve or snooze from the message

Every message about an open ticket, by email, Slack, Teams or webhook, carries three links: Acknowledge, Resolve and Snooze 1 h. Each link is signed by your install, names one ticket and one recipient, works once, and expires after 12 hours.

• Opening a link shows a page with one button. It does not act by itself, because mail security scanners open the links in a letter before you do. Pressing the button acts.
• The action is recorded in the recipient's name. For a channel, it is recorded as the channel's name with "(link)".
• When the button is pressed, the recipient is checked again. They must still be in the organisation and able to see the ticket, and a viewer cannot act. A browser signed in as somebody else is refused.
• Viewers get no links. A message about a closed ticket has none.

The webhook JSON carries the same links as actions.acknowledge, actions.resolve, actions.snooze and actions.expires_at, so an outside tool can offer the same buttons.

Who would be told

You should not have to wait for a real outage to know whether a policy works. The Who would be told card tries a ticket against your policies before you save anything. Pick a severity, a kind and a machine, or use a real ticket, and press Try. Each policy answers fires or quiet. A quiet policy gives its reason, such as "below serious". A firing one lists who it would tell, including who the escalation would reach after N minutes.

The delivery log

Delivery log lists recent deliveries, newest first. Each row shows when, the ticket, the event (opened, escalated, escalation ring, resolved or digest), the recipient, the policy that caused it, and the result. It answers "why did I get this", and "why didn't I".

Maintenance windows

For one machine

On a host's page, 🔧 Maintenance silences that machine for 1 hour, 1 day, 1 week, or until a time you choose, up to 90 days. You can give a reason, which is shown while it lasts. During the window no email, webhook, policy, personal or workflow delivery goes out about the machine, neither for openings nor for closings. Tickets still open and record. When the window ends, tickets that opened during it and are still open are sent then. A banner on the host's page shows the window, with End now.

For the whole install

On the notifications card, Silence everything for a while asks for a number of hours (four by default) and a reason, up to seven days. This is for the install's administrator. It stops channel, policy and personal problem mail, and escalation policies wait until it ends. It does not stop a policy's second ring, workflows or the daily summary. Tickets still open and record, and nothing held back is re-sent afterwards. End it now lifts it.

Muting by name

For something that should never wake anybody, such as a staging box, Mute it by name is permanent rather than a window. See Notification channels.

Beyond notifications

A policy sends a message. To do something more when a ticket opens, such as wait, branch, enrich it, ask a model or call your own API, draw a workflow.