L45.0 Introduction to Alerting on Metrics with Detectors10%6.0 Create Efficient Dashboards and Alerts10%8.0 Detectors for Common Use Cases15%
Detectors and alerts
A detector evaluates signals against built-in alert conditions and creates alerts with a severity and notifications. This lesson covers building detectors, muting, efficient dashboards and alert messages, and how to make detectors reliable: trigger sensitivity, late data, populations, cycles and ephemeral hosts.
A detector is a smoke alarm for your numbers. You tell it what 'smoke' looks like and who to call. The skill is making it loud enough to catch real fires but quiet enough that people do not learn to ignore it, like switching off a doorbell that rings every time the wind blows.
INTERACTIVE
Tune a detector until it stops flappingTeach the smoke alarm to ignore toast
Thirty minutes of cpu.utilization at 10-second resolution with a real 3-minute spike, a single-datapoint blip and a noisy stretch near the line. Move the threshold, then change trigger sensitivity and watch the alert count.
A smoke alarm that screams every time you make toast gets unplugged. Immediately screams at the first wisp. Duration waits until there's been smoke the whole time, say for a full minute. Percent of duration waits until most of the last few minutes were smoky. The exam word for an alarm that keeps going on and off is flapping.
Trigger sensitivity
Window
01 · BLUEPRINT 5.1 · 5.2 · 5.3
Detectors, rules, alerts and how to create them
A detector watches one or more signals and creates alert and clear events when values cross the limits in its alert rules. A rule sets the condition, the severity (Critical, Major, Minor, Warning or Info) and the recipients. An alert exists when data matches a rule. When the condition stops being true, a second event (the clear) and a second set of notifications go out.
There are three ways to start a detector:
From a chart: bell icon, then New detector from chart. The signal is pre-selected and the detector stays linked to the chart.
From scratch:Create detector on the Detectors page, or Custom Detector in the create menu.
Clone:Alerts > Detectors, pick a similar detector, then Clone in the actions menu.
A rule is built in steps: alert signal, alert condition, alert settings (with a preview), alert message, recipients, then activate.
Think of a smoke alarm in your house. The detector is the alarm box. Each rule is one setting on it: what counts as smoke, how loud, and who gets called. An alert is the alarm actually going off. When the air is clean again the box also tells you, and that is the clear.
You can get a new alarm three ways. Press the bell on a chart you are already looking at (it comes pre-wired to that chart). Build one from a blank page. Or copy an alarm that already works and tweak it. Copying is usually fastest.
Smoke alarm box. The detector is the alarm box, each rule is one setting on it, and the alert is the alarm going off.
Remember for the exam
A detector monitors signals on a plot line and triggers alert and clear events when values cross the thresholds in its alert rules.doc
The five severity labels for a rule are Critical, Major, Minor, Warning and Info.doc
Selecting the bell icon on a chart and choosing New detector from chart pre-selects the signal and keeps the detector linked to the chart.doc
Using the double-bell icon to monitor multiple signals switches the alert condition to Custom Threshold.doc
To clone a detector, go to Alerts > Detectors, select a similar detector and choose Clone from the actions menu.doc
Deactivating an alert rule also clears any of its active alerts.doc
Email recipients (To, Cc, Bcc combined) are limited to 50.doc
02 · BLUEPRINT 5.1 · 5.3 · 8.1
Built-in alert conditions
Splunk Observability Cloud ships seven built-in conditions. Pick the one that matches the question you are asking.
Static Threshold for fixed limits. Options are Above, Below, Out of Range and Within Range.
Heartbeat Check when a signal stops reporting.
Resource Running Out when a trend is heading to empty or full.
Outlier Detection when one source differs from its peers.
Sudden Change when recent values differ from a longer recent past.
Historical Anomaly when values differ from the same period in earlier cycles.
Custom Threshold to compare two signals or combine conditions with AND and OR.
Static Threshold offers a Suggested Threshold based on the median absolute deviation over the last 14 days. Sudden Change, Outlier and Historical Anomaly use dynamic thresholds, so a clear event does not mean the value dropped below the number that triggered it.
Back to the smoke alarm. Each condition is a different kind of sensor.
Static Threshold is a fixed line on the wall: too hot is 90 degrees. Heartbeat is a pulse check: if the patient goes silent, shout. Running Out is a fuel gauge that also looks at how fast you are burning gas. Outlier asks which kid in the class is acting differently from the others. Sudden Change asks if things just jumped compared with the last hour. Historical Anomaly asks if this Monday looks unlike other Mondays. Custom Threshold lets you compare two gauges or say 'this AND that'.
Different sensors. Each condition is a different sensor: a fixed line, a pulse check, a fuel gauge, a classroom comparison, a recent-jump check, and a same-day-last-week check.
Condition
Use it when
Example
Static Threshold
You know the fixed good and bad values
Availability drops below 99.9, or latency leaves a 200 to 300 ms range
Heartbeat Check
A host or metric may go silent
A host has not reported for 15 minutes
Resource Running Out
A signal trends toward empty or full
Disk space is projected to hit zero within 24 hours
Outlier Detection
One source should look like its peers
One instance's requests are 2.5 standard deviations below the peer mean
Sudden Change
You have no fixed baseline, only recent behavior
CPU is 3 standard deviations above the previous hour's mean
Historical Anomaly
The metric follows daily or weekly cycles
This 2-hour login average is far above the same window last week (cycle length 1w)
Custom Threshold
You compare two signals or combine conditions
cache-misses is above cache-hits, or A is below 70 OR (B below 70 AND C below 70)
Remember for the exam
Static Threshold alerts when a signal goes above or below a fixed value, or moves inside or outside a range.doc
Within Range is inclusive and Out of Range is exclusive.doc
Suggested Threshold is calculated from the median absolute deviation over the last 14 days and must be reselected if you change the signal scope.doc
Resource Running Out has an Alert when nearing setting of Empty or Capacity, and Capacity has no default so you must set it.doc
Custom Threshold allows up to 10 conditions, AND is applied before OR, and a compound condition such as a AND (b OR c) needs the API.doc
Sudden Change requires the historical window to be longer than the current window.doc
Historical Anomaly accepts 1 to 8 previous cycles, and the current window should be shorter than the cycle length.doc
03 · BLUEPRINT 5.1 · 8.1 · 8.4
Alert preview, trigger sensitivity and flapping
The alert preview on the Alert Settings tab shows when the rule would have fired and cleared, up to the previous week. Use it to tune before you activate. Saving changed alert conditions on an existing detector sets currently active alerts to Stopped.
Flapping is an alert that fires and clears again and again. The fix is trigger sensitivity:
Immediately is the most sensitive and can flap on volatile signals.
Duration needs the condition true for the whole period. If any expected datapoint is missing or late, no alert fires.
Percent of duration (default 80% of 5 minutes) counts anomalous points against the expected points, so it tolerates some missing data.
Example: 75% of 3 minutes at 5 second resolution expects 36 points and needs 27 anomalous ones. Some conditions also let you set separate trigger and clear thresholds, with clear below trigger.
The preview is a rehearsal. It replays the last week and shows where your alarm would have gone off, so you can adjust before it wakes anyone up.
Flapping is a doorbell that rings, stops, rings, stops, because a delivery driver keeps tapping it. You fix it by saying 'only ring if someone is pressing for a full minute' (Duration) or 'ring if they pressed for at least most of the minute' (Percent of duration). The second one still works if a few taps were not heard. Duration is strict: miss one tap and you get silence.
Doorbell taps. Duration says ring only if someone presses for the whole minute. Percent of duration says ring if they pressed for most of it.
Remember for the exam
Alert preview covers up to the previous week, and for a longer detector range markers appear only for the most recent week.doc
Saving a detector with modified alert conditions sets the status of currently active alerts to Stopped.doc
With Duration sensitivity, if any expected datapoint is missing or delayed during the window, no alert fires even if all received points breach the threshold.doc
Percent of duration measures the share of expected points, not received points, that meet the condition.doc
The default trigger sensitivity for Static Threshold is Percent of duration at 80% of 5 minutes.doc
In a worked example, setting Duration to 1m dropped a detector from several thousand expected alerts a day to zero, and 80% percent of duration then gave 22.doc
04 · BLUEPRINT 5.4
Muting rules and notification states
A muting rule stops notifications during a known event such as maintenance. Alerts and events are still generated and still show in the product. Only notifications are held back. The one exception: clear notifications for alerts that were active before the muting period began still go out.
Three ways to create one: from a detector or alert (the ... menu on Alerts > Detectors or Active alerts), from a Group By view so the rule covers several detectors, or from scratch on Alerts > Muting rules. Filters you add use AND logic. Schedules can be a preset, a custom window, recurring, or indefinite.
After the window ends, notifications resume for alerts that are still active or fired within the 90 days before it ended. Only active and scheduled rules can be edited. Separately, per detector you can turn notifications on or off for resolution states: Stopped, Back to normal, Manually resolved and Auto-cleared.
A muting rule is a 'do not disturb' sign on your doorbell. Visitors still come and the camera still records them, so you can look later. You just do not hear the chime.
You pick which doors (detectors or properties) and for how long. Say the plumber is working Saturday 8 to noon: set a one-time sign. If the plumber comes every Saturday, make it repeat. One catch: if someone was already at the door before you hung the sign, the 'they left' message still reaches you.
Do not disturb sign. The doorbell camera still records every visitor. Muting only silences the chime.
Remember for the exam
Muting rules suppress notifications only. Alerts and events that match a rule are still generated.doc
Clear notifications for alerts that were active before the muting period started are not muted.doc
When a muting period ends, notifications resume for alerts still active or triggered within the 90 days before the period ended.doc
Multiple detector or property entries in a muting rule use AND logic.doc
A rule can take up to a minute to become active, and an organization can have at most 9,500 muting rules.doc
Search sf_eventType:alertMuting in the Events sidebar to find the events created when a muting rule starts and ends.doc
Only active and scheduled rules can be edited, and expired rules are read-only.doc
The resolution states you can turn notifications on or off for are Stopped, Back to normal, Manually resolved and Auto-cleared.doc
05 · BLUEPRINT 6.1 · 6.2 · 6.3 · 6.4
Efficient dashboards: notes, mirrors, events and data links
Instructions (6.1): add a Text Note chart from the Create menu or Chart Builder. It accepts plain text or Markdown. Use it to explain how to read or filter the dashboard.
Single-instance dashboards (6.2): most built-in Infrastructure Monitoring dashboards come as a single-instance version (one host or service) and a multi-instance version. To avoid a separate dashboard per environment, use dashboard variables and filters. A mirror (Enterprise Edition) shows the same dashboard in more than one group, each with its own name, description, filters and variables. Chart edits apply to every mirror.
Events (6.3): an Event Overlay draws event markers, such as detector alerts, on every chart. Overlays are not kept unless you save. A dashboard filter applies only if the dimension exists on the event.
Local data links (6.4): available on one dashboard only, created by anyone with write permission, not shown in alert messages. Global links need an admin and can appear in alert messages.
A dashboard is a wall of gauges. A text note is a sticky note saying 'if the red one is high, call Dana.' It can use simple formatting.
A single-instance dashboard is the gauges for one machine. A multi-instance one shows many at once. Instead of printing a new wall for every team, use a variable, like a dial that switches which team you look at. A mirror is the same wall shown in two rooms with different dial settings, but if you add a gauge it shows up in both.
An event overlay draws vertical lines such as 'alarm fired here' over every gauge. A local data link is a door on one wall only. A global one is a door on every wall, including the alarm email.
Wall of gauges. A dashboard is a wall of gauges. Notes are sticky notes, variables are dials, and mirrors are the same wall in another room.
Remember for the exam
A text note is a chart type that holds explanatory information and accepts plain text or Markdown.doc
Event overlays can include detector-triggered events and custom events, and are not kept unless you Save or Save As.doc
For event overlays, a dashboard filter applies only if the filter's dimension exists on the event.doc
Most built-in Infrastructure Monitoring dashboards include both a single-instance and a multi-instance version.doc
The best practices page recommends dashboard variables instead of building a separate dashboard for each environment, service or team.doc
Dashboard mirroring is an Enterprise Edition feature, and each mirror can have its own name, description, filters and dashboard variables.doc
A local data link is available on only one dashboard and can be added by anyone with write permission on it.doc
Only administrators can create global data links, and a global link is needed for the link to appear in alert messages.doc
When a local and a global link are equally specific, the local link wins on its own dashboard.doc
06 · BLUEPRINT 6.5
Customizing alert messages
In the Alert message step of a rule you can set the severity, edit the message text (Markdown and variables are supported) and add a runbook URL and a tip. Variables use double braces, for example {{detectorName}}. Use triple braces, such as {{{ruleName}}}, so the value is not escaped.
detectorName, ruleName, ruleSeverity, runbookUrl, tip
anomalous (true when triggered) and normal (true when cleared), often used in {{#if anomalous}}
dimensions for all dimensions, or dimensions.host for one
inputs.A.value for a signal value
Only signals used in the detect condition are available in the message. Each built-in condition adds its own variables, such as inputs.hours_left.value for Resource Running Out. Helpers include dateTimeFormat and doubleFormat.
The alert message is a form letter. The words that never change are typed in. The blanks, like name of the alarm, how bad, which machine, are filled in by the system when the letter is sent. The blanks are written in curly braces: {{detectorName}}.
You can write 'if it is going off, say this, otherwise say all clear'. You can also attach a runbook (a link to the how-to page) and a tip (the first thing to try). One limit: the form only knows about signals the alarm is actually checking, not ones you only glanced at.
Form letter. Fixed text plus blanks the system fills in when the alert goes out.
Remember for the exam
Alert message variables use double braces, and triple braces are used where the value must not be escaped.doc
The ruleSeverity variable renders as Critical, Major, Minor, Warning or Info.doc
runbookUrl is the page to open when the alert fires and tip is plain-text first course of action.doc
The anomalous variable is true when an alert triggers and the normal variable is true when it clears.doc
Only variables for signals used in the detect condition appear in the alert body, and dimension names containing dots need square brackets.doc
Each built-in condition accepts only its own extra variables, for example inputs.hours_left.value for Resource running out.doc
Alert messages support Markdown and variables, and the alert message step lets you add an optional runbook or tip.doc
07 · BLUEPRINT 6.6 · 8.1 · 8.2
Late data, extrapolation and troubleshooting detectors
Detectors evaluate on a schedule. If a datapoint arrives after its window closes, the detector may miss an alert it should have sent, or send one the chart later does not justify, because the chart shows the corrected timestamp.
Max Delay: how long the engine waits for data. Default is Auto, which uses the maximum observed delay. A fixed value helps with sudden momentary delays. A detector may evaluate up to 15 minutes late.
Min Delay: default 0, set through the API, for delayed or sparse new MTS.
Extrapolation policy: what to do with a missing point. By default it is treated as null. Options discussed in the docs include Zero and Last Value.
Percent of duration tolerates gaps without changing values.
Other causes of silent detectors: a changed reporting resolution, aperiodic data, and Custom Threshold plots that do not share the same dimensions (aggregate both to strip the extras). To catch a down host, use Heartbeat Check, and set Null extrapolation on its signals.
Imagine a referee who decides at the buzzer. If a player's score is called in late, the referee already ruled. Later the scoreboard shows the right score, so the ruling looks wrong. That is late data.
Max Delay is how long the referee waits for stragglers. Leave it on Auto and the referee learns how slow your usual stragglers are. Extrapolation policy is what to write down for a missing score: nothing, zero, or 'same as last time'. Filling in guesses can hide a real problem, so be careful. Percent of duration says 'good enough if most scores came in'.
Late referee. The referee rules at the buzzer. Scores called in late do not change the ruling, even though the scoreboard later shows the truth.
Remember for the exam
Max Delay defaults to Auto, where the engine uses the maximum observed delay, and Splunk's best practice is to let the engine decide when to run the computation.doc
Min Delay defaults to 0 and can only be set through the API.doc
A detector may evaluate up to 15 minutes late if data points are regularly delayed.doc
By default late or missing points are treated as null and excluded from calculations.doc
In the troubleshooting example, a 40 timestamped 11:09 arrived at 11:15, so an alert fired at 11:13 although the chart later showed values under 50.doc
When two plots are compared in a Custom Threshold they must share the same dimensions, otherwise aggregate both to remove the mismatched ones.doc
Setting the extrapolation policy to Last Value fills late points with the last received value, but many missing points can then cause unwanted alerts.doc
For Heartbeat Check, set the extrapolation policy to Null on every signal feeding the heartbeat metric.doc
08 · BLUEPRINT 8.3 · 8.5 · 8.6 · 8.7
Populations, cycles, many sources and ephemeral hosts
Populations (8.3): one detector can filter by a shared dimension such as service:kafka and alert on each member separately. New members with that dimension are included automatically. Keep one threshold and notification policy per population, or use const() in SignalFlow for different thresholds. For sub-groups, apply an analytic like P95 and group by a dimension or property (grouping does not work with tags). Outlier Detection flags members that differ from the group.
Cyclic patterns (8.5): use Historical Anomaly with a cycle length such as 1w, not a static line.
Many sources (8.6): create a detector for a signal, not for a microservice, and avoid wildcards like jvm.*. For small peer groups use median plus MAD.
Ephemeral infrastructure (8.7): turn on Auto-Clear Alerts so alerts close when a series stops reporting. They show as Auto-Cleared. Existing detectors must be edited to enable it.
A population is a herd. One shepherd (detector) watches the whole herd and calls out which sheep is in trouble. A new lamb joins and is watched with no extra work. Everyone in the herd follows the same rule, so keep the herd uniform.
A cyclic metric is like shop traffic: busy Saturdays are normal. Compare Saturday with last Saturday, not with Tuesday. With many sources, write rules per kind of measurement, not per app.
Ephemeral servers come and go like pop-up stalls at a market. When a stall packs up, its alarm should close itself. That is auto-clear.
Herd and pop-up stalls. One shepherd watches the whole herd, and alarms for stalls that packed up close themselves.
Remember for the exam
A single detector can monitor a population such as all hosts with service:kafka, and new members with that dimension are included without detector changes.doc
Grouping by a dimension or property works for sub-group analytics, but grouping does not work with tags.doc
Use the const function in SignalFlow to apply different thresholds to different dimension values instead of creating several detectors.doc
Median plus median absolute deviation is recommended for Outlier Detection when the population is under 15.doc
Historical Anomaly suits cyclical or seasonal metrics, for example a weekly pattern with Cycle length set to 1w.doc
The best practices page says to create a detector for a signal rather than for a microservice and to avoid wildcards that mix metric types.doc
Auto-clear resolves active alerts after a set period when an MTS stops reporting as expected, and the alert shows as Auto-Cleared.doc
Auto-clear settings do not apply to detectors created before you enable them, so you must edit an existing detector.doc
Elastic infrastructure such as EC2 autoscaling can vary by time of day, which makes static thresholds noisy.doc
HANDS ON
Try it in your Observability Cloud org
Uses send.sh detector-demo: about 20 minutes of normal data, a 3-minute spike on web-01, recovery, a one-datapoint blip, then web-02 goes silent for 2 minutes. It repeats until you stop it.
Steps
Start `send.sh detector-demo`. Open your Observability Cloud org and go to Metric Finder, search lab.cpu.utilization, and open it in a chart. Add filter env:prod and set the time range to the last 15 minutes. You should see: Two lines, host web-01 and web-02, wandering between 30 and 70.
Click the bell icon on the chart and choose New detector from chart. Name it Lab CPU high. You should see: The detector opens with lab.cpu.utilization already selected as the alert signal, linked to the chart.
At Alert signal, leave the signal as is. At Alert condition choose Static Threshold. At Alert settings set Alert when to Above and Threshold to 90. Set Trigger sensitivity to Immediately for now. You should see: The preview chart shows when this rule would have fired in the last week.
Watch the preview as the web-01 spike and the one-round blip arrive. Then change Trigger sensitivity to Duration and enter 1m. You should see: With Immediately, the one-round blip is marked as an alert. With 1m duration, only the sustained spike is marked.
Try Percent of duration at 80% of 1m. Note what the preview says, then leave Duration 1m selected. You should see: Six expected points per minute, so at least 5 must be above 90. The sustained spike still fires, the blip does not.
At Alert message, set severity to Major. Edit the message to read: {{#if anomalous}}{{ruleSeverity}}: CPU {{doubleFormat inputs.A.value precision=1}} on {{dimensions.host}}{{else}}Back to normal on {{dimensions.host}}{{/if}}. Add a tip such as Check the load generator. Open the Message preview tab. You should see: The preview shows your sentence with the variables filled in.
At Alert recipients add your email. Name the rule and click Activate Alert Rule. You should see: The detector is saved and appears under Alerts > Detectors.
Wait for the next spike. Go to Alerts > Active alerts. You should see: After about 1 minute of the spike, one Major alert appears with source host web-01 and your custom message. web-02 never alerts. You get an email.
Wait for the recovery phase and open the email or the alert details. You should see: The alert clears with your 'Back to normal' text. The blip in phase 4 produces no alert.
On Alerts > Detectors find Lab CPU high, open the ... menu and choose Create Muting Rule. Set a description, a schedule for the next 30 minutes, and confirm the detector filter. Click Create. You should see: The rule is listed on Alerts > Muting rules as Active or Scheduled. It can take up to a minute to take effect.
Let the next spike run while the rule is active. Check Alerts > Active alerts and your email. You should see: The alert still appears in Active alerts with a muted label, but no notification email arrives. Searching sf_eventType:alertMuting in the Events sidebar shows the mute start event.
Cancel the muting rule from Alerts > Muting rules using Resume notifications. You should see: Notifications resume. Later alerts email you again.
Optional: on Alerts > Detectors clone Lab CPU high, name it Lab heartbeat, and change the condition to Heartbeat Check with Hasn't reported for 1m, grouped by host. Set the extrapolation policy of the signal to Null. You should see: When web-02 stops sending in step 6, an alert fires for web-02 only, and clears when it resumes.
Practice · lesson 4
Check yourself
Questions are worded like the real exam. Press 1 to 4 to answer and Enter to continue.
Questions are worded like the real exam. Read the explanation after each one, even when you get it right.