Alert fatigue is real. Your monitoring system sends 100 emails per day. 95 of them are noise—a single interface flapping sends 50 emails in 2 minutes. The on-call tech stops reading alerts. The one critical alert about your core router being down gets missed in the noise.

This guide shows you how to configure alerting so that your team actually reads the alerts and responds to real incidents.

Alert Fatigue: The Problem

Example: Interface Flapping

A PoE switch port has a bad cable. Every 3 seconds, the interface bounces:

14:00:01 Interface eth0 DOWN → alert email
14:00:04 Interface eth0 UP → alert email (clear)
14:00:07 Interface eth0 DOWN → alert email
14:00:10 Interface eth0 UP → alert email (clear)
...repeat 50 times...
14:02:45 Interface eth0 DOWN (final)

Your on-call tech receives 100+ emails for a single bad cable. After 3 incidents like this per week, alerts are now spam. Real outages get ignored.

Example: False Positives (Transient Network Glitches)

A router reboots. For 30 seconds, interfaces are unreachable. Your monitoring tool sends:

14:10:00 Router DOWN
14:10:30 Router UP

That’s legitimate, but your tech checked the device at 14:10:15 and it was perfectly fine. The alert adds nothing.

Example: Too Many Alerts per Issue

Your core router fails. The monitoring tool sends:

14:15:00 Router A DOWN (email)
14:15:02 Router A CPU high (email) — cascade failure
14:15:04 Router A memory full (email) — same failure
14:15:06 Customer A lost connectivity (email) — child of Router A
14:15:08 Customer B lost connectivity (email) — child of Router A
14:15:10 Customer C lost connectivity (email) — child of Router A
...30+ emails in 5 seconds...

One failure creates 30 alerts. Your tech is overwhelmed.

Solution 1: Flap Detection (Threshold-Based)

Flap detection prevents rapid on/off toggling from generating multiple alerts.

Algorithm:

Track: Interface state changes over last N minutes
Trigger alert only if: ≥X state changes in N minutes, then stable for 5 minutes

Example config:
- N = 5 minutes
- X = 3 changes
- Stable threshold = 5 minutes

Scenario: Interface bounces 50 times in 2 minutes
- Monitor detects 3 changes within 5-min window
- Flags as "flapping"
- Suppresses individual alerts
- Once stable for 5 min, sends 1 alert: "Interface flapping, now stable"

LibreNMS flap detection (example):

alert:
  - name: Interface Flapping
    description: "Interface changed state multiple times"
    condition: "port_flap_count > 3"
    window: 5m
    delay: 5m  # Wait 5min after stability before alerting
    severity: "warning"

YAD flap detection (webhook config):

{
  "rule": "interface_flap",
  "trigger": {
    "device": "*",
    "interface": "*",
    "state_changes": { "count": 3, "period_seconds": 300 }
  },
  "action": {
    "after_stable": 300,
    "send": {
      "webhook": "https://slack.com/hooks/...",
      "message": "Interface {{device}}/{{interface}} flapped {{count}} times, now stable"
    }
  }
}

Result: Instead of 50 emails for a flapping port, you get 1 email after 5 minutes of stability. Actionable, not spam.

Solution 2: Dependency-Aware Alerting (Hierarchical Suppression)

When a parent device fails, suppress alerts on all children. Otherwise, one failure creates 100 alerts.

Example Topology

Internet ← Your Border Router (10.0.0.1)
           ├─ Regional Hub 1 (10.1.1.1)
           │   ├─ Access Switch A (10.1.1.10)
           │   │   ├─ Customer 1 (192.168.1.1)
           │   │   └─ Customer 2 (192.168.1.2)
           │   └─ Access Switch B (10.1.1.20)
           │       ├─ Customer 3 (192.168.1.3)
           │       └─ Customer 4 (192.168.1.4)
           └─ Regional Hub 2 (10.1.2.1)

Without Dependency Alerting

Border Router fails → generates 8 alerts (router itself + 2 hubs + 5 customers).

With Dependency Alerting

Border Router DOWN
→ Automatically suppress alerts for: Regional Hub 1, Hub 2, Switches A/B, Customers 1-4
→ You get 1 alert: "Border Router down (affects 7 downstream devices)"

LibreNMS setup:

-- Add parent relationship
mysql> INSERT INTO device_relationships (host_a, host_b, relationship) 
  VALUES (1, 2, 'parent');
-- Where host_a=Border Router, host_b=Regional Hub 1

-- Enable alert rule
CREATE RULE parent_down_suppress_children:
  IF parent_device.status = 'down' THEN
    SUPPRESS child_device.interface_* alerts
    SUPPRESS child_device.device_down alert

Zabbix setup:

Configuration → Event correlation → New Correlation
  Conditions: trigger.name matches "Border Router" AND trigger.value = PROBLEM
  Operations: Add custom escalation (suppress child triggers)

Solution 3: Alert Escalation (Time-Based Severity)

Different notification channels for different severity levels, with time-based escalation.

Multi-Channel Escalation

Time 0min:   Interface down (minor) → Log only, no alert
Time 3min:   Still down → Email NOC
Time 15min:  Still down → Slack @channel mention
Time 30min:  Still down → PagerDuty (calls on-call engineer)
Time 60min:  Still down → SMS to manager

Why this works:

Log-only (0 min): Transient network blips clear themselves. Don’t bother anyone.
Email (3 min): If it persists 3 min, NOC reads email when they see it
Slack (15 min): If not fixed in 15 min, escalate to async chat (someone might see it in next 5 minutes)
PagerDuty (30 min): If still broken at 30 min, wake up the on-call engineer with a phone call
SMS (60 min): If on-call didn’t respond in 30 min, escalate to manager

LibreNMS escalation rules:

alerts:
  - name: Customer Interface Down
    condition: "port.ifOperStatus = down"
    
    escalations:
      - delay: 0m
        action: log
        
      - delay: 3m
        action: email
        recipients: ["[email protected]"]
        
      - delay: 15m
        action: slack
        channel: "#noc"
        mentions: ["@noc-team"]
        
      - delay: 30m
        action: pagerduty
        integration_key: "xxx"

Solution 4: Maintenance Windows

Suppress all alerts during scheduled maintenance or expected downtime.

Use case: You’re upgrading a switch every Sunday 22:00–23:00. Don’t alert on interface flaps during the upgrade.

LibreNMS maintenance window:

Monitoring → Maintenance → Add Maintenance
  Device: Access Switch A
  Start: 2026-04-20 22:00
  End: 2026-04-20 23:00
  Recurring: Weekly (every Sunday)
  
Result: All alerts from Switch A suppressed during window

Zabbix equivalent:

Configuration → Maintenance → Create maintenance period
  Name: "Weekly Switch Upgrade"
  Active: Every Sunday 22:00–23:00
  Hosts: "Access_Switch_A"
  Suppress: All triggers

Solution 5: Alert Severity Thresholds

Not all down = equal severity. Distinguish minor issues from critical ones.

Severity	Condition	Channel	Response Time
CRITICAL	Core router down, Core switch down, OLT down	PagerDuty + SMS	5 minutes
HIGH	Regional hub down, Access switch down, 50+ customers affected	Email + Slack	15 minutes
MEDIUM	Single customer down, single interface down	Slack only	30 minutes
LOW	High CPU/memory (not full), low free disk space	Email daily digest	No immediate response
INFO	Device rebooted, backup completed	Log only	No response

Configuration example (YAD):

{
  "rules": [
    {
      "name": "Core Router Down",
      "condition": "device.location = 'core' AND device.status = 'down'",
      "severity": "CRITICAL",
      "actions": [
        { "type": "pagerduty" },
        { "type": "sms", "to": "+420724000000" },
        { "type": "slack", "channel": "#critical-alerts", "mentions": "@critical" }
      ]
    },
    {
      "name": "Customer CPE Down",
      "condition": "device.type = 'cpe' AND device.status = 'down'",
      "severity": "MEDIUM",
      "actions": [
        { "type": "slack", "channel": "#alerts" }
      ]
    }
  ]
}

Alert Message Quality

Your alert must answer 4 questions immediately:

What is broken? “Interface eth0 on router-hub01”
Why does it matter? “Affects 10 customers”
What should I do? “Check cable connection or reboot device”
How urgent? “CRITICAL (response required in 5 min)”

Bad alert:

"Alert triggered on 192.168.1.1"

(What device? What broke? What do I do?)

Good alert:

"[CRITICAL] Router HUB-01 (core) is DOWN
  - Affects: 12 customer sites
  - Action: SSH to router, check status with 'interface print', or restart
  - Escalation: If not resolved in 5 min, call manager

Testing Alert Rules

Before deploying, test:

# Simulate device down
down_device=$(get_device_by_name "test-router")
simulate_down $down_device

# Check alerts generated
check_alerts_in_last_5_minutes

# Verify:
# ✅ Flap detection works (5+ bounces = 1 alert, not 5)
# ✅ Dependencies work (child alerts suppressed)
# ✅ Escalation triggers (log → email → Slack progression)
# ✅ Maintenance window blocks alerts (if configured)

# Bring device back up
simulate_up $down_device

# Check clear alert sent
check_clear_alert_sent

Real ISP Example: 50-Customer Network

Network composition:

1 core router
3 regional hubs
8 access switches
50 customer CPEs

Alert rules:

Core router down → CRITICAL → PagerDuty call in 2 min
Regional hub down → HIGH → Email + Slack in 5 min
Access switch down → MEDIUM → Slack in 15 min
Customer CPE down → LOW → Log only, email digest once daily
Interface flapping → Suppress for 5 min, then alert if still flapping
Interface errors >100/sec → MEDIUM → Slack with suggestion “Check cable”

Result: On-call tech receives ~5 alerts per day (real issues), not 500. Response time improves from “eh, maybe ignore” to “respond in 5 min.”

Testing Your Alert System

Run a monthly alert drill:

Simulate core router failure
Verify PagerDuty call happens within 2 min
Verify child device alerts are suppressed
Verify Slack message includes “which devices affected”
Recover router, verify clear alerts sent

This catches configuration bugs before they cause SLA breaches.

Conclusion

Alert fatigue kills your on-call culture. Fix it with:

Flap detection — stops interface flaps from spamming
Dependency suppression — one failure, one alert
Multi-channel escalation — email → Slack → phone calls based on time
Maintenance windows — suppress expected downtime
Severity levels — distinguish critical from informational
Good message quality — answer “what, why, do what, urgent?”

Implement these, and your team will actually read and respond to alerts. Your MTTR (mean time to resolution) drops from hours to minutes.

Deploy YAD with advanced alerting → yetanotherdude.io
Flap detection, escalation policies, and webhook integrations built-in.