Alerting & Runbooks

Status: Reviewed (deep-dive v1)
Stack: PowerShell (optional email/webhook hooks) → SQL alert warehouse/procs → Blazor Server UI (ack/assign)
Depends on: Findings from Inventory & configuration baseline through HA/DR awareness (inventory stale, backup Red, job fail, security, capacity, perf, patch overdue, HA unhealthy)
Goal: One severity model, one ack workflow, and a short runbook link per alert type — so noise doesn’t bury real fires.

Why this closes the outline

Topics 01–09 create signals. Without alerting hygiene you get either silence or pager fatigue. This note is the glue: normalize, route, ack, and point to the right deep-dive.

What “good” looks like

• Shared severity: Critical / High / Medium / Low (or Red/Yellow mapping)
• Every alert type maps to a SharePoint runbook doc (topic deep-dive)
• Blazor: open alerts, ack, assign, snooze/exception with expiry
• Dedup / correlation so one root cause ≠ 50 tickets
• Audit: who acked what and when
• No auto-remediation from UI in v1 (observe + human action)

Severity model (starting — confirm with stakeholders)

Map each collector flag → severity in dbo.AlertRule.

Alert catalog (wire to existing topics)

Add types sparingly; prefer enriching existing ones.

Checklist — detect & store

Procs:

• [ ] usp_Alert_Evaluate — run after collects; upsert open alerts; auto-close cleared
• [ ] usp_Alert_Ack — Blazor ack
• [ ] usp_Alert_GetOpenBoard
• [ ] usp_Alert_GetDetail — timeline + runbook link

Dedup: same type + instance + object fingerprint within window updates LastSeenAt, doesn’t spam new rows.

Checklist — notify (keep lean)

• [ ] Blazor badge counts (always on)
• [ ] Optional: email via Database Mail / PS Send-MailMessage / webhook — ops lead chooses channel
• [ ] Critical: notify on open; High: notify on open or digest Q1h; Medium/Low: daily digest
• [ ] Never put secrets in notification body

PS role: optional dispatcher reading usp_Alert_GetUnnotified.

Checklist — present (Blazor)

• [ ] Open alerts board (severity, age, instance, type, ack state)
• [ ] Filters: Critical only, unacked, mine
• [ ] Detail: description, runbook button (deep link to SQL Knowledge doc), timeline
• [ ] Actions: Ack, Assign, Snooze until, Request exception — no kill/failover/patch
• [ ] Home: count by severity

Operator workflow (short)

1. Open board → Critical unacked first
2. Open detail → read runbook link (correct SQL-0N)
3. Ack with short note (“investigating backup share”)
4. Fix using that topic’s checklist
5. Confirm signal cleared on next evaluate (or manually close if rule allows)
6. If recurring noise: tune threshold or add dated exception — don’t forever-mute Critical without stakeholder approval

Post-incident mini checklist

• [ ] Timeline captured in AlertEvent
• [ ] Root cause one-liner in Notes / ticket
• [ ] Runbook gap? → amend the SQL-0N doc (small PR-style edit)
• [ ] Rule tune if false positive

Common failure patterns

Permissions notes

• Evaluate job: write to alert tables only
• Blazor: exec ack/board procs; authenticated user identity for audit
• Notification creds: secured; not in source docs

Done definition

• [ ] AlertRule rows for core types covering Inventory & configuration baseline through HA/DR awareness
• [ ] Open board + ack path working
• [ ] At least one Critical path tested end-to-end (open → ack → clear)
• [ ] List Notes link this doc

Outline complete (Inventory & configuration baseline through Alerting & runbooks)

Beyond this outline: wait for stakeholder direction (implementation spikes, estate-specific targets, major-upgrade playbook, etc.).

SQL Dude — Alerting & Runbooks v1