Live incidents
- CriticalOpenTelemetryraised 16:22Leak sensor wet · Water Leak — B2 Pump
Wet state held for 6 minutes; valve auto-closed by automation rule.
owner: Unassigned - HighAcknowledgedGatewayraised 15:48Gateway CPU sustained · EDGE-BSMT
CPU averaged 88% for 15 minutes; ingest latency p95 at 1.4 s.
owner: Marcus Dean - HighEscalatedUtilityraised 14:10Grid load spike · Main incomer
Load peaked at 1.12 MW during tariff peak window.
owner: Facility Head - HighAcknowledgedUtilityraised 11:05Water consumption deviation · Domestic water meter
Consumption 31% above learned baseline for this hour-of-week.
owner: Sam Reyes - CriticalResolvedGatewayraised 09:32Gateway offline · EDGE-GATE
Heartbeat missed x3; failed over to LTE backhaul.
owner: Ryan Foster - MediumOpenTelemetryraised 08:14Sensor stale data · Turnstile — Lobby A
No telemetry for 41 minutes; device shows offline.
owner: Unassigned
Incident timeline · INC-4412
- Rule
- Leak sensor wet
- Source
- Water Leak — B2 Pump
- Severity
- Critical
- Owner
- Unassigned
- Aug 01 · 16:22TriggerRule AR-101 firedLeak sensor B2-PUMP reported WET.
- Aug 01 · 16:22AutomationValve closedBasement 2 domestic line isolated automatically.
- Aug 01 · 16:23NotifyOps pagedSMS to on-call plumber and security desk.
Alert rules
| Rule | Domain | Condition | Severity | Notify | 24h | Actions |
|---|---|---|---|---|---|---|
Leak sensor wet AR-101 · LoRa leak fleet · 268 nodes | Telemetry | state = WET · instant | Critical | SMS · Ops · Siren | 2 | |
Sensor stale data AR-102 · All critical sensors | Telemetry | no message > 10 min · 10 m | High | Ops · Email | 4 | |
Elevator fault code AR-103 · 6 controllers | Telemetry | fault code ≠ 0 · instant | Critical | Vendor page · Ops | 1 | |
Gas threshold AR-104 · Kitchen + plant rooms | Telemetry | ppm > 250 · 30 s | Critical | Siren · Security | 0 | |
Gateway offline AR-201 · 4 edge gateways | Gateway | heartbeat missed x3 · 3 m | Critical | On-call · Slack | 1 | |
Gateway CPU sustained AR-202 · All gateways | Gateway | cpu > 85% · 15 m avg | High | Ops · Email | 3 | |
Store-and-forward backlog AR-203 · Edge buffers | Gateway | queue > 25k msgs · 5 m | Medium | Ops | 0 | |
Certificate expiry AR-204 · mTLS client certs | Gateway | expires < 30 d · daily | Low | 0 | ||
Grid load spike AR-301 · Main incomer | Utility | load > 1.05 MW · 10 m avg | High | Ops · Facility head | 2 | |
Water consumption deviation AR-302 · Domestic water | Utility | > +25% vs baseline · 1 h | High | Ops · Email | 1 | |
Solar underperformance AR-303 · Rooftop PV 480 kWp | Utility | < 70% of expected yield · 3 h | Medium | Vendor · Email | 0 | |
Power factor drop AR-304 · Main incomer | Utility | PF < 0.92 · 15 m | Medium | Ops | 0 |
Utility meter anomaly detection
- AnomalyDomestic water — Tower Ahour-of-week baseline · 8 weeks48.2 kL/hbaseline 36.8 · +31.0% · z 3.1
Sustained overnight flow suggests a riser leak or stuck float valve.
- AnomalyGrid load — Main incomerhour-of-week baseline · 12 weeks1,120 kWbaseline 940 · +19.1% · z 2.4
Chiller staging overlapped with EV charging peak.
- WatchSolar yield — Rooftop PVirradiance-adjusted baseline118 kWhbaseline 164 · -28.0% · z -2.0
String 4 output 28% low — soiling or inverter derate likely.
- NormalCommon area energyhour-of-week baseline · 8 weeks402 kWhbaseline 412 · -2.4% · z -0.3
Within learned band; lighting schedule tracking as expected.
- WatchSTP reuse linedaily baseline · 6 weeks22.4 kLbaseline 30.1 · -25.6% · z -1.9
Reuse dropped after pump 2 tripped; irrigation on fresh water.
Detection model
- Detector
- NX-Sense · seasonal-hybrid ESD + EWMA
- Baseline
- Hour-of-week, 8–12 week trailing
- Sensitivity
- z ≥ 2.0 watch · z ≥ 2.8 alert
- Retrain cadence
- Nightly 02:00, drift-checked
- Warm-up
- 14 days per new meter
- False positive rate
- 3.4% (30d rolling)
Each meter learns an hour-of-week baseline. Deviations past the sensitivity band raise a utility-domain incident with the deviation, z-score and probable cause attached for acknowledgement or escalation.