Splunk alert fatigue

A Splunk Enterprise Security screenshot circulating among security teams shows a live queue of about 40,000 alerts a day, with roughly 90% labeled noise. (x.com) The post is being passed around by security operations center analysts and detection engineers as a concrete example of alert fatigue: too many alarms, too little time to investigate each one. Splunk Enterprise Security creates a “notable event” when a correlation search detects a suspicious pattern, then surfaces those notables in Incident Review for triage. (x.com) (help.splunk.com) In Splunk’s design, a notable is the unit analysts review, assign, suppress, and investigate. The product documentation says correlation searches generate those notables, and administrators can change fields, statuses, and suppressions to manage the queue. (help.splunk.com 1) (help.splunk.com 2) The basic problem is simple: a detection rule fires on a pattern, but real networks produce huge amounts of normal activity that can look suspicious out of context. Splunk’s own documentation now frames risk-based alerting as a way to “reduce false positives” and “reduce alert volume” by combining lower-level signals before creating a higher-priority case. (help.splunk.com 1) (help.splunk.com 2) Splunk also publishes tuning guidance that tells customers to adjust risk thresholds “to avoid high alert volume,” create allow lists to cut noisy alerts, and tune detections by reviewing likely false positives and projected alert reduction. Those documents amount to an official acknowledgment that default or poorly tuned content can swamp analysts. (help.splunk.com 1) (help.splunk.com 2) (help.splunk.com 3) That matters because security teams increasingly map detections to the MITRE ATT&CK framework, which catalogs real-world adversary tactics and techniques across enterprise environments. More coverage can mean more rules, and more rules can mean more low-value hits if the data, thresholds, and exclusions are not tuned for a specific network. (attack.mitre.org) (attack.mitre.org) (www.cisa.gov) Splunk’s risk-based model tries to change that math by collecting multiple risk events into a single risk index and creating one risk notable only after a threshold is met. The company says that approach is meant to keep analysts from drilling through raw alert floods one event at a time. (help.splunk.com) (help.splunk.com) Vendors and defenders outside Splunk describe the same pattern. Cisco wrote in March 2026 about an “alert storm” at Cisco Live EMEA, where investigators used correlated context from Cisco XDR, Splunk, firewall telemetry, and packet data to separate genuine threats from environmental noise. (blogs.cisco.com) The viral 40,000-a-day example does not prove every Splunk deployment looks like that, and the post is still a social-media anecdote rather than a published benchmark. But Splunk’s own manuals now devote entire sections to suppressions, threshold changes, allow lists, and tuning steps built to keep the queue from turning into exactly that. (x.com) (help.splunk.com) (help.splunk.com)