Lab 4: Measuring detectors — precision, recall and alert fatigue

AI for Cyber Defenders · lesson 4 of 8 · 120 min

A vendor says their detector is “99% accurate”. Sounds great, until you learn that a detector that never alerts at all is also 99% accurate. Today you learn the numbers that actually matter in a SOC.

The confusion matrix

Every prediction falls into one of four boxes:

	Detector says ATTACK	Detector says SAFE
Really an attack	True positive (TP)	False negative (FN): missed!
Really safe	False positive (FP): wasted time	True negative (TN)

Precision = TP / (TP + FP). Of the alerts we raised, how many were real? Low precision means analysts chase ghosts.

Recall = TP / (TP + FN). Of the real attacks, how many did we catch? Low recall means attackers get through.

F1 combines both into one number. It is high only when precision and recall are both decent.

Why accuracy lies

Attacks are rare. If 1 in 100 events is malicious, a detector that always says “safe” is right 99% of the time and useless. This is the base-rate problem: when the thing you look for is rare, accuracy hides failure. Security teams talk about precision and recall instead.

Thresholds and alert fatigue

Most detectors output a score, and you choose a threshold: alert if score ≥ 0.6, say. Lower thresholds raise recall but flood analysts with false positives. Higher thresholds give clean alerts but miss attacks.

Alert fatigue is what happens when analysts get too many alerts: they rush, skim, and eventually ignore real ones. A threshold has to fit the team’s time. If each alert takes 6 minutes and the team has 16 analyst-hours a day, that is a hard limit of 160 alerts.

The lab code

We simulate a day of 5,000 events with about 1% attacks and a detector whose scores overlap, like real life. Open a new notebook at colab.research.google.com, paste the code into one cell, and press Shift+Enter. Running locally instead? Save it as a .py file and run python lab4.py.

# Lab 4: Measuring detectors - precision, recall, false positives, alert fatigue
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

rng = np.random.default_rng(seed=4)

# One simulated day at the credit union: 5,000 events, 1% are real attacks
N = 5000
is_attack = rng.random(N) < 0.01
# A detector gives each event a score from 0 to 1. Attacks tend to score higher,
# but the two groups overlap - just like real life.
score = np.where(is_attack, rng.beta(6, 2, N), rng.beta(2, 6, N))
events = pd.DataFrame({"is_attack": is_attack.astype(int), "score": score.round(3)})
print("Events:", N, "| real attacks:", events["is_attack"].sum())

def evaluate(threshold):
    pred = (events["score"] >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(events["is_attack"], pred).ravel()
    return {"threshold": threshold, "TP": tp, "FP": fp, "FN": fn, "TN": tn,
            "precision": round(precision_score(events["is_attack"], pred, zero_division=0), 3),
            "recall": round(recall_score(events["is_attack"], pred), 3),
            "f1": round(f1_score(events["is_attack"], pred), 3),
            "alerts": tp + fp}

# Compute one by hand so you can see the formulas
r = evaluate(0.5)
print("\nAt threshold 0.5:")
print(f"  TP={r['TP']} FP={r['FP']} FN={r['FN']} TN={r['TN']}")
print(f"  precision = TP/(TP+FP) = {r['TP']}/{r['TP'] + r['FP']} = {r['TP'] / (r['TP'] + r['FP']):.3f}")
print(f"  recall    = TP/(TP+FN) = {r['TP']}/{r['TP'] + r['FN']} = {r['TP'] / (r['TP'] + r['FN']):.3f}")

table = pd.DataFrame([evaluate(t) for t in [0.3, 0.5, 0.6, 0.7, 0.8, 0.9]])
# Alert fatigue: each alert takes an analyst about 6 minutes to triage
MINUTES_PER_ALERT = 6
table["analyst_hours"] = (table["alerts"] * MINUTES_PER_ALERT / 60).round(1)
print("\n== Threshold trade-off ==")
print(table.to_string(index=False))

# A 2-person SOC has about 16 analyst-hours per day. Which thresholds fit?
BUDGET_HOURS = 16
ok = table[table["analyst_hours"] <= BUDGET_HOURS]
best = ok.sort_values("recall", ascending=False).iloc[0]
print(f"\nBest threshold within {BUDGET_HOURS} analyst-hours: {best['threshold']} "
      f"(recall {best['recall']}, precision {best['precision']}, {int(best['alerts'])} alerts)")

# Base-rate trap: accuracy looks great even for a useless detector
always_safe = np.zeros(N, dtype=int)
acc = (always_safe == events["is_attack"]).mean()
print(f"\nA 'detector' that NEVER alerts is {acc:.1%} accurate - and catches 0 attacks.")

What you should see

About 55 real attacks. At threshold 0.5, recall is high (around 0.95) but precision is only about 0.15: roughly 357 alerts and 36 analyst-hours, more than double the team’s budget. At 0.6, you get about 139 alerts, 14 hours, and recall near 0.78. At 0.9 precision is perfect but recall falls to about 0.16. The “never alert” detector is 98.9% accurate and catches nothing.

Every threshold has a cost. Picking one is a business decision: how many misses can we accept, and how many alerts can people really handle?

Try it

1. Run the code and copy the threshold table into your notes.
2. Change MINUTES_PER_ALERT to 15 (harder alerts). Which threshold fits the budget now? What recall do you lose?
3. Make attacks rarer: change 0.01 to 0.002. What happens to precision at 0.5? Explain why.
4. Add 0.65 and 0.75 to the threshold list. Find the threshold with the best F1.
5. Write a short memo to your SOC lead recommending a threshold, with one number for misses and one for workload.

Check yourself

Precision is 0.15 at threshold 0.5. What does that mean for analysts?

About 85% of alerts are false alarms, so most triage time is wasted. That drives alert fatigue.

Why is accuracy misleading for rare attacks?

Because predicting “safe” for everything is right almost all the time. Accuracy hides the fact that every attack was missed.

Raising the threshold usually does what to precision and recall?

Precision goes up (fewer false alarms) and recall goes down (more missed attacks).

Key words

• Confusion matrix — a 2×2 table of TP, FP, FN and TN
• Precision — share of alerts that were real attacks
• Recall — share of real attacks that triggered an alert
• Base rate — how common the thing you are looking for is
• Alert fatigue — worn-out analysts missing real threats because of too many alerts

---

Previous lesson: Lab 3: Build a phishing email classifier
Back to AI for Cyber Defenders
Next lesson: Lab 5: Anomaly detection on login behavior