AI for Cyber Defenders · lesson 8 of 8 · 120 min
Monday morning: 200 alerts in the queue and two analysts. Which ten do you open first? Your capstone answers that question with everything you have built: rules, model scores, and a clear write-up a human can act on.
What triage means
Triage is sorting by urgency, the way an emergency room decides who is seen first. In a SOC, the goal is to put the most likely, most damaging incidents at the top of the queue so limited human time goes where it matters.
The pipeline
- Collect alerts from different sensors: logins (auth), email, web, and endpoint (computers).
- Rules add simple, explainable flags: brute-force failures (Lab 2) and high URL scores (Lab 6).
- Model: a logistic regression trained on last week’s alerts that analysts already closed as real or not. It turns all the scores into one probability.
- Risk score: combine model probability, rule hits and asset criticality (how bad it would be if that system were hit) into a 0–100 number.
- Rank and label: sort by risk and tag P1 (urgent), P2, P3.
- Write-up: a short, plain explanation for the top alert so the analyst knows why it ranked first.
The risk formula uses weights you can change: W_MODEL = 0.6, W_RULES = 0.25, W_ASSET = 0.15. A loan server holding member data has criticality 5; a lobby kiosk has 1. Weights are a judgment call, so write down why you chose them.
How to judge a triage queue
Precision and recall still matter, but triage asks a sharper question: how many real incidents are in the top K? If the team can only open 20 alerts before lunch, the queue is good when most incidents are in those 20. We compare against working alerts in the order they arrived.
The lab code
We generate 400 alerts with about 8% real incidents, train on half (last week), and rank the other half (this week). Open a new notebook at colab.research.google.com, paste the code into one cell, and press Shift+Enter. Running locally instead? Save it as a .py file and run python lab8.py.
# Lab 8: Capstone - a mini alert-triage pipeline (rules + model scores -> ranked queue)
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(seed=8)
# ---- 1. Generate a week of alerts from different sensors ----
N = 400
sources = rng.choice(["auth", "email", "web", "endpoint"], N, p=[0.35, 0.3, 0.2, 0.15])
assets = rng.choice(["teller-pc", "loan-server", "member-portal", "hr-laptop", "lobby-kiosk"],
N, p=[0.35, 0.15, 0.15, 0.2, 0.15])
criticality = pd.Series(assets).map({"teller-pc": 2, "loan-server": 5, "member-portal": 5,
"hr-laptop": 3, "lobby-kiosk": 1}).to_numpy()
is_incident = rng.random(N) < 0.08 # ground truth: about 8% are real incidents
def noisy(base_true, base_false, spread):
return np.clip(np.where(is_incident, base_true, base_false) + rng.normal(0, spread, N), 0, 1)
alerts = pd.DataFrame({
"alert_id": [f"A{1000 + i}" for i in range(N)],
"source": sources,
"asset": assets,
"criticality": criticality,
"failed_logins": np.where(is_incident, rng.poisson(9, N), rng.poisson(4, N)),
"phish_score": noisy(0.6, 0.35, 0.25).round(2),
"anomaly_score": noisy(0.6, 0.35, 0.25).round(2),
"url_score": noisy(0.6, 0.3, 0.3).round(2),
"off_hours": (rng.random(N) < np.where(is_incident, 0.6, 0.15)).astype(int),
"is_incident": is_incident.astype(int),
})
# ---- 2. Rules (from Lab 2): simple, explainable flags ----
alerts["rule_bruteforce"] = (alerts["failed_logins"] >= 10).astype(int)
alerts["rule_bad_url"] = (alerts["url_score"] >= 0.8).astype(int)
alerts["rule_hits"] = alerts["rule_bruteforce"] + alerts["rule_bad_url"]
# ---- 3. A model trained on LAST week's analyst decisions ----
feat = ["failed_logins", "phish_score", "anomaly_score", "url_score", "off_hours", "rule_hits"]
history, this_week = train_test_split(alerts, test_size=0.5, random_state=0,
stratify=alerts["is_incident"])
model = LogisticRegression(max_iter=1000).fit(history[feat], history["is_incident"])
queue = this_week.copy()
queue["model_prob"] = model.predict_proba(queue[feat])[:, 1].round(3)
# ---- 4. Combine into one risk score (0-100) ----
W_MODEL, W_RULES, W_ASSET = 0.6, 0.25, 0.15
queue["risk"] = (100 * (W_MODEL * queue["model_prob"]
+ W_RULES * queue["rule_hits"] / 2
+ W_ASSET * queue["criticality"] / 5)).round(1)
queue = queue.sort_values("risk", ascending=False).reset_index(drop=True)
queue["priority"] = pd.cut(queue["risk"], [-1, 30, 60, 101], labels=["P3", "P2", "P1"])
print("== Top 10 of the triage queue ==")
show = ["alert_id", "source", "asset", "failed_logins", "url_score", "model_prob",
"rule_hits", "risk", "priority", "is_incident"]
print(queue[show].head(10).to_string(index=False))
print("\nPriority counts:\n", queue["priority"].value_counts().sort_index().to_string())
# ---- 5. Did ranking help? Compare to working alerts in arrival order ----
total = queue["is_incident"].sum()
arrival = this_week.reset_index(drop=True)
for k in [10, 20, 40]:
ranked_hits = queue["is_incident"].head(k).sum()
random_hits = arrival["is_incident"].head(k).sum()
print(f"Top {k:>2}: ranked queue finds {ranked_hits}/{total} incidents | arrival order finds {random_hits}/{total}")
# ---- 6. Analyst write-up for the #1 alert ----
top = queue.iloc[0]
reasons = []
if top["rule_bruteforce"]: reasons.append(f"{top['failed_logins']} failed logins (brute-force rule)")
if top["rule_bad_url"]: reasons.append(f"URL score {top['url_score']} (bad-URL rule)")
if top["off_hours"]: reasons.append("activity outside business hours")
reasons.append(f"model probability {top['model_prob']}")
reasons.append(f"asset {top['asset']} has criticality {top['criticality']}/5")
print(f"""
== Analyst write-up ==
Alert: {top['alert_id']} ({top['source']} sensor) on {top['asset']}
Risk: {top['risk']} -> {top['priority']}
Why it ranked first: {'; '.join(reasons)}.
Recommended next step: confirm with the account owner, review sign-in history,
and reset credentials if the activity is not recognized. A human analyst decides.""")What you should see
The top 10 is almost all real incidents (9 of 10 in our run), many of them on the high-criticality loan server. There are 19 incidents this week. The ranked queue finds about 15 in the top 20 and 18 in the top 40. Arrival order finds only 2 in the first 40. The write-up explains the #1 alert in plain words and ends with a recommended next step that a human approves.
A ranked queue does not replace analysts. It gives them their morning back, so they spend it on the alerts that matter.
Try it
- Run the pipeline and record how many incidents land in the top 10, 20 and 40.
- Change the weights (try
W_MODEL = 0.3,W_RULES = 0.5). Does the top-20 count get better or worse? - Find the false positive in your top 10. Which feature pushed it up? Suggest one fix.
- Add a rule of your own (for example, off-hours activity on a criticality-5 asset) and include it in
rule_hits. - Write your own analyst report (150–250 words) on the #1 alert: what happened, evidence, risk, recommended action, and one thing you are unsure about. This is your capstone deliverable.
Check yourself
Why include asset criticality in the risk score?
The same attack is more serious on a system holding member data than on a lobby kiosk, so it should be handled first.
What does “incidents in the top 20” measure?
How well the queue puts real threats where a busy team will actually look.
Why does the write-up end with “a human analyst decides”?
Models and rules can be wrong. A person checks the evidence and owns the decision to act.
Key words
- Triage — sorting alerts by urgency so the most important are handled first
- Risk score — a combined number that ranks how urgent an alert is
- Asset criticality — how much damage there would be if a system were compromised
- Pipeline — a series of steps where each step’s output feeds the next
- Top-K — the first K items in a ranked list
Previous lesson: Lab 7: LLMs in the SOC — using AI assistants safely
Back to AI for Cyber Defenders
You finished the course! Find your next course