Lab 6: Suspicious URL scoring with a random forest

AI for Cyber Defenders · lesson 6 of 8 · 120 min

A text says your credit union account is locked and gives you a link. Before anyone clicks, a defender can score that link in milliseconds. Today you build that scorer from clues you design yourself.

What a URL tells you

A URL (web address) has parts: the scheme (https://), the host (www.riverbendcu.example), the path (/rates/savings), and sometimes parameters after ?. Phishing links tend to abuse these parts:

• Burying a trusted name in a long chain of subdomains: riverbendcu.verify-login.account4471.example.test. The real owner is the end of the host, not the start.
• Look-alike spelling: rlverbend instead of riverbend.
• A raw IP address instead of a name.
• The @ trick: browsers ignore everything before @ in the host part.
• Bait words like verify, login, secure, reset.

Hand-built features

Instead of feeding raw text to the model, we design features: length, number of digits, dots in the host, hyphens, whether there is an @, whether the host is an IP, whether it uses https, how many bait words, how many parameters. Hand-built features are easy to explain to an analyst: “flagged because it has 3 bait words and 4 dots in the host.”

Random forests

A decision tree asks yes/no questions in order: “more than 1 bait word? more than 3 dots?” A random forest trains many trees, each on a random slice of the data and features, then lets them vote. Voting smooths out the mistakes of any single tree. The forest also reports feature importance: which clues it leaned on most.

The lab code

We generate 300 legit URLs on reserved domains and 200 phishing-style URLs in five styles, including short, clean-looking ones that are hard to catch. Nothing here is a real site. Open a new notebook at colab.research.google.com, paste the code into one cell, and press Shift+Enter. Running locally instead? Save it as a .py file and run python lab6.py.

# Lab 6: Suspicious URL scoring with hand-built features + RandomForest
import re
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

rng = np.random.default_rng(seed=6)

# Every domain here is reserved for examples (.example, .test, example.com/org/net)
legit_hosts = ["www.riverbendcu.example", "news.example.org", "www.example.com",
               "library.example.net", "schools.riverbend.test", "shop.example.com"]
legit_paths = ["/", "/about", "/rates/savings", "/events/youth-day", "/contact",
               "/news/2026/spring-fair", "/careers", "/help/faq"]
bait_words = ["login", "verify", "secure", "account", "update", "confirm", "wallet", "reset"]

def legit_url():
    url = "https://" + rng.choice(legit_hosts) + rng.choice(legit_paths)
    if rng.random() < 0.25:   # some real sites use IDs and tracking params too
        url += f"?id={int(rng.integers(100, 99999))}&ref=newsletter"
    if rng.random() < 0.1:
        url = url.replace("/help/faq", "/account/login")
    return url

def phish_url():
    style = rng.integers(0, 5)
    w1, w2 = rng.choice(bait_words, 2, replace=False)
    num = int(rng.integers(10, 99999))
    if style == 0:   # brand name buried in a long subdomain chain
        host = f"riverbendcu.{w1}-{w2}.account{num}.example.test"
    elif style == 1:  # raw IP address instead of a name
        host = f"203.0.113.{int(rng.integers(1, 254))}"
    elif style == 2:  # look-alike spelling
        host = f"rlverbend-cu-{w1}.example"
    elif style == 3:  # @ trick: browser ignores everything before @
        host = f"www.riverbendcu.example@{w1}{num}.test"
    else:             # short, clean-looking page - the hardest kind to catch
        return f"https://giftcard-{w1}.example/claim"
    scheme = "http://" if rng.random() < 0.7 else "https://"
    return f"{scheme}{host}/{w2}?id={num}&session={rng.integers(1000, 9999)}"

urls = [(legit_url(), 0) for _ in range(300)] + [(phish_url(), 1) for _ in range(200)]
df = pd.DataFrame(urls, columns=["url", "is_bad"])

def features(url):
    host = re.sub(r"^https?://", "", url).split("/")[0]
    return {
        "length": len(url),
        "num_digits": sum(c.isdigit() for c in url),
        "num_dots_host": host.count("."),
        "num_hyphens": url.count("-"),
        "has_at": int("@" in url),
        "is_ip_host": int(bool(re.fullmatch(r"\d+\.\d+\.\d+\.\d+", host))),
        "uses_https": int(url.startswith("https://")),
        "bait_words": sum(w in url.lower() for w in bait_words),
        "num_params": url.count("&") + url.count("?"),
    }

X = pd.DataFrame([features(u) for u in df["url"]])
y = df["is_bad"]
print(pd.concat([df["url"], X], axis=1).sample(5, random_state=1).to_string(index=False))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
forest = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=0)
forest.fit(X_train, y_train)
print("\n== Test results ==")
print(classification_report(y_test, forest.predict(X_test), target_names=["legit", "bad"]))

print("== Feature importance ==")
imp = pd.Series(forest.feature_importances_, index=X.columns).sort_values(ascending=False)
print(imp.round(3).to_string())

new_urls = [
    "https://www.riverbendcu.example/rates/auto-loans",
    "http://riverbendcu.verify-login.account4471.example.test/reset?id=4471",
    "http://198.51.100.23/secure/confirm?user=1",
    "https://events.example.org/youth-basketball-camp",
]
scores = forest.predict_proba(pd.DataFrame([features(u) for u in new_urls]))[:, 1]
print("\n== Scoring new URLs (0 = safe, 1 = bad) ==")
for s, u in sorted(zip(scores, new_urls), reverse=True):
    print(f"{s:4.2f}  {u}")

What you should see

Test scores near 100%, and the top features are bait_words, num_digits and num_dots_host. The new URLs score as you would hope: the long subdomain chain and the raw-IP link around 0.7–0.8, the real-looking credit union and community pages near 0. Remember Lab 3’s warning: our generator is simpler than real attackers, so a perfect score here is a sign the data is too easy, not that the problem is solved.

Attackers read the same playbooks we do. If your model relies on one clue, like bait words, they will just stop using those words. Diverse features make a detector harder to dodge.

Try it

1. Run the code and write down the top three features.
2. Score 5 URLs you invent (reserved domains only). Try to write a phishing-style one that scores below 0.5. What made it slip through?
3. Add a feature: "brand_not_at_end" = 1 if riverbend appears in the host but the host does not end in riverbendcu.example. Does importance shift?
4. Remove bait_words and retrain. How much do the scores drop?
5. Write one sentence an analyst could paste into a ticket explaining why a URL scored high.

Check yourself

In riverbendcu.verify-login.account4471.example.test, who controls the site?

Whoever owns the end of the host (example.test here). The trusted-looking name at the start is just a subdomain the attacker chose.

Why does a random forest often beat a single decision tree?

Many trees trained on different random slices vote together, which cancels out individual mistakes.

Why are hand-built features useful in a SOC?

They are easy to explain, so analysts can understand and trust why something was flagged.

Key words

• URL — a web address made of scheme, host, path and parameters
• Host — the part of a URL that names the server; the end of it shows the owner
• Decision tree — a model that makes predictions through a series of yes/no questions
• Random forest — many decision trees that vote together
• Feature importance — how much a model relied on each feature

---

Previous lesson: Lab 5: Anomaly detection on login behavior
Back to AI for Cyber Defenders
Next lesson: Lab 7: LLMs in the SOC — using AI assistants safely