Detection Engineers and SOC analysts constantly write and deploy new detection rules. The problem is knowing whether those rules actually catch real attacker behavior or if they only catch theoretical, textbook patterns. Most portfolio projects test detection rules against simulated, self-written data.

This project doesn't. It tests rules against genuine, historically captured malware telemetry to prove they work in the real world.

Setting up a live VM and generating attacker behavior yourself is incredibly valuable, but it is time and equipment intensive. Using EVTX-ATTACK-SAMPLES (a public, highly respected security research dataset of real captured Windows Event Logs from actually-executed attacks) gives the exact same core value.

It ensures rules are tested against genuine technique execution, not invented or "clean" data, in a way that is fully reproducible and scriptable. This trades away "I personally generated the attack" for "I can prove and reproduce this against a citable, standard dataset that anyone can verify."

flowchart TD
    subgraph Data Acquisition
    A[(EVTX-ATTACK-SAMPLES<br>GitHub Dataset)] -->|fetch_samples.py| B[data/raw/<br>Binary .evtx Logs]
    end
    
    subgraph Parsing & Detection
    B -->|parse_evtx.py| C[data/parsed/<br>JSONL format]
    C -->|match_rules.py| D{Sigma Rules Evaluator}
    end
    
    subgraph Output
    D -->|Export| E(CLI Results Table)
    D -->|Export| F(public/data.json)
    end
    
    subgraph Vercel Deployment
    F --> G[Static Web Frontend]
    G <-->|Live JSON testing| H((Vercel Serverless API))
    end

• EVTX-ATTACK-SAMPLES (Source): The public GitHub repository of raw .evtx files.
• fetch_samples.py: Pulls down the massive binary log files via the GitHub API.
• data/raw/: Local storage for the unparsed binary Windows Event Logs.
• parse_evtx.py: Converts the unreadable .evtx binaries into structured JSONL.
• data/parsed/: Local storage for the readable, queryable JSON logs.
• sigma-rules/: The directory containing the actual detection logic written in Sigma YAML.
• match_rules.py: Our custom evaluation engine that tests the parsed logs against the Sigma rules.
• results table: The CLI output displaying exactly which rules caught which attacks.
• data.json export: A structured export of the findings for the UI.
• Frontend + Serverless Backend: A responsive UI that displays the findings, alongside a Vercel serverless backend that allows users to run the Python detection logic live.

Here is the final execution output of the detection rules run against the historical dataset:

13 rules were written across 10 unique techniques — 3 early rules were superseded by log-source-specific versions after testing revealed a log-source mismatch (see Key Finding #1 for an example).

I initially wrote a rule assuming attackers would spawn command-line utilities (like reg.exe) to modify the registry for persistence. When tested against the real Tendyron malware sample, the rule completely failed.

The malware bypassed command-line tooling entirely, directly utilizing Windows APIs to write its persistence key. The fix was to pivot the detection source from Process Creation (Sysmon Event ID 1) to Registry Events (Sysmon Event ID 12/13/14), which successfully caught the behavior regardless of execution method. Read the full coverage gap analysis here.

An initial rule targeting Signed Binary Proxy Execution (rundll32.exe) resulted in 85 matches across the dataset. Upon investigating the raw telemetry, I found that 74 of these matches were benign OS telemetry events—like module loads (Event ID 7)—where rundll32.exe simply happened to be involved, rather than actively executed.

By diagnosing this inflation and tightening the rule to explicitly require EventID: '1', the matches dropped to 11 precise, actionable alerts. This successfully caught a massive false-positive problem before it ever reached a production SIEM.

Even when a rule fires, analysts must quickly determine if it's benign or malicious. Here are 3 scenarios I documented:

1. Suspicious PowerShell Script Block: IT administrators frequently run management scripts. The differentiator is the execution path: IT scripts run from secure repositories (C:\Program Files\IT_Scripts), whereas our real malware staged from C:\Users\Public\lsass_wer_ps.ps1.
2. LSASS Object Access: Legitimate EDR/Antivirus tools constantly inspect LSASS memory. The differentiator is the requesting process: EDR uses signed binaries (CSFalconService.exe), whereas the malicious sample used a generic scripting engine (cscript.exe) to request high-privilege access masks.
3. Direct Registry Run Key Modification: Legitimate software like Microsoft Teams constantly writes Run keys. The differentiator is the payload path: legit software points to C:\Program Files\, whereas our malware pointed to a world-writable staging directory (C:\Users\Public\tools\apt\tendyron.exe).

Read the full false-positive guide here.

The Suspicious Microsoft Office Child Process rule was validated only incidentally (I found it executing inside a broader process-injection sample), not through an independently sourced and isolated Office-macro sample. This is not a full validation of macro detection capabilities.

I built an interactive "Test a Detection" feature in the frontend. It runs real Sigma rule logic (via a Vercel serverless Python function) against either real historical samples from the dataset or any JSON log you paste into it. It returns genuine match/no-match results on-demand against your static input, not live network traffic.

Try the Live Demo Here

# 1. Install requirements
pip install -r requirements.txt

# 2. Fetch the raw log files from GitHub
python scripts/fetch_samples.py

# 3. Parse .evtx binaries into queryable JSON
python scripts/parse_evtx.py

# 4. Run the evaluation engine
python scripts/match_rules.py

# 5. Exports rules, coverage-gap findings, and false-positive data into data.json for the frontend to consume.
python scripts/export_for_frontend.py

(Note: The web frontend runs automatically via Vercel utilizing api/index.py and public/)

All underlying event log data is sourced from the excellent EVTX-ATTACK-SAMPLES dataset authored by Samir Bousseaden.