You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This demo is part of `telemetry-lab` and is intentionally framed as a portfolio-grade security engineering prototype.
4
-
5
-
It demonstrates constrained AI-assisted case drafting for SOC-style workflows, not autonomous detection or response.
6
-
7
-
It combines deterministic detections with a tightly constrained LLM stage:
8
-
9
-
- the rules decide which activity is interesting
10
-
- the grouping logic decides which hits belong in the same case
11
-
- the LLM is limited to structured summaries, likely causes, uncertainty notes, and suggested next steps
12
-
13
-
The LLM does **not** make final incident decisions, modify rules, call tools, or execute response actions. Human verification is always required.
14
-
1
+
# AI-Assisted Detection Demo
2
+
3
+
This demo is part of `telemetry-lab` and is intentionally framed as a portfolio-grade security engineering prototype.
4
+
5
+
It demonstrates constrained AI-assisted case drafting for SOC-style workflows, not autonomous detection or response.
6
+
7
+
It combines deterministic detections with a tightly constrained LLM stage:
8
+
9
+
- the rules decide which activity is interesting
10
+
- the grouping logic decides which hits belong in the same case
11
+
- the LLM is limited to structured summaries, likely causes, uncertainty notes, and suggested next steps
12
+
13
+
The LLM does **not** make final incident decisions, modify rules, call tools, or execute response actions. Human verification is always required.
14
+
15
15
## Purpose
16
16
17
17
The goal is to show a credible bridge between deterministic telemetry analytics and safe analyst assistance.
18
18
19
19
This is not an autonomous SOC. It is a constrained drafting pipeline that keeps rule logic, ATT&CK mapping, case grouping, and evidence handling deterministic.
20
20
21
21
For a no-run reviewer pack, see [docs/ai-assisted-detection-examples.md](../../docs/ai-assisted-detection-examples.md).
22
-
23
-
## Pipeline
24
-
25
-
1. ingest sample auth, web, and process events from JSONL
26
-
2. normalize them into a shared internal schema
27
-
3. apply deterministic detection rules
28
-
4. group rule hits into cases by shared entities and time proximity
29
-
5. attach ATT&CK mappings from rule metadata
30
-
6. build a case bundle with raw evidence, rule hits, severity, and evidence highlights
31
-
7. pass the case bundle to a constrained local demo LLM adapter with strict instruction and data separation
32
-
8. require JSON-only output against a local schema
33
-
9. validate the response and reject invalid output
34
-
10. emit analyst-facing artifacts and audit traces
35
-
36
-
## Guardrails
37
-
38
-
- telemetry content is marked as untrusted data
39
-
- system instructions are separated from the evidence payload
40
-
- the response must pass local JSON schema validation
41
-
- the response must pass a semantic validation layer after schema validation
42
-
-`human_verification` is required and must be `required`
43
-
- no external tool use is allowed in the LLM stage
44
-
- no automated response actions are allowed
45
-
- forbidden action-taking or final-verdict language is rejected and recorded
46
-
- summaries are rejected if the returned `case_id` does not exactly match the input case bundle
47
-
- a prompt-injection-like sample event is included and treated as telemetry, not instruction
48
-
- rejected summaries are fail-closed: they do not enter `case_summaries.json`
49
-
- accepted and rejected outcomes are both recorded in `audit_traces.jsonl`
50
-
51
-
## Quick start
52
-
53
-
From the repository root:
54
-
55
-
```bash
56
-
python -m pip install -e .
57
-
python -m telemetry_window_demo.cli run-ai-demo
58
-
```
59
-
60
-
Generated artifacts are written to `demos/ai-assisted-detection-demo/artifacts/`.
@@ -85,33 +86,34 @@ The bundled sample run should report:
85
86
-`3` audit records
86
87
87
88
## Artifact semantics
88
-
89
-
-`rule_hits.json`: deterministic rule hits with rule metadata, ATT&CK mapping, entities, and evidence highlights
90
-
-`case_bundles.json`: grouped cases with severity, rule hits, ATT&CK mappings, raw evidence, and untrusted-data marking
91
-
-`case_summaries.json`: only accepted JSON summaries that passed schema and semantic validation
89
+
90
+
-`rule_hits.json`: deterministic rule hits with rule metadata, ATT&CK mapping, entities, and evidence highlights
91
+
-`case_bundles.json`: grouped cases with severity, rule hits, ATT&CK mappings, raw evidence, and untrusted-data marking
92
+
-`case_summaries.json`: only accepted JSON summaries that passed schema and semantic validation
92
93
-`case_report.md`: analyst-facing report with run counts, accepted summaries, and explicit notes for rejected case summaries
93
94
-`case_report.md`: includes a top-level run integrity section that surfaces rule/config degradation
94
-
-`audit_traces.jsonl`: stable per-record audit log for accepted and rejected paths, using `schema_version = ai-assisted-detection-audit/v1` and including `ts`, `case_id`, `validation_status`, `rejection_reason`, `rule_ids`, `prompt_input_digest`, `evidence_digest`, and bounded response excerpts
95
-
96
-
## Rejection behavior
97
-
98
-
- non-JSON or malformed JSON responses are rejected and recorded
99
-
- missing required fields or invalid enum values are rejected and recorded
100
-
- schema-valid summaries with the wrong `case_id` are rejected and recorded
101
-
- action-taking language is rejected
102
-
- final-verdict or confirmed-compromise language is rejected
103
-
- malformed rule or ATT&CK metadata is rejected before detection logic uses it
104
-
105
-
Rejected outputs do not become analyst summaries. Analysts can still inspect deterministic evidence through `case_bundles.json`, `case_report.md`, and `audit_traces.jsonl`.
106
-
95
+
-`audit_traces.jsonl`: stable per-record audit log for accepted and rejected paths, using `schema_version = ai-assisted-detection-audit/v1` and including `ts`, `case_id`, `validation_status`, `rejection_reason`, `rule_ids`, `prompt_input_digest`, `evidence_digest`, and bounded response excerpts
96
+
-`run_manifest.json`: synthetic-local run manifest with tool version, input/config digests, and artifact schema versions
97
+
98
+
## Rejection behavior
99
+
100
+
- non-JSON or malformed JSON responses are rejected and recorded
101
+
- missing required fields or invalid enum values are rejected and recorded
102
+
- schema-valid summaries with the wrong `case_id` are rejected and recorded
103
+
- action-taking language is rejected
104
+
- final-verdict or confirmed-compromise language is rejected
105
+
- malformed rule or ATT&CK metadata is rejected before detection logic uses it
106
+
107
+
Rejected outputs do not become analyst summaries. Analysts can still inspect deterministic evidence through `case_bundles.json`, `case_report.md`, and `audit_traces.jsonl`.
108
+
107
109
## Reviewer walkthrough
108
110
109
111
### Accepted summary path
110
-
111
-
Use the default sample run artifacts in `artifacts/case_summaries.json`, `artifacts/case_report.md`, and `artifacts/audit_traces.jsonl`.
112
-
113
-
Verify that `CASE-001` appears in all three places, that the `case_id` matches exactly, that `human_verification` is `required`, and that the audit record shows `validation_status = accepted` with `schema_version = ai-assisted-detection-audit/v1`.
114
-
112
+
113
+
Use the default sample run artifacts in `artifacts/case_summaries.json`, `artifacts/case_report.md`, and `artifacts/audit_traces.jsonl`.
114
+
115
+
Verify that `CASE-001` appears in all three places, that the `case_id` matches exactly, that `human_verification` is `required`, and that the audit record shows `validation_status = accepted` with `schema_version = ai-assisted-detection-audit/v1`.
Then inspect the `case_report.md`, `case_summaries.json`, and `audit_traces.jsonl` files under `.pytest-artifacts-ai-demo-rejections/test_*/artifacts/`.
124
-
125
-
Verify that the rejected case is absent from `case_summaries.json`, appears in `case_report.md` as `Summary status: rejected`, and has an audit record with `validation_status = rejected` plus a concrete `rejection_reason` such as `missing_required_fields`, `semantic_validation_failed`, or `case_id_mismatch`.
126
-
126
+
127
+
Verify that the rejected case is absent from `case_summaries.json`, appears in `case_report.md` as `Summary status: rejected`, and has an audit record with `validation_status = rejected` plus a concrete `rejection_reason` such as `missing_required_fields`, `semantic_validation_failed`, or `case_id_mismatch`.
Then inspect the generated `case_report.md` and `audit_traces.jsonl` files under `.pytest-artifacts-ai-demo-degraded/test_*/artifacts/`.
136
-
137
-
Verify that `case_report.md` exposes `## Run Integrity`, `coverage_degraded: yes`, and the rejected rule id, and that `audit_traces.jsonl` contains a global rejection record with `case_id = null` and `rejection_reason = rule_metadata_validation_failed`.
138
-
139
-
## Limitations
140
-
141
-
- the LLM stage is a constrained local demo adapter, not a production model integration
142
-
- detections are intentionally small and rule-based
143
-
- grouping is simple and optimized for readability over recall
144
-
- sample telemetry is synthetic and limited in volume
145
-
- there is no ticketing, SOAR, sandboxing, or live data ingestion
146
-
- artifacts are for analyst review only and do not represent final incident disposition
147
-
- rejection logic is intentionally conservative and favors fail-closed behavior over model flexibility
138
+
139
+
Verify that `case_report.md` exposes `## Run Integrity`, `coverage_degraded: yes`, and the rejected rule id, and that `audit_traces.jsonl` contains a global rejection record with `case_id = null` and `rejection_reason = rule_metadata_validation_failed`.
140
+
141
+
## Limitations
142
+
143
+
- the LLM stage is a constrained local demo adapter, not a production model integration
144
+
- detections are intentionally small and rule-based
145
+
- grouping is simple and optimized for readability over recall
146
+
- sample telemetry is synthetic and limited in volume
147
+
- there is no ticketing, SOAR, sandboxing, or live data ingestion
148
+
- artifacts are for analyst review only and do not represent final incident disposition
149
+
- rejection logic is intentionally conservative and favors fail-closed behavior over model flexibility
0 commit comments