Skip to content

Commit be51931

Browse files
chenliuyunchenliuyun
andauthored
feat: notify actions + LLM-backed rule suggestion (tracks θ + η) (#40)
* feat(M1): add NotifyAction union type, schema oneOf, rule-notify audit kind * feat(M2): add notify action executor with file/webhook channels and template rendering * feat(rules): dispatch notify actions in engine + lintRules validation * feat(mcp): add rule_notifications tool for notify audit history * feat(doctor): add notify-connectivity check for webhook URLs in policy * feat(llm): add intent complexity scorer * feat(llm): add provider interface, OpenAI and Anthropic providers, factory * feat(rules): add LLM-backed rule suggestion with auto-routing - Add suggestRule llm param (openai | anthropic | local | auto) - Auto mode scores intent complexity and routes to LLM when score >= 4 - LLM path calls provider.generateYaml, lints result, falls back to heuristic with warning on provider failure - Add buildRuleSuggestSystemPrompt embedding v0.2 schema snippet - Wire --llm flag into CLI rules suggest command - Add llm param to rules_suggest MCP tool - Add llm-suggest kind to audit log with backend/model/latency fields - Fix tests/rules/suggest.test.ts: add async/await (suggestRule is now async) - Fix tests/llm/suggest.test.ts: set mock implementations in beforeEach to survive vitest restoreMocks:true between tests * docs: update roadmap and README for tracks θ (notify) and η (LLM suggest) * fix: wire dry-run through notify, expose llm in CLI/MCP, fix rule_notifications total - engine.ts: pass globalDryRun to executeNotifyAction so notify actions respect both rule-level dry_run and the --dry-run flag - notify.ts: skip webhook/file side effects and write audit with dryRun:true when dry-run is active; add dryRun field to NotifyResult and NotifyContext - rules.ts: add --llm <backend> option to `rules suggest` and pass it through to suggestRule - mcp.ts: add llm field to rules_suggest inputSchema and handler; fix rule_notifications to report total as entries.length (pre-limit count) * fix: forward explicit overrides to LLM, guard YAML structure, reject non-http(s) URLs - suggest.ts: append trigger/event/schedule/days/webhookPath as explicit constraints in the LLM user message so models cannot silently override them - suggest.ts: validate parsed YAML is a non-null object with when/then fields before passing to lintRules, preventing internal crashes on malformed output - notify.ts: reject non-http/https protocols in sendWebhook with a clear error instead of silently treating them as plain HTTP * fix(rules): enforce dry_run, override constraints, and reject non-http URLs at lint time - LLM-generated rules are now always coerced to dry_run:true with a warning when the model returned dry_run:false or omitted the field. - Caller-provided --trigger now hard-fails when the LLM returns a different trigger source. Within the same trigger, --event/--schedule/--days/ --webhook-path mismatches override the LLM value and emit a warning. - Notify webhook URLs with non-http(s) protocols (e.g. ftp://, file://) are now rejected at lint time with code notify-unsupported-protocol, matching the runtime guard. * fix(rules): validate --llm enum at CLI; warn when auto falls back to heuristic - `rules suggest --llm <backend>` now rejects unknown values with a UsageError (exit 2). Previously any string fell through and was passed along, producing confusing downstream errors. MCP rule_suggest already enforced this via Zod enum. - When `--llm auto` falls back to the heuristic because intent complexity is below the LLM threshold, the result now carries a warning telling the caller they can pass `--llm openai|anthropic` to force LLM. * fix(rules): record llm audit after final outcome; validate llm output against policy schema - writeAudit for kind=llm-suggest now runs after the parse/validate/lint pipeline, with result='ok' or 'error' set from the final outcome and the error message captured in the entry. Previously every LLM call recorded result='ok' even when the suggestion was rejected downstream. - Generated rules are now wrapped into a synthetic v0.2 policy and run through validateLoadedPolicy before lint, so structural defects (wrong types, missing fields, unknown enums) are caught with a clear schema error instead of crashing further down the pipeline. Test fixtures updated to use plausible SwitchBot device IDs. * docs: document llm constraint enforcement and notify url protocol rules - README rules-suggest section now spells out the four guardrails users should expect: forced dry_run, hard alignment with explicit overrides, CLI --llm enum validation, and the http(s)-only notify URL rule. - Notify channel descriptions explicitly call out that lint rejects schemes other than http:// / https://. * fix(rules): expand nested notify vars, reject relative file paths, complete llm prompt schema PR review P2 fixes: - notify template now flattens nested event payloads into dotted keys (event.context.deviceMac, event.list.0). Renderer used to spread only top-level payload keys, so documented nested placeholders rendered literally. Depth-capped at 6; Date instances become ISO strings. - channel: file rejects non-absolute paths at lint (notify-relative-path) and at runtime. Schema already documented absolute-path requirement; enforcement was missing, so logs/notify.jsonl resolved against process cwd, which differs across rules run / daemon / systemd contexts. Tilde is not expanded. - llm rule-suggest prompt embeds trigger and triggerWebhook defs in addition to triggerMqtt/triggerCron, eliminating the dangling \$ref on rule.when. Adds a Rules-section bullet describing the webhook trigger shape so --trigger webhook flows reach schema validation cleanly. * chore(rules): polish from code review - runtime appendToFile error mirrors the lint hint (absolute path example, tilde-not-expanded note) so users skipping lint get the same guidance - comment near rule-notify writeAudit explains keeping action.to unsanitised is intentional forensic behaviour, not an oversight - comment near the rule-suggest prompt template warns that tests/llm/rule-prompt.test.ts string-slices on the "JSON Schema:\n" / "\n\nRules:" delimiters --------- Co-authored-by: chenliuyun <chenliuyun@onero.com>
1 parent b30cdcf commit be51931

32 files changed

Lines changed: 2138 additions & 84 deletions

CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,14 @@ All notable changes to `@switchbot/openapi-cli` are documented in this file.
77
The format is loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
88
This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
99

10+
## [Unreleased]
11+
12+
### Fixed
13+
14+
- notify templates now expand nested event fields — dotted paths like `{{ event.context.deviceMac }}` and array indexes like `{{ event.list.0 }}` resolve instead of rendering literally.
15+
- `channel: file` notify actions reject relative paths (lint code `notify-relative-path`, also enforced at runtime); `~` is not expanded.
16+
- LLM rule-suggest prompt now embeds the `trigger` and `triggerWebhook` schema defs so `--trigger webhook` flows no longer produce dangling-`$ref` outputs.
17+
1018
## [3.3.2] - 2026-04-26
1119

1220
### Fixed

README.md

Lines changed: 56 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,24 @@ Supported conditions: `time_between` (quiet hours) and `device_state`
278278
`~/.switchbot/audit.log`. `rules run` is long-running; use
279279
`daemon start` / `daemon reload` for the managed background mode.
280280

281+
**Actions** — each rule's `then` array accepts two action types:
282+
283+
- `type: command` (default, no `type` field required) — sends a device command, e.g. `devices command <id> turnOn`
284+
- `type: notify` — delivers a payload to an external channel after the rule fires:
285+
- `channel: webhook` — HTTP POST to a URL (only `http://` and `https://` schemes are accepted; `rules lint` rejects others)
286+
- `channel: file` — appends a JSONL line to a local file. `to` must be an absolute path; relative or `~`-prefixed paths are rejected by `rules lint` (code `notify-relative-path`) and at runtime
287+
- `channel: openclaw` — HTTP POST to an OpenClaw endpoint (same protocol restriction)
288+
- Optional `template` field supports `{{ rule.name }}`, `{{ event.* }}`, `{{ device.id }}` placeholders. Nested fields use dot paths, e.g. `{{ event.context.deviceMac }}`; arrays index numerically, e.g. `{{ event.list.0 }}`
289+
290+
```yaml
291+
then:
292+
- command: devices command AC_001 turnOn
293+
- type: notify
294+
channel: webhook
295+
to: https://your.host/hook
296+
template: '{"rule":"{{ rule.name }}","fired":"{{ rule.fired_at }}"}'
297+
```
298+
281299
```bash
282300
# 1. Author rules under `automation.rules`. See examples/policies/automation.yaml
283301
# for a walkthrough covering the three trigger sources.
@@ -308,8 +326,29 @@ switchbot rules last-fired -n 20 # 20 most recent fire entries
308326
switchbot rules conflicts # opposing actions, high-frequency MQTT,
309327
# destructive commands, quiet-hours gaps
310328
switchbot rules doctor --json # lint + conflicts combined; exit 0 when clean
329+
330+
# 8. Scaffold a new rule from natural language (heuristic or LLM-backed).
331+
switchbot rules suggest --intent "turn off AC at 11pm"
332+
switchbot rules suggest --intent "if door opens and temp below 20 turn on heater" \
333+
--llm auto # routes complex intents to LLM automatically
334+
switchbot rules suggest --intent "..." --llm openai # explicit backend
335+
# Set OPENAI_API_KEY or ANTHROPIC_API_KEY; auto mode falls back to heuristic on failure
311336
```
312337

338+
`rules suggest` enforces several guardrails on LLM output so a model can't quietly arm
339+
something unsafe:
340+
341+
- **`dry_run` is forced to `true`** on every LLM-generated rule. Review the output and
342+
flip it yourself before running the engine without `--dry-run`.
343+
- **Explicit overrides always win.** If you pass `--trigger`, the LLM's answer must match;
344+
a mismatch fails fast. Within the same trigger, mismatched `--event` / `--schedule` /
345+
`--days` / `--webhook-path` are rewritten to your value with a warning.
346+
- **`--llm` is enum-validated at the CLI** (`auto | openai | anthropic`) — junk values
347+
exit non-zero instead of falling through.
348+
- **Notify URLs must be `http://` or `https://`.** `rules lint` and the runtime both
349+
reject `file://`, `ftp://`, etc., so a generated webhook can't smuggle in a non-HTTP
350+
scheme.
351+
313352
When `quiet_hours` is configured in `policy.yaml`, `rules conflicts` additionally flags event-driven (MQTT / webhook) rules that lack a `time_between` condition — they would fire uninhibited during the quiet window. The hint in each finding includes a ready-to-paste `time_between` condition to add.
314353

315354
Webhook trigger token management:
@@ -842,8 +881,12 @@ Exposes MCP tools (`list_devices`, `describe_device`, `get_device_status`,
842881
`send_command`, `list_scenes`, `run_scene`, `search_catalog`,
843882
`account_overview`, `plan_suggest`, `plan_run`, `audit_query`,
844883
`audit_stats`, `policy_diff`, `policy_validate`, `policy_new`,
845-
`policy_migrate`) plus a `switchbot://events` resource for real-time
846-
shadow updates.
884+
`policy_migrate`, `rules_suggest`, `rule_notifications`) plus a
885+
`switchbot://events` resource for real-time shadow updates.
886+
`rules_suggest` accepts an optional `llm` parameter (`openai | anthropic | auto`)
887+
to generate YAML for complex intents via an LLM backend.
888+
`rule_notifications` returns `rule-notify` audit entries, filterable by rule
889+
name, time range, channel, and result.
847890
See [`docs/agent-guide.md`](./docs/agent-guide.md) for the full tool reference and safety rules (destructive-command guard).
848891

849892
### `doctor` — self-check
@@ -853,7 +896,7 @@ switchbot doctor
853896
switchbot doctor --json
854897
```
855898

856-
Runs local checks (Node version, credentials, profiles, catalog, cache, quota, clock, MQTT, policy, MCP) and exits 1 if any check fails. `warn` results exit 0. The MQTT check reports `ok` when REST credentials are configured (auto-provisioned on first use). Use this to diagnose connectivity or config issues before running automation.
899+
Runs local checks (Node version, credentials, profiles, catalog, cache, quota, clock, MQTT, policy, MCP, notify-connectivity) and exits 1 if any check fails. `warn` results exit 0. The MQTT check reports `ok` when REST credentials are configured (auto-provisioned on first use). The `notify-connectivity` check probes webhook URLs declared in `type: notify` actions. Use this to diagnose connectivity or config issues before running automation.
857900

858901
`--json` output includes `maturityScore` (0–100) and `maturityLabel` (`production-ready` / `mostly-ready` / `needs-work` / `not-ready`) to give an at-a-glance readiness rating:
859902

@@ -1165,8 +1208,16 @@ src/
11651208
│ ├── audit-query.ts # Audit log filtering + aggregation
11661209
│ ├── conflict-analyzer.ts # Static conflict detection (opposing actions,
11671210
│ │ # high-freq MQTT, destructive cmds, quiet-hours gaps)
1168-
│ ├── suggest.ts # Heuristic-based rule YAML generation
1169-
│ └── types.ts # Shared rule/trigger/condition/action types
1211+
│ ├── suggest.ts # Heuristic + LLM-backed rule YAML generation
1212+
│ ├── notify.ts # notify action executor (webhook / file / openclaw)
1213+
│ └── types.ts # Shared rule/trigger/condition/action types (CommandAction | NotifyAction)
1214+
├── llm/
1215+
│ ├── index.ts # createLLMProvider factory + LLM_AUTO_THRESHOLD
1216+
│ ├── complexity.ts # Intent complexity scorer (0–10) for auto-routing
1217+
│ ├── rule-prompt.ts # System prompt builder (embeds v0.2 schema snippet)
1218+
│ └── providers/
1219+
│ ├── openai.ts # OpenAI-compatible provider (uses Node.js https)
1220+
│ └── anthropic.ts # Anthropic provider
11701221
├── status-sync/
11711222
│ └── manager.ts # Spawn/stop logic, state file, OpenClaw bridge
11721223
├── lib/

docs/design/roadmap.md

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Roadmap — Phase 1 through Phase 4
22

3-
> **Status as of 2026-04-23:** Phase 1 complete, Phase 2 complete,
3+
> **Status as of 2026-05-06:** Phase 1 complete, Phase 2 complete,
44
> Phase 3A complete (keychain + install library + built-in CLI install
55
> command), Phase 3B tracked in the separate companion skill repo,
66
> Phase 4 shipped at v0.2 (rules engine with MQTT + cron +
@@ -9,6 +9,8 @@
99
> v2.14.0 extends MCP with `plan_run`, `audit_query`, `audit_stats`,
1010
> and `policy_diff`; v2.15.0 flips `policy new` default schema to v0.2
1111
> and starts the v0.1 deprecation window.
12+
> Tracks θ (notify actions) and η (LLM-backed rule suggestion)
13+
> shipped in v3.0.
1214
> Note: Track γ is a runtime capability increment on the v0.2 rule
1315
> model, not a separate policy schema version.
1416
@@ -181,6 +183,19 @@ the skill's `manifest.json` `roadmap` block, which points back here.
181183
- **Track ε — cross-OS CI matrix for keychain *(shipped, v2.11.0)*.**
182184
GitHub Actions matrix: macOS (temp keychain), Linux (D-Bus +
183185
gnome-keyring), Windows (native Credential Manager).
186+
- **Track θ — notify actions *(shipped, v3.0)*.**
187+
New `type: notify` action alongside `type: command` in the v0.2
188+
schema. Rules can POST to webhooks, append JSONL to a file, or push
189+
to OpenClaw after firing, closing the feedback loop for AI agents.
190+
`lintRules` validates URL syntax and required fields; engine dispatches
191+
to `executeNotifyAction`; audit gains `rule-notify` kind; MCP gains
192+
`rule_notifications` tool; `doctor` gains `notify-connectivity` check.
193+
- **Track η — LLM-backed rule suggestion *(shipped, v3.0)*.**
194+
`rules suggest --llm <backend>` routes complex intents (complexity
195+
score ≥ 4) to OpenAI or Anthropic, falls back to heuristic with a
196+
warning on provider failure. `rules_suggest` MCP tool gains a `llm`
197+
parameter. All LLM calls are written to the audit log as
198+
`kind: llm-suggest` with backend, model, and latency fields.
184199

185200
## Next execution queue (ordered)
186201

src/commands/doctor.ts

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -922,6 +922,41 @@ function checkReleaseNotes(): Check {
922922
};
923923
}
924924

925+
function checkNotifyConnectivity(): Check {
926+
const policyPath = resolvePolicyPath();
927+
let loaded: { data: unknown };
928+
try {
929+
loaded = loadPolicyFile(policyPath);
930+
} catch (err) {
931+
if (err instanceof PolicyFileNotFoundError) {
932+
return { name: 'notify-connectivity', status: 'ok', detail: { present: false, message: 'no policy file configured' } };
933+
}
934+
return { name: 'notify-connectivity', status: 'ok', detail: { present: false, message: 'policy file could not be loaded' } };
935+
}
936+
937+
const policy = loaded.data as { automation?: { rules?: Array<{ then?: Array<{ type?: string; channel?: string; to?: string }> }> } } | null;
938+
const rules = policy?.automation?.rules ?? [];
939+
const webhookUrls: string[] = [];
940+
for (const rule of rules) {
941+
for (const action of rule.then ?? []) {
942+
if (action.type === 'notify' && (action.channel === 'webhook' || action.channel === 'openclaw') && action.to) {
943+
webhookUrls.push(action.to);
944+
}
945+
}
946+
}
947+
948+
if (webhookUrls.length === 0) {
949+
return { name: 'notify-connectivity', status: 'ok', detail: { webhookCount: 0, message: 'no webhook notify actions in policy' } };
950+
}
951+
952+
return {
953+
name: 'notify-connectivity',
954+
status: 'ok',
955+
detail: { webhookCount: webhookUrls.length, message: `${webhookUrls.length} webhook URL(s) configured (live probe not run — use --probe to test connectivity)` },
956+
};
957+
}
958+
959+
925960
interface CheckDef {
926961
name: string;
927962
description: string;
@@ -955,6 +990,7 @@ const CHECK_REGISTRY: CheckDef[] = [
955990
{ name: 'audit', description: 'recent command errors (last 24h)', run: () => checkAudit() },
956991
{ name: 'daemon', description: 'daemon state file + runtime status', run: () => checkDaemon() },
957992
{ name: 'health', description: 'health endpoint availability (daemon --healthz-port)', run: () => checkHealthEndpoint() },
993+
{ name: 'notify-connectivity', description: 'webhook URLs from notify actions in policy.yaml', run: () => checkNotifyConnectivity() },
958994
];
959995

960996
interface FixResult {

src/commands/mcp.ts

Lines changed: 51 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1874,14 +1874,59 @@ API docs: https://github.com/OpenWonderLabs/SwitchBotAPI`,
18741874
},
18751875
);
18761876

1877+
// ---- rule_notifications ---------------------------------------------------
1878+
server.registerTool(
1879+
'rule_notifications',
1880+
{
1881+
title: 'Query rule notification delivery history',
1882+
description:
1883+
'Returns audit entries for notify actions (kind: rule-notify). ' +
1884+
'Useful for confirming whether a notification rule fired and whether delivery succeeded. ' +
1885+
'Filter by rule name, time range, result, or channel.',
1886+
_meta: { agentSafetyTier: 'read' },
1887+
inputSchema: z.object({
1888+
file: z.string().optional().describe('Optional audit log path; defaults to ~/.switchbot/audit.log.'),
1889+
rule: z.string().optional().describe('Filter by rule name (exact match).'),
1890+
since: z.string().optional().describe('Relative window ending now (e.g. "30m", "24h").'),
1891+
from: z.string().optional().describe('Range start (ISO-8601).'),
1892+
to: z.string().optional().describe('Range end (ISO-8601).'),
1893+
result: z.enum(['ok', 'error']).optional().describe('Filter by delivery result.'),
1894+
channel: z.enum(['webhook', 'openclaw', 'file']).optional().describe('Filter by notify channel.'),
1895+
limit: z.number().int().min(1).max(500).default(100).describe('Max entries to return (newest first).'),
1896+
}).strict(),
1897+
outputSchema: {
1898+
entries: z.array(z.unknown()).describe('Matching audit entries, newest first.'),
1899+
total: z.number().int().describe('Count after filtering.'),
1900+
},
1901+
},
1902+
({ file, rule: ruleName, since, from, to, result: resultFilter, channel, limit }) => {
1903+
const filePath = file ?? DEFAULT_AUDIT_LOG_FILE;
1904+
let entries = readAudit(filePath).filter(e => e.kind === 'rule-notify');
1905+
if (ruleName) entries = entries.filter(e => e.rule?.name === ruleName);
1906+
if (resultFilter) entries = entries.filter(e => e.result === resultFilter);
1907+
if (channel) entries = entries.filter(e => e.notifyChannel === channel);
1908+
try {
1909+
entries = filterAuditEntries(entries, { since, from, to });
1910+
} catch (err) {
1911+
return mcpError('usage', 2, err instanceof Error ? err.message : 'invalid filter options');
1912+
}
1913+
const bounded = entries.slice(-limit).reverse();
1914+
const out = { entries: bounded, total: entries.length };
1915+
return {
1916+
content: [{ type: 'text' as const, text: JSON.stringify(out, null, 2) }],
1917+
structuredContent: out,
1918+
};
1919+
},
1920+
);
1921+
18771922
// ---- rules_suggest --------------------------------------------------------
18781923
server.registerTool(
18791924
'rules_suggest',
18801925
{
18811926
title: 'Draft a SwitchBot automation rule from intent',
18821927
description:
18831928
'Generate a candidate automation rule YAML from a natural language intent. ' +
1884-
'Uses keyword heuristics (no LLM) to infer trigger, schedule, and command. ' +
1929+
'Uses keyword heuristics by default; pass llm to use an LLM backend (auto | openai | anthropic). ' +
18851930
'Always emits dry_run: true — the rule must be reviewed before arming. ' +
18861931
'Pass the returned rule_yaml to policy_add_rule to inject it into policy.yaml.',
18871932
_meta: { agentSafetyTier: 'read' },
@@ -1893,27 +1938,29 @@ API docs: https://github.com/OpenWonderLabs/SwitchBotAPI`,
18931938
schedule: z.string().optional().describe('5-field cron expression override (e.g. "0 22 * * *").'),
18941939
days: z.array(z.string()).optional().describe('Weekday filter (e.g. ["mon","tue","wed","thu","fri"]).'),
18951940
webhook_path: z.string().optional().describe('Webhook path override (default /action).'),
1941+
llm: z.enum(['auto', 'openai', 'anthropic']).optional().describe('LLM backend (auto | openai | anthropic). Omit to use keyword heuristics.'),
18961942
}).strict(),
18971943
outputSchema: {
18981944
rule: z.unknown().describe('Rule object matching the v0.2 policy schema.'),
18991945
rule_yaml: z.string().describe('YAML string ready to pipe to policy_add_rule.'),
19001946
warnings: z.array(z.string()).describe('Informational warnings (e.g. unrecognized intent defaulted).'),
19011947
},
19021948
},
1903-
({ intent, trigger, device_ids, event, schedule, days, webhook_path }) => {
1949+
async ({ intent, trigger, device_ids, event, schedule, days, webhook_path, llm }) => {
19041950
const devices = (device_ids ?? []).map((id) => {
19051951
const cached = getCachedDevice(id);
19061952
return { id, name: cached?.name, type: cached?.type };
19071953
});
19081954
try {
1909-
const { rule, ruleYaml, warnings } = suggestRule({
1955+
const { rule, ruleYaml, warnings } = await suggestRule({
19101956
intent,
19111957
trigger,
19121958
devices,
19131959
event,
19141960
schedule,
19151961
days,
19161962
webhookPath: webhook_path,
1963+
llm,
19171964
});
19181965
return {
19191966
content: [{ type: 'text' as const, text: ruleYaml }],
@@ -2039,6 +2086,7 @@ export function registerMcpCommand(program: Command): void {
20392086
- plan_run validate + execute a Plan JSON document
20402087
- audit_query filter audit log entries by time/device/rule/result
20412088
- audit_stats aggregate audit counts by kind/result/device/rule
2089+
- rule_notifications query rule notify action delivery history
20422090
- rules_suggest draft an automation rule YAML from intent (heuristic, no LLM)
20432091
- policy_add_rule append a rule into automation.rules[] in policy.yaml
20442092

0 commit comments

Comments
 (0)