AI Agents framework, Phase A: synthetic users + user-meta definitions, abilities runner, chat - #428
Draft
epeicher wants to merge 7 commits into
Draft
AI Agents framework, Phase A: synthetic users + user-meta definitions, abilities runner, chat#428epeicher wants to merge 7 commits into
epeicher wants to merge 7 commits into
Conversation
Agents are durable workers that live on the site as real WordPress
users and act through the Abilities API under their own role. An agent
is a login-blocked wp_users row plus a _desktop_mode_agent_* user-meta
family holding the whole definition: description, instructions
(system prompt), ability allowlist, triggers, model override, rate
limit. No wp_guideline dependency; the audit trail for definition
changes is the desktop_mode_agent_{created,updated,deleted} actions,
each carrying before/after values.
Behind the new 'agents' extended option (default off, games-style
module gating). Phase A ships:
- includes/agents/: store (meta CRUD + orchestrators), identity
(login/password-reset/app-password blocks, bot avatar, Users list
column), abilities bridge (desktop-mode/get-post + update-post,
picker catalogue with readonly badges), runner on the Core AI
Client (8-turn cap, per-agent hourly rate limit, identity switch in
try/finally, provider-safe tool-schema normalization shared with
the Copilot), REST CRUD + /invoke + catalogues, privacy
exporter/eraser, Agent chat native window, My WordPress entity.
- Client: 'agent' entity-kind renderer in the My WordPress bundle
(list, Define/Tools/Triggers panes, create flow, live AI-provider
probe) and a new lazy agent-run-window bundle fed through the
cross-bundle desktop-mode/agents-chat shared store.
- 45 PHPUnit + 20 vitest tests; docs (hooks-reference,
javascript-reference, api-index, architecture, rest README,
examples/agents.md) and the implementation plan under docs/plans/.
Chat is the only wired trigger; send-to/drag, hook, endpoint, and
agent-to-agent intakes are declared in the trigger-kind catalogue and
land in later phases.
Two Gemini-surfaced fixes in the agent invocation path: - Strip the WordPress-only arg-schema keys (sanitize_callback, validate_callback, arg_options) from tool schemas at every depth, in the shared desktop_mode_ai_normalize_tool_schema() so the Copilot benefits too. Strict providers reject any unknown field and 400 the whole request over one property. The walk is structure-aware: property NAMES are never stripped, only schema-level keys; covers properties, patternProperties, single and tuple items, array-shaped additionalProperties, and nested combinators. - Stop replaying assistant functionCall turns to the provider. Each generate turn now sends one user message: the original request plus a text transcript of executed tool calls and their results. Replaying functionCall parts requires provider-specific signatures (Gemini thought_signature, Anthropic thinking signatures) that the current provider plugins do not round-trip, and one missing signature 400s the request. Text transcripts carry the same information with no signature or pairing constraints on any provider.
Agents (and the Copilot, via the readonly annotation) can now read a media library item by id: file URL, mime type, dimensions, alt text, caption, and the post it is attached to. Closes the gap where no ability on a stock site could read attachments, so image-referencing prompts dead-ended. Permission gates on upload_files (author+), deliberately not on read_post: for inherit-status attachments that check defers to the parent (and effectively requires edit rights when unattached), which wrongly blocks read-only access to media whose file URL is public on a standard site anyway.
…ity) New standalone extension plugin under extensions/, riding the agents PR as the first real consumer of the framework: an ability that removes the background from a media library image and sideloads the result as a NEW png attachment (original untouched), authored by the calling user — for agents, the agent's own account, completing the attribution story end to end. Mutating ability (no readonly annotation): invisible to the Copilot, reachable only through an agent's explicit allowlist. Pluggable backends behind desktop_mode_remove_background_backends: remove.bg (default, key required), a self-hosted rembg server, and an experimental WordPress AI Client generative-editing backend that reuses the site's configured connectors. Settings live natively on Settings -> Media. A desktop_mode_remove_background_pre short-circuit filter keeps PHPUnit network-free; seven tests cover registration, the execute lifecycle, permissions, and runner dispatch with agent attribution.
The extension's only user-facing surface is the ability itself — no
Settings -> Media section. Configuration is code/CLI-level, resolved
option -> constants (DESKTOP_MODE_REMOVE_BG_{BACKEND,API_KEY,ENDPOINT})
-> desktop_mode_remove_background_settings filter, with an unknown
backend slug falling back to the default. README documents all three
paths; error messages point at them instead of the removed screen.
The ai backend rides the site's existing Connectors credentials, so a stock install needs no extension-specific key at all — remove.bg and rembg become the opt-in paths for mask-based quality. The AI Client resolves the model from the prompt's modalities (an input image requires an image-input-capable model), so providers that only do text-to-image are never silently picked for an edit.
wp_ai_client_prompt() returns WordPress's snake_case wrapper (WP_AI_Client_Prompt_Builder), not the SDK PromptBuilder: generating methods are generate_* and failures come back as WP_Error instead of exceptions. The camelCase generateImageResult() call fell through the wrapper's fluent path and returned the builder itself, which the backend then reported as 'no readable image data' regardless of what the provider said. Now calls generate_image_result(), propagates WP_Error verbatim (so provider errors like quota exhaustion surface readably in the agent's tool trace), and keeps the try/catch for result-shape surprises. Regression test drives the real builder path with no connector configured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
Ships Phase A of the AI Agents framework: durable workers that live on the site as real WordPress users, take orders by chat, and act through the WordPress Abilities API under their own role and capabilities.
Behind the new
agentsextended option (OS Settings → Features → Extended options, admin-only, default off). With the flag on, a new Agents section appears in My WordPress: create an agent (name, role, description, system prompt), tick the abilities it may use in the Tools pane, configure triggers, and talk to it in the Agent chat window. Every tool call runs as the agent's own user and lands in the standard audit trail (revisions, comments,_edit_lock) attributed to the agent.Rationale
This is the buildable slice of the direction #240 mocked, with one deliberate storage change: no
wp_guidelineCPT. An agent is split across exactly two layers:wp_usersrow. Real role, real capability checks, real attribution. Every login path is blocked (authenticatefilter, password resets, application passwords), the email is a never-delivered synthetic address, and the wp-admin Users list labels the row "Agent"._desktop_mode_agent_*): description, instructions, ability allowlist, triggers, model override, rate limit.Dropping the guideline CPT removes the Gutenberg Guidelines-experiment dependency entirely (no soft-gate, no 412 paths). The accepted trade-off is losing free revisions on prompt edits; instead, every mutation fires
desktop_mode_agent_{created,updated,deleted}with before/after values per changed field, so logging plugins get a complete audit trail.Implementation
Runner (
includes/agents/runner.php):desktop_mode_agent_invoke()generates through the same Core AI Client adapter the Copilot uses (desktop_mode_ai_client_generate()overwp_ai_client_prompt()), loops function calls to a hard 8-turn cap, and dispatches each call viaWP_Ability::execute()so the ability's ownpermission_callbackand schemas gate it. The whole loop runs withwp_set_current_user()switched to the agent, restored infinally:Tool schemas advertised to the model are projected through the Copilot's
desktop_mode_ai_normalize_tool_schema()(providers 400 the whole request over one tool with a top-leveloneOf/anyOf/allOf). Per-agent hourly rate limits ride a transient counter (default 60/hour, filterable). Adesktop_mode_agent_runner_generatepre-filter short-circuits the AI Client, which is how PHPUnit exercises the full loop network-free.Tools: the picker is a view over
wp_get_abilities()with honest read-only vs mutating badges frommeta.annotations.readonly. Unlike the Copilot (read-only only), agents may be granted mutating abilities; the compensating controls are the explicit allowlist set by anedit_usershuman plus the agent's role. Three new abilities ship:desktop-mode/get-postanddesktop-mode/get-media(read-only) anddesktop-mode/update-post(mutating, reachable only through an agent allowlist).REST (
/desktop-mode/v1/agents): CRUD +/invoke+ abilities/trigger-kinds/hooks/roles catalogues. Reads and invokes default toedit_posts, writes toedit_users(all filterable); role assignment is constrained to a whitelist intersected with the acting user'sget_editable_roles().Client: an
agententity kind registered through the existingregisterEntityKind()seam in the My WordPress bundle (list, Define/Tools/Triggers panes, create flow, live provider probe against/ai/status), plus a new lazyagent-run-windowbundle for the chat window, fed through the cross-bundledesktop-mode/agents-chatshared store.Triggers: chat is the only wired intake in Phase A. The other kinds (
send-to,drag,hook,endpoint,agent) are declared in the trigger-kind catalogue so configuration can be stored now; their intakes land in later phases. All of them collapse to the samedesktop_mode_agent_invoke()engine, anddesktop_mode_agent_completedis the chaining seam.First consumer — Remove Background extension (
extensions/desktop-mode-remove-background/): a standalone plugin registeringmedia-tools/remove-background, a mutating ability that removes an image's background and sideloads the result as a new PNG attachment authored by the agent. It touches zero desktop-mode internals — its whole integration surface iswp_register_ability()— which is the framework's compatibility claim demonstrated inside the PR. Backends are pluggable; the default is the WordPress AI Client (generative editing riding the site's Connectors credentials — no extension-specific key), with remove.bg and self-hosted rembg as opt-in mask-based alternatives. Configuration is deliberately code-level only (option, constants, or filter — no admin settings UI), so the ability is the extension's entire user-facing surface.Testing instructions
Manual, on a WP 7.0+ site with an AI connector configured:
author, prompt like "You audit posts"). Tickdesktop-mode/get-post+desktop-mode/update-postin Tools.