Skip to content

Add hosting-plans-helper skill for plan and pricing answers#4010

Open
gavande1 wants to merge 10 commits into
trunkfrom
stu-1940-hosting-plans-helper-skill
Open

Add hosting-plans-helper skill for plan and pricing answers#4010
gavande1 wants to merge 10 commits into
trunkfrom
stu-1940-hosting-plans-helper-skill

Conversation

@gavande1

@gavande1 gavande1 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Related issues

How AI was used in this PR

AI assisted with investigating the WordPress.com plans and products APIs, and with drafting the skill, the system-prompt guardrail, and the tests. I reviewed all changes.

Proposed Changes

Studio Code answered WordPress.com plan, pricing, and feature questions from stale training knowledge, so it recommended renamed plans and incorrect feature gating. This adds a hosting-plans-helper skill and a system-prompt rule that requires the agent to answer plan, pricing, and feature questions from current data instead of memory. The skill reads plan names and per-tier features from wpcom/v2/plans/features and prices from wpcom/v2/products, then answers directly without citing the source.

Note: the skill depends on a new wpcom/v2/plans/features endpoint that does not exist yet (separate backend work). Until it ships, the feature fetch returns nothing; the guardrail and price path are already in place.

Testing Instructions

  • Checkout 225833-ghe-Automattic/wpcom on sandbox
  • Checkout this branch
  • npm start
  • Open Studio Code, Terminal, Agentic UI
  • Ask any question about wordpress.com plans.
  • Confirm hosting-plans-helper appears in the Skill tool list.
  • After the backend endpoint is live, ask Studio Code a plan question and confirm it answers from fetched data with correct names and localized prices.
CleanShot.2026-06-30.at.18.29.45.mp4

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors? (npm run typecheck passes)

@gavande1 gavande1 marked this pull request as ready for review June 30, 2026 13:45
@gavande1 gavande1 requested a review from a team June 30, 2026 13:54

@bcotrim bcotrim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some feedback from testing:

  • ✅ Skill is consistently triggered when asking about hosting/plans.
  • ⚠️ It still says Personal/Premium can’t install plugins — but plans/features lists install-plugins on Personal and up. So it’s answering from memory and contradicting the fetched data, which is the exact thing the skill is meant to prevent.
  • ⚠️ The slug in wpcom/v2/plans/features (premium) doesn’t match the key in wpcom/v2/products (value_bundle) — and there’s no shared field to join on, so the agent can only guess the price mapping (Premium → value_bundle shares no substring, so that one’s especially unreliable). Could we add the product slug to plans/features so consumers can join directly?

@wpmobilebot

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing 682fe94 vs trunk

app-size

Metric trunk 682fe94 Diff Change
App Size (Mac) 1316.77 MB 1316.77 MB +0.00 MB ⚪ 0.0%

site-editor

Metric trunk 682fe94 Diff Change
load 1062 ms 1114 ms +52 ms 🔴 4.9%

site-startup

Metric trunk 682fe94 Diff Change
siteCreation 6502 ms 6487 ms 15 ms ⚪ 0.0%
siteStartup 1860 ms 6553 ms +4693 ms 🔴 252.3%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

@gavande1

gavande1 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

@bcotrim Thanks for testing. I addressed both points. The plan guardrail is now a top-level rule that forbids answering plan questions from memory and calls out the plugins case specifically. For the price join, the backend now returns product_slug and the skill uses it. Do you mind taking another look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants