Theorizer should surface whether user-supplied PDFs are used
Relocated from allenai/gas2own#660 — this is a plugin/skill behavior gap and belongs on the Asta team's tracker, not gas2own. Sourced from Asta feedback submission 2026-08-12-theorizer-web-viewer (retrieved from the feedback bucket).
Skill
generate-theories, interacting with local-paper-index when local PDFs were supplied earlier in the research workflow.
Prompt / setup
The user supplied four qualitative-methodology PDFs during a multi-stage Theorizer workflow. The agent indexed them locally, then ran find-and-extract, form-theory, and evaluate-novelty against published literature.
Current behavior
The agent did not say that the local PDFs would be excluded from Theorizer's evidence base, did not offer the available paper_store injection path, and did not report evidence provenance after the run. The user only learned after asking that none of the supplied PDFs had influenced extraction or theory formation.
This is a silent correctness risk: a user can reasonably interpret the resulting theories as grounded in sources they explicitly supplied when those sources were never used.
Desired behavior
When local PDFs or a local paper index are in session context and Theorizer is about to search/extract:
- Explicitly offer to inject the supplied papers through the supported
paper_store input, or explain that they will not be used.
- Do not imply that local indexing alone puts those papers into Theorizer's evidence base.
- After the evidence-gathering stage, report provenance: papers retrieved by PaperFinder versus user-supplied papers.
- Discover the current input schema from the agent card rather than hard-coding an obsolete CLI shape.
Validation target
Add an asta_skills behavior case where a user supplies local PDFs and then requests theory generation. The metric should require the agent to disclose exclusion or offer the supported injection path before running Theorizer, plus regression guards for ordinary theory-generation requests without local documents.
Suggested-by: @jbragg
Theorizer should surface whether user-supplied PDFs are used
Relocated from allenai/gas2own#660 — this is a plugin/skill behavior gap and belongs on the Asta team's tracker, not gas2own. Sourced from Asta feedback submission
2026-08-12-theorizer-web-viewer(retrieved from the feedback bucket).Skill
generate-theories, interacting withlocal-paper-indexwhen local PDFs were supplied earlier in the research workflow.Prompt / setup
The user supplied four qualitative-methodology PDFs during a multi-stage Theorizer workflow. The agent indexed them locally, then ran
find-and-extract,form-theory, andevaluate-noveltyagainst published literature.Current behavior
The agent did not say that the local PDFs would be excluded from Theorizer's evidence base, did not offer the available
paper_storeinjection path, and did not report evidence provenance after the run. The user only learned after asking that none of the supplied PDFs had influenced extraction or theory formation.This is a silent correctness risk: a user can reasonably interpret the resulting theories as grounded in sources they explicitly supplied when those sources were never used.
Desired behavior
When local PDFs or a local paper index are in session context and Theorizer is about to search/extract:
paper_storeinput, or explain that they will not be used.Validation target
Add an
asta_skillsbehavior case where a user supplies local PDFs and then requests theory generation. The metric should require the agent to disclose exclusion or offer the supported injection path before running Theorizer, plus regression guards for ordinary theory-generation requests without local documents.Suggested-by: @jbragg