simplify workflow history and add page tracking

[hanna-paasivirta]

Short Description

Supports merging workflow and job code chats OpenFn/lightning#4109

Adds page navigation tracking to both job_chat and workflow_chat services to:

• Show context: Display page info in user messages (visible in history)
• Auto-refresh RAG: In job_chat, automatically refresh documentation when user navigates to a different page
• Redacts workflow_chat history to match job_chat history format

Fixes #350

Implementation Details

TODO: Prompt:

• Add note in job_chat and workflow_chat prompts to explain that earlier conversation turns may have had different context & different answer formats, and that the attachments have been removed
• Add a note in job_chat prompt as well to ask the user to navigate to the workflow if relevant. Match to the note in workflow_chat. These could specify that the user can also copy the context into the conversation.

History/RAG:

• Normalise history across the two services: redact workflow_chat history to only have plain text, i.e. remove the nested suggested YAML key
• Use Anthropic structured outputs to help with consistency
• Add a new page identifier (type, name) to history and the latest user message, which will identify what the conversation relates to, and whether we need to run RAG. We should re-run RAG if the current message relates to a job & that job is different from the last message in the history.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

put page.get(name) into an if as well

We should know the page type because in job chat, the page type is always job chat