Merge dev into main: integration ecosystem refactor and runtime memory alignment

[ruttydm]

Summary

This PR merges the current dev branch into main and consolidates three layers of work that were previously split across separate branches and PRs:

1. the integration ecosystem refactor
2. the runtime memory-alignment pipeline
3. the merge follow-up fixes required to make those two lines of work coexist cleanly on one branch

In practical terms, this PR replaces the previous incremental branch-by-branch review flow with a single integration branch that represents the combined product direction.

This PR supersedes:

• feat: Integration ecosystem refactor — monorepo, shared core, multi-account #4 feat: Integration ecosystem refactor — monorepo, shared core, multi-account
• [codex] Implement runtime memory alignment pipeline #5 [codex] Implement runtime memory alignment pipeline

Why this PR exists

The repo had two meaningful active lines of work:

• feat/integration-refactor
  This changed how integrations, MCP tools, provider metadata, Lua docs, and multi-account configuration are structured.

• codex/runtime-memory-alignment
  This changed how agent prompts are shaped, cached, compacted, truncated, and tested under context pressure.

Reviewing or merging them independently no longer made much sense because they overlap in runtime wiring, provider resolution, docs, and dependency layout. The dev branch was created to integrate both branches, resolve conflicts, and then fix the branch-combination regressions discovered during validation.

High-level outcome

After this PR:

• integrations are organized around the sibling ../integrations monorepo and shared integration-core
• runtime memory handling is more explicit, testable, and configurable
• AI gateway wiring routes through prompt-cache-aware Prism bridging
• multi-account integration configuration is supported in app models/controllers
• Lua docs and tool catalog generation are more aligned with the shared ecosystem direction
• large blocks of stale planning/test-map docs are replaced by current ecosystem/runtime documentation
• the merged branch-specific regressions around provider resolution and tool-history message conversion have been addressed locally in this repo, with companion package work expected where shared package behavior is involved

Branch history included in this PR

This PR contains these branch-level steps:

• c4bbbfc feat: multi-account integrations, MCP improvements, ecosystem docs, and comprehensive QA test plans
• 2d76d65 merge of feat/integration-refactor into dev
• a216b36 merge of codex/runtime-memory-alignment into dev
• 234ed7f Fix integration path repositories in composer.lock
• 50370d1 Fix provider resolution on dev merge

Detailed change breakdown

1. Integration ecosystem refactor

This PR lands the app-side part of the larger integration ecosystem reorganization.

Key changes:

• Composer repository layout now points at the sibling integrations monorepo via:
  • ../integrations/core
  • ../integrations/packages/*
• the app now relies on integration-core contracts and shared infrastructure more directly
• integration settings and MCP server models gain multi-account-related behavior
• integration configuration flows in the API are expanded and refactored
• MCP tool registration and MCP tool-provider loading are reworked
• the tool system keeps moving away from one giant inline registry toward clearer grouped/provider-based registration
• Lua catalog/doc generation is aligned with the shared integration ecosystem direction instead of being purely app-local

Why this matters:

• it reduces duplicated integration runtime logic across repos
• it creates a cleaner ownership boundary between app code and integration-package code
• it makes future integration additions easier to maintain
• it enables multi-account integrations instead of assuming one config per provider

Representative files:

• composer.json
• composer.lock
• app/Http/Controllers/Api/IntegrationController.php
• app/Models/IntegrationSetting.php
• app/Models/McpServer.php
• app/Services/IntegrationSettingCredentialResolver.php
• app/Services/Mcp/McpServerRegistrar.php
• app/Services/Mcp/McpToolProvider.php
• config/integrations.php
• routes/api.php
• database/migrations/2026_04_05_000001_add_multi_account_to_integration_settings.php

2. Runtime memory-alignment pipeline

This PR also lands the runtime memory work that makes prompt construction and context pressure handling much more explicit.

New memory/runtime pieces include:

• ContextBudget
• ContextPruner
• PromptFrameBuilder
• ToolResultDeduplicator
• OutputTruncator
• CompactionPlan
• CompactionMemoryExtractor

Existing memory services were updated to use that pipeline more coherently:

• ConversationCompactionService
• MemoryFlushService
• ModelContextRegistry
• OpenCompanyAgent
• related jobs and checkpoint listeners

Other support changes include:

• new memory config in config/memory.php
• conversation summary failure tracking migration
• updated tests for the new runtime helpers
• runtime docs and planning docs updated to reflect the new memory architecture

Why this matters:

• prompt construction is no longer a loose accumulation of runtime state
• context pressure handling is explicit instead of implicit
• oversized tool outputs can be truncated deliberately instead of poisoning later turns
• retry/replay flows have clearer pressure-management hooks
• compaction and summary logic have more observable structure

Representative files:

• app/Agents/OpenCompanyAgent.php
• app/Jobs/AgentRespondJob.php
• app/Jobs/ExecuteAgentTaskJob.php
• app/Jobs/RunAutomationJob.php
• app/Listeners/CheckpointToolCall.php
• app/Services/Memory/*
• config/memory.php
• database/migrations/2026_04_09_120000_add_compaction_failure_tracking_to_conversation_summaries_table.php
• tests/Unit/ContextPrunerTest.php
• tests/Unit/OutputTruncatorTest.php
• tests/Unit/PromptFrameBuilderTest.php
• tests/Unit/ToolResultDeduplicatorTest.php

3. Provider and gateway runtime alignment

Because the two branches touched provider wiring from different angles, this PR also contains the merged runtime/provider alignment work.

What changed:

• AiManager provider wiring is overridden so providers run through prompt-cache-aware Prism gateway behavior
• custom provider gateways remain in place for relay-backed drivers and Codex
• DynamicProviderResolver was fixed after the merge so built-in providers such as anthropic and openai resolve through the built-in path before relay classification
• the merged branch’s composer.lock was refreshed so the integration path repositories point to the real ../integrations/... paths instead of the older tmp/integrations/... paths

Why this matters:

• without the lockfile fix, a clean composer install on the merged branch fails
• without the resolver precedence fix, standard providers can be mis-routed into relay-backed setup logic and fail with false configuration errors
• this is the minimum glue needed to make the two parent branches coexist on one branch without obvious runtime breakage

Representative files:

• app/Providers/AppServiceProvider.php
• app/Agents/Providers/DynamicProviderResolver.php
• app/Agents/Providers/GlmPrismGateway.php
• app/Agents/Providers/CodexPrismGateway.php
• composer.lock
• tests/Feature/DynamicProviderResolverTest.php

4. Tool-call and tool-result history regression follow-up

The runtime-memory branch exposed an important conversion problem in the Laravel AI -> Prism bridge used by the caching gateway path: assistant tool calls and tool-result messages were not being preserved the way the merged runtime now expects.

In this repo, that follow-up is represented by:

• the regression coverage in tests/Unit/PrismMessagesTest.php
• the app-side branch fix commit that keeps the merged branch coherent from the OpenCompany perspective

Why this matters:

• replay, checkpoint, and prompt-cache-aware continuation paths need tool execution history to survive message conversion
• losing that structure makes the agent runtime less reliable under retries and context reconstruction

Important note:

• the generic bridge behavior belongs in the shared package layer rather than permanently living as OpenCompany-only app code
• local validation for this merged branch assumes the corresponding prism-relay bridge behavior is aligned in the path dependency used during development
• that package-level ownership boundary is one reason this PR also updates the repo guidance files

5. Lua/runtime documentation alignment

This PR includes a broad documentation refresh around the integration ecosystem, runtime alignment, and Lua/tooling docs.

What changed:

• a large new docs/ecosystem/ tree was added
• integration package docs were added for multiple integrations and the shared core
• KosmoKrator comparison, audit, proposal, and research docs were imported for reference and reuse planning
• runtime alignment docs were added
• memory planning docs were heavily updated
• stale planning/todo artifacts and the old feature test map were removed
• Lua docs were refreshed

Why this matters:

• the codebase needed documentation that reflects the new ecosystem structure instead of the old one-off repo layout
• the runtime-memory work needed a written architecture anchor, not just code
• outdated planning/todo docs were creating false signals during audits and reviews

Representative files:

• docs/architecture/kosmokrator-reuse-audit.md
• docs/architecture/runtime-alignment-implementation-audit.md
• docs/ecosystem/**
• docs/planning/kosmokrator-runtime-alignment-checklist.md
• docs/planning/memory-implementation.md
• docs/testing/qa-strategy.md
• docs/external-channel-sync.md
• resources/lua-docs/_overview.md

6. Repo guidance cleanup

The repo guidance files were normalized and simplified.

What changed:

• AGENTS.md was added in mirrored form with CLAUDE.md
• both guidance files now explicitly call out package ownership rules
• both files are intentionally kept identical to reduce drift between agent instruction entrypoints
• low-signal and stale references were removed, including the old OC-1 investigation pointer
• the local/ngrok/MCP guidance is still present, but condensed

Why this matters:

• agents were too likely to fix shared package behavior inside app code by default
• duplicate but drifting guidance files create noisy context and inconsistent behavior
• a short mirrored ruleset is easier to maintain than two partially overlapping instruction files

Representative files:

• AGENTS.md
• CLAUDE.md

Conflict resolution and dev-branch follow-up work

This branch is not a raw merge of the two parent branches. It includes explicit merge conflict resolution and follow-up fixes made on dev.

The most important overlap points were:

• AI provider gateway wiring
• Lua API doc generation and integration account handling
• tool registry / integration-runtime touchpoints
• docs cleanup versus docs expansion

The dev branch then added two important follow-up fixes:

1. lockfile path repair so merged dependency installation works
2. provider-resolution precedence repair so built-in providers are not incorrectly treated as relay-only providers

Validation

What I validated on the merged branch:

• targeted PHPUnit regression coverage for the merged-branch fixes

Executed:

• vendor/bin/phpunit tests/Unit/PrismMessagesTest.php tests/Feature/DynamicProviderResolverTest.php

Result:

• OK (7 tests, 21 assertions)

Additional context:

• earlier merge validation also used focused syntax checks and branch-specific diagnosis around composer install behavior, provider resolution, and the runtime-memory regression surface
• this PR description is intentionally explicit about the follow-up fixes because the dev branch is the integrated result of those validations, not just the direct union of the two source branches

Risks and review guidance

This is a large integration PR. Review it by area rather than trying to read it as one flat diff.

Suggested review order:

1. dependency/repository layout and integration runtime ownership
2. provider/gateway wiring
3. memory runtime helpers and agent prompt construction
4. controllers/models/config for multi-account integrations
5. docs and cleanup

Specific risks to keep in mind:

• dependency layout assumes the sibling ../integrations monorepo structure
• some runtime expectations depend on companion package alignment where behavior is properly package-owned rather than app-owned
• docs volume is large in this PR because the ecosystem reference material was intentionally imported together with the code restructuring

Post-merge impact

If merged, main gains:

• the current integration ecosystem structure
• the current runtime memory-alignment work
• the merged-branch fixes needed to make those changes coexist
• a single forward branch that replaces the older isolated PRs

Superseded PRs

This PR is intended to replace and close:

• feat: Integration ecosystem refactor — monorepo, shared core, multi-account #4
• [codex] Implement runtime memory alignment pipeline #5