Skip to content

feat(job): move metadata storage to a database - #1307

Open
berstpander wants to merge 9 commits into
alibaba:masterfrom
berstpander:codex/job-metadata-database
Open

feat(job): move metadata storage to a database#1307
berstpander wants to merge 9 commits into
alibaba:masterfrom
berstpander:codex/job-metadata-database

Conversation

@berstpander

Copy link
Copy Markdown
Contributor

Summary

Move Job and Group metadata from OSS to a database-backed SDK and decouple the local CLI from persisted metadata.

Closes #1306

Changes

Metadata SDK

  • Add database models for Group and Job metadata.
  • Use UUID group_id and job_id as canonical identifiers.
  • Add transactional create and update operations.
  • Support complete Job lookup by job_id.
  • Support namespace Group listing.
  • Support Group Job filtering by status, category, task, and Job type.
  • Add cursor pagination, Group statistics, and combined Group details.
  • Accept a caller-owned SQLAlchemy session factory so connection credentials remain in the trusted metadata service.

Job identity

  • Generate an independent UUID for every Job.
  • Allow callers to provide an existing job_id.
  • Propagate the same ID through SDK results, labels, environment variables, and JSONL events.
  • Remove process-local run_id behavior.

CLI

  • Remove dependencies on persisted metadata repositories.
  • Keep --task, --tasks, and --all.
  • Add optional --job-id for single-job execution.
  • Return the effective job_id in JSONL Job lifecycle events.
  • Replace metadata-based resume with explicit sandbox, process, and session parameters.
  • Keep OSS access for dataset discovery and execution artifacts.

Compatibility Notes

  • Remove legacy Run metadata commands and Run ID lookup.
  • Remove ROCK_RUN_ID, rock_run_id, and JSONL run_id.
  • OSS remains responsible for datasets and execution artifacts.
  • The local CLI does not access the metadata database.

Testing

  • SDK Job: 329 passed
  • CLI Job: 35 passed
  • Legacy SDK Agent Job: 27 passed
  • Ruff and format checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Improve Job and Group metadata persistence

1 participant