Replace the global function deployment scheduler with regional schedulers

[sanjay-saxena]

Problem

Today, a single global scheduler leader processes every function deployment across all regions. As the deployment count grows, this increases reconciliation latency and prevents scheduler capacity from scaling with the number of regions.

Proposed change

Replace the single global function deployment scheduler with multiple regional schedulers. Each configured region will have one active scheduler leader and will process a deterministic subset of deployments.

Add phase 1 regional sharding for function deployment reconciliation:

• Run one ShedLock leader per configured region by including the current region in the lock name.
• Configure the same ordered reconciliation-region list in every region.
• Assign each function deployment to exactly one region using a deterministic hash of its function version ID.
• Scan all deployment rows in each region, then process only the rows owned by that region.
• Keep Cassandra stream consumption on the scheduler thread and close the stream before submitting work to the executor.
• Move deployment processing into a dedicated FunctionDeploymentsTask so ScheduledTaskService owns only scheduling, locking, and application-ready coordination.
• Keep reconciliation and graceful deployment cleanup in function-domain services.
• Log the number of processed deployments and the complete task start time, end time, and duration.
• Trace the complete regional scheduler run and identify the reconciliation region on deployment spans.

Acceptance criteria

• The system uses multiple regional scheduler leaders instead of one global scheduler leader.
• Only one pod in each configured region runs function deployment reconciliation at a time.
• Every deployment maps to exactly one configured region.
• Ownership is deterministic for the same function version ID and ordered region list.
• An invalid current region or duplicate configured region fails during startup.
• Only region-owned deployments are submitted for processing.
• The scheduler waits for all submitted deployment work before completing its span and releasing the lock.
• Existing function status handling and deployment cleanup behavior remain unchanged.
• Unit and service integration tests pass.

Follow-up

Phase 2 can add a queryable scheduler shard column or index so each region retrieves only its assigned deployments instead of scanning the full table.