Fix OpenSearch duplicate/multi-generation alias bugs; harden index lifecycle management

[P4sca1]

Summary

OpenSearch aliases route writes/reads to whichever generation currently owns them, with no cross-generation _id awareness. Any index that both (a) rolls over and (b) is updated in place by re-writing the same _id (rather than being pure append-only) will eventually split a logical record across two generations, or worse, leave its alias pointing at more than one index - which OpenSearch refuses to run single-document operations against:

status: 400, type: illegal_argument_exception, reason: alias [shuffle_org_statistics] has more than
one index associated with it [shuffle_org_statistics-000001, shuffle_org_statistics-000002],
can't execute a single index op

workflowexecution, notifications, and org_statistics were all rollover-enabled and keyed-write stores, and hit exactly this - org_statistics is a real, observed production error, not a hypothetical.

This PR closes that gap for both new writes going forward and for any deployment that already has the problem today, with no manual intervention required. Along the way it also hardens and simplifies the index lifecycle code itself, since a lot of related logic had accumulated as ad-hoc pieces inside db-connector.go/health.go.

What changed

1. workflowexecution: hot/cold live + archive split

The highest-volume index gets its own dedicated write path instead of sharing rollover with everything else:

• workflowexecution_live - a single, non-rolling index. All in-flight execution updates land here, so in-place _id writes are always safe.
• workflowexecution (same name, new role) - becomes the archive: append-only and still rollover-managed. A background sweep moves executions here about an hour after they reach a terminal status, and re-archiving an already-archived id updates the existing copy instead of duplicating it.
• All execution readers (GetWorkflowExecution, GetAllWorkflowExecutions, GetUnfinishedExecutions, run counts/search, etc.) transparently query both indices and dedupe results.
• Existing deployments migrate automatically at startup with no bulk reindex and no downtime.

2. org_statistics / notifications: stop rolling stateful stores, self-heal existing deployments

These are keyed, stateful stores, not append logs, and are no longer rollover-enabled - each stays a single backing index going forward. For any deployment that already hit the multi-generation alias problem (or upgraded from a version that rolled these over), a new startup step automatically collapses all existing generations back into one, safely and without data loss, before detaching rollover so it can't happen again. This runs idempotently on every startup.

3. Index lifecycle management: reorganized and hardened

The index/mapping/rollover management code was split out of db-connector.go and health.go into two dedicated files (opensearch_indices.go for index/mapping definitions, opensearch_lifecycle.go for init/rollover/migration logic), and several real bugs found during hardening were fixed along the way:

• Legacy "double-prefixed" indices (an old bug that could leave e.g. shuffle_shuffle_notifications behind) are now detected and cleaned up automatically, scoped safely to the configured index prefix so this can't accidentally touch other tenants' indices on a shared cluster.
• Large, one-time legacy index migrations now run in the background instead of blocking backend startup, and are safe to interrupt/retry across restarts and multiple backend replicas.
• Freshly created indices immediately get correct field mappings (previously a fresh install could trigger unnecessary self-healing migrations on empty indices).
• Fixed a false-positive startup warning that could appear after the double-prefix cleanup above ran.
• Shard count, replica count, refresh interval, and rollover thresholds are no longer hardcoded - they're configurable via environment variables, with the previous values kept as defaults.

4. Notification retention (new, opt-in)

An optional background sweep that hard-deletes old, already-read/ignored notifications after a configurable number of days. Fully disabled by default.

Configuration

All new behavior is additive and off-by-default or safely defaulted; nothing here requires action to keep existing behavior. For full details, including OpenSearch permission requirements for the credential Shuffle uses (a few additional cluster-level grants are needed for the new background migration/task-polling behavior), see the docs. Relevant new environment variables:

Variable	Purpose	Default	Example
SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT	Skip all index/mapping management at startup (pre-existing flag)	unset (runs)	true
SHUFFLE_SKIP_EXECUTION_LIVE_MIGRATION	Skip the one-time migration of in-flight executions into workflowexecution_live	unset (runs)	true
SHUFFLE_SKIP_EXECUTION_ARCHIVAL_SWEEP	Skip the recurring live-to-archive sweep	unset (runs)	true
OPENSEARCH_EXECUTION_GRACE_PERIOD	How long after completion an execution stays "live" before archiving	1h	90m
OPENSEARCH_EXECUTION_ARCHIVE_SWEEP_INTERVAL	How often the archival sweep runs	30m	15m
OPENSEARCH_NOTIFICATION_RETENTION_DAYS	Enables notification retention; unset/0 keeps it off	0 (disabled)	30
OPENSEARCH_INDEX_ROLLOVER	Override the default rollover thresholds (age/size/doc count)	{"max_age":"90d","max_size":"40gb","max_docs":1000000}	{"max_age":"30d"}
OPENSEARCH_USE_ISM_ROLLOVER	Disable ISM-managed rollover if not desired	true	false
OPENSEARCH_ISM_POLICY_NAME	Override the default ISM policy name	shuffle-rollover	my-org-rollover
SHUFFLE_OPENSEARCH_INDEX_PREFIX	Existing multi-tenant index prefix, now also used to safely scope the double-prefix cleanup	unset (no prefix)	prod

Testing

Verified against a real OpenSearch cluster (fresh install, upgrade-from-existing-data, and multi-generation collision scenarios), including under an OpenSearch security-plugin role scoped to the minimum permissions documented for Shuffle, to confirm all migrations complete correctly and no data is lost or duplicated.

[frikky]

@yashsinghcodes 🔥

[P4sca1]

I merged latest main into this branch and made a few more improvements:

• extracted opensearch specific functionality to dedicated files for improved readability
• consistent naming (indices instead of indexes) to follow opensearch api naming
• async index rollover in the background using opensearch tasks api, instead of sync rollover on backend start, which might take a few hours

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​klauspost/​compress@​v1.19.2

View full report

[P4sca1]

Tried these changes on a live cluster with around 200GB of workflowexecution data.
I experienced an error during reindexing that stemmed from opensearch date detection. OpenSearch mapped a value that looked like a date to the date type, causing subsequent documents to fail.
This issue was fixed by disabled the date detection feature explicitly.