Background
The aem-commerce-prerender app uses an AEM Edge Delivery Services spreadsheet index (published-products-index) as its source of truth for which product pages have been published. This index is used in two critical workflows:
- Cleanup / unpublish (
mark-up-clean-up): finds products in the index that no longer exist in the catalog, then unpublishes and deletes their markup
- Deletion detection (
check-product-changes/poller.js): identifies published products that have been removed from the catalog
Problems Discovered at Scale (500k SKU test)
1. Hard 50k-path limit per index
AEM Edge Delivery Services indices have a hard limit of 50k indexed paths. An index with >50k paths will silently stop indexing new paths. During our test, the index grew to 258k entries due to a platform bug in the limiter — this is not reliable behavior.
2. Index file is too large to download at scale
At 258k entries, the compressed JSON is already ~2.6MB. The file cannot be reliably fetched by client applications at this size. The app's requestPublishedProductsIndex() paginates in 1000-item increments but the file size itself becomes a bottleneck.
3. Index misses paths published before the index definition was created
Any product published before the index configuration existed will not appear in the index, creating silent gaps that the cleanup workflow cannot detect.
4. No practical path to recovery at scale
Bulk reindex is limited to batches of 1k paths and cannot reindex prefixes that yield >50k results. For a 500k-product catalog, backfilling a single index is not feasible.
Potential Directions
- Use App Builder State (already tracks every rendered product with SKU, timestamp, and markup hash) as the source of truth for cleanup and deletion detection
- Use the Commerce Catalog API directly (already queried by
mark-up-clean-up) to detect deletions without an external index dependency
- Shard into multiple indexes by URL prefix so each stays under 50k entries
References
Background
The
aem-commerce-prerenderapp uses an AEM Edge Delivery Services spreadsheet index (published-products-index) as its source of truth for which product pages have been published. This index is used in two critical workflows:mark-up-clean-up): finds products in the index that no longer exist in the catalog, then unpublishes and deletes their markupcheck-product-changes/poller.js): identifies published products that have been removed from the catalogProblems Discovered at Scale (500k SKU test)
1. Hard 50k-path limit per index
AEM Edge Delivery Services indices have a hard limit of 50k indexed paths. An index with >50k paths will silently stop indexing new paths. During our test, the index grew to 258k entries due to a platform bug in the limiter — this is not reliable behavior.
2. Index file is too large to download at scale
At 258k entries, the compressed JSON is already ~2.6MB. The file cannot be reliably fetched by client applications at this size. The app's
requestPublishedProductsIndex()paginates in 1000-item increments but the file size itself becomes a bottleneck.3. Index misses paths published before the index definition was created
Any product published before the index configuration existed will not appear in the index, creating silent gaps that the cleanup workflow cannot detect.
4. No practical path to recovery at scale
Bulk reindex is limited to batches of 1k paths and cannot reindex prefixes that yield >50k results. For a 500k-product catalog, backfilling a single index is not feasible.
Potential Directions
mark-up-clean-up) to detect deletions without an external index dependencyReferences
requestPublishedProductsIndex():actions/utils.js:247actions/mark-up-clean-up/index.js:65actions/check-product-changes/poller.js:137