Scale image processing pipeline for 2B+ images #18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Update Ray base image to 2.51.1 and vLLM to 0.11.0 - Add boto3 dependency for S3 operations - Update transformers to 4.57.1 for compatibility - Configure compute resources with auto-selection (max 520 CPU, 128 GPU) - Add disk size configuration options for customer-hosted deployments - Implement robust URL validation and error handling - Add base64 image encoding for Arrow serialization - Add JPEG format validation and 128x128 image resizing - Scale model replicas from 1 to 32 for higher throughput - Optimize batch sizes and memory usage for large-scale processing - Implement session pooling for HTTP requests with retry logic - Add timestamp-based output paths to /mnt/shared_storage - Add run.sh script for job submission with HF_TOKEN
- Switch from synchronous requests to async aiohttp for better performance - Update num_model_replicas to 96 for higher throughput - Implement concurrent image downloading with semaphore control - Add async image processing with proper event loop handling - Update compute config: CPU max to 1024, GPU min to 96, GPU max to 128 - Add RAY_DEFAULT_OBJECT_STORE_MEMORY_PROPORTION env var - Simplify pipeline by removing base64 encoding intermediate step - Process images as bytes directly in vision preprocessing
- Add detailed overview of the 3-stage pipeline (download, preprocess, inference) - Document key features: scale, throughput, fault tolerance, cost optimization - Provide clear setup instructions with HuggingFace token requirements - Include configuration details for compute, model, and batch processing - Use human-friendly tone for Anyscale users
- Update max CPU limit from 1024 to 530 in job.yaml and README - Update max GPU limit from 128 to 48, set min GPU to 0 for better resource allocation - Change from fixed num_model_replicas to dynamic min/max GPU range (32-96) - Use concurrency tuple (min_gpu_num, max_gpu_num) for autoscaling - Simplify async event loop handling (remove run_async_in_thread wrapper) - Update batch sizes: 100 -> 50 for better memory management - Add concurrency=1024 to image_download for higher throughput - Fix code formatting (quotes, spacing, line endings)
- Add detailed docstring explaining pipeline architecture and goals - Document GPU autoscaling configuration with best practice notes - Annotate async I/O patterns for image downloading (semaphores, connection pooling) - Explain ThreadPoolExecutor usage for CPU-bound image processing - Document vLLM configuration with tuning recommendations - Add inline comments explaining Ray Data performance tuning - Provide reasoning for resource allocation decisions (CPU, memory, GPU) - Include best practice notes for batch processing, error handling, and filtering - Add comments explaining why certain patterns are used (fail fast, graceful degradation) - Make code suitable as educational reference for building large-scale pipelines
- Remove extensive inline documentation and docstrings - Add concise 5-step pipeline overview comment - Keep code clean and readable without excessive comments - All documentation is now in README.md for better maintainability
- Pin vLLM concurrency to 64 L4 replicas to match g6.xlarge worker pool - Configure explicit g6.xlarge worker nodes (min 0, max 64) in job.yaml - Switch accelerator_type from A10G to L4 for proper GPU targeting - Reduce max_num_batched_tokens from 2048 to 1024 for L4 memory constraints - Update README to reflect L4-based scaling (up to 64 replicas) - Reduce num_cpus from 1 to 0.5 for process_image_bytes batch processing
- Pin worker nodes to g6e.12xlarge (8 L40S GPUs each, max 8 nodes = 64 GPUs) - Update max GPU from 48 to 32 for better resource allocation - Set max CPU to 384 to match 8 worker nodes - Configure vLLM to use L40S accelerator type - Update vLLM concurrency to 32 for L40S replicas - Adjust batch processing concurrency for optimal throughput
xyuzh
commented
Nov 19, 2025
| loop = asyncio.new_event_loop() | ||
| try: | ||
| results = loop.run_until_complete(download_images_async(urls)) | ||
| finally: |
Author
There was a problem hiding this comment.
I am using a verbose approach for the async download logic but I found the aysnc support for the map_batches at https://github.com/ray-project/ray/pull/46129/files
@robertnishihara do you recommend the async approach mentioned over the PR?
xyuzh
commented
Nov 19, 2025
| ), | ||
| ), | ||
| batch_size=128, | ||
| max_concurrent_batches=128, |
Author
There was a problem hiding this comment.
changing the batch_size and max_concurrent_batches didn't change the GPU memory usage and VLM throughput
| compute_config: | ||
| # Pin worker nodes to g6e.12xlarge so the vision workload lands on L40S GPUs. | ||
| worker_nodes: | ||
| - instance_type: g6e.12xlarge |
Author
There was a problem hiding this comment.
This instance type has 4 GPUs in a node, use 1 GPU node instance type would produce error after sometime of running
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR scales the image processing pipeline to handle 2B+ images efficiently.
Key Changes
Infrastructure Updates
Scalability Improvements
Robustness Enhancements
Configuration
New Files
run.shscript for easy job submission with HF_TOKEN environment variableTesting
Tested with large-scale image processing workloads on distributed Ray clusters.