A semantic image search engine over the Amazon Products 2023 dataset, powered by local CLIP embeddings and background-removal inference — all running on local GPU via ONNX Runtime + DirectML (Also CUDA). No external APIs or cloud inference where used in this application.
Application is still in development and many things with be improved in future
- Setup pipeline — on first run, downloads models, tokenizer files, and the dataset from Hugging Face automatically.
- Data seeding — the dataset is parsed, converted from .arrow to JSON files, and inserted into MS SQL Server. For each product, the pipeline:
- Strips the background from the product image using RMBG-2.0 (background removal model), so embeddings focus on the product itself rather than noise and reduces vector size.
- Generates a CLIP image embedding from the cleaned image for image based search.
- Generates a CLIP text embedding from the product title for meaning based search. e.g: Apple Airpods, Samsung EarBuds -> Earphone, Headphone, ect...
- Stores metadata (title, price, category, etc.) in MS SQL Server.
- Stores both vectors in Qdrant.
- Search — at query time, your input (text or image) is embedded using the same CLIP models and compared against both vector collections. Results are scored and ranked using a simple weighted fusion:
| Signal | Weight |
|---|---|
| Visual (image vector similarity) | 60% |
| Text (title vector similarity) | 40% |
The top 10 products by combined score are returned.
| Model | Purpose | Source |
|---|---|---|
openai/clip-vit-large-patch14 — vision |
Image embedding | Hugging Face |
openai/clip-vit-large-patch14 — text |
Text embedding | Hugging Face |
briaai/RMBG-2.0 |
Background removal | Hugging Face |
All models run locally via ONNX Runtime with DirectML GPU acceleration.
- .NET 8 — ASP.NET Core Web API, layered architecture, EF Core
- ONNX Runtime + DirectML — local GPU inference (Works with CUDA too)
- MS SQL Server — product metadata storage
- Qdrant — vector similarity search (separate collections for image and text vectors)
- Hugging Face — model and dataset source
AMAZON-Products-2023 — a large-scale Amazon product catalog with titles, images, categories, and pricing metadata.