Skip to content

README.md

Latest commit

 

History

History
244 lines (189 loc) · 10.6 KB

README.md

File metadata and controls

244 lines (189 loc) · 10.6 KB
Raw

DeepImageSearch Web Demo

A FastAPI + React web demo that lets you browse a photo library, search photos with natural language, and watch the AI Agent's reasoning process in real time.

How It Differs from the ImageSeeker CLI

You may already be familiar with ImageSeeker/src/demo.py:

# ImageSeeker CLI (command-line args, full environment required)
python -m src.demo \
  --dataset-path ../DISBench/ \
  --retriever Qwen/Qwen3-VL-Embedding-8B \
  --agent openai/gpt-4o \
  --user-id "15352839@N00" \
  --query "Find all photos of sunsets"

Key differences:

ImageSeeker CLI (src/demo.py) Web Demo (demo/server.py)
Launch python -m src.demo --args... uvicorn demo.server:app
Configuration CLI flags (--agent, --retriever) Environment variables (DEMO_AGENT, DEMO_RETRIEVER)
Data source Requires DISBench dataset (--dataset-path) Any image folder (DEMO_IMAGE_DIR)
User selection Must specify --user-id Loads the entire image directory automatically
Querying One-shot --query "..." Interactive browser input, unlimited searches
Output Terminal text Browser UI (animations, photo lightbox, reasoning steps)
Mock mode None; requires Retriever + LLM Yes; a single command shows the full UI
Prerequisites vLLM + FAISS index + LLM API key Mock mode: none

The key takeaway: the CLI requires completing all preparation steps in the ImageSeeker README (building FAISS indices, deploying the vLLM Retriever, configuring API keys), while the Web Demo's Mock mode runs with nothing but an image folder.

Quick Start (3 Steps)

Step 1: Install Dependencies

# Backend
cd demo
pip install -r requirements.txt

# Frontend build (one-time)
cd frontend
npm install && npm run build
cd ../..

Step 2: Configure .env

Create a .env file in the demo/ directory (a template is provided):

# === Web Demo Configuration ===
DEMO_IMAGE_DIR=DISBench/images/15352839@N00   # Image directory (required)
DEMO_MOCK=1                                    # Mock mode (no external services needed)

# === Only needed for Live mode (skip for Mock) ===
# DEMO_METADATA_PATH=DISBench/metadata/15352839@N00.jsonl
# DEMO_RETRIEVER=Qwen/Qwen3-VL-Embedding-8B
# DEMO_AGENT=openai/gpt-4o

# === ImageSeeker configuration (Live mode) ===
# OPENAI_API_KEY=sk-...
# SEARCH_API_URL_MAP={"Qwen/Qwen3-VL-Embedding-8B": "http://localhost:51282/global_user_search"}

The .env file lives at demo/.env and is loaded automatically regardless of your working directory.

You can also point to any image folder instead of DISBench:

DEMO_IMAGE_DIR=my_photos    # Supports jpg/jpeg/png/webp/bmp/gif

Step 3: Launch

# Run from the project root (auto-reads .env)
uvicorn demo.server:app --port 8000

Open http://localhost:8000 in your browser to see:

  1. A photo library grouped by date
  2. A search bar at the top for AI-powered search
  3. Click a suggestion or type any query to watch the Agent's animated reasoning process

Note: In Mock mode, search results are random and do not reflect your query. Its purpose is to showcase the UI interactions.

Live Mode (Real AI Search)

Live mode runs the actual ImageSeeker Agent, so the prerequisites are the same as the CLI:

Prerequisites (Same as ImageSeeker)

  1. FAISS index built — follow "Build the retriever index" in ImageSeeker/README.md
  2. Retriever service deployed — bash scripts/deploy_vllm_retriever.sh ... running with "Application startup complete."
  3. .env configured — with OPENAI_API_KEY (or other LLM key) and SEARCH_API_URL_MAP
  4. ImageSeeker dependencies installed — cd ImageSeeker && pip install -r requirements.txt

Launch Live Mode

Update your demo/.env (set DEMO_MOCK=0, add Retriever and metadata paths):

# demo/.env
DEMO_IMAGE_DIR=DISBench/images/15352839@N00
DEMO_METADATA_PATH=DISBench/metadata/15352839@N00.jsonl
DEMO_MOCK=0
DEMO_RETRIEVER=Qwen/Qwen3-VL-Embedding-8B
DEMO_AGENT=openai/gpt-4o

# ImageSeeker configuration
OPENAI_API_KEY=sk-...
SEARCH_API_URL_MAP={"Qwen/Qwen3-VL-Embedding-8B": "http://localhost:51282/global_user_search"}

Then launch (auto-reads .env):

uvicorn demo.server:app --port 8000

Using Your Own Photos (Non-DISBench)

To use your own photos instead of the DISBench dataset:

  1. Place photos in a folder, e.g., my_photos/
  2. Create a metadata file my_photos_metadata.jsonl with one JSON object per line:
{"photo_id": "IMG_001", "taken_time": "2024-06-15 14:30:00", "address": "United States, California, San Francisco"}
{"photo_id": "IMG_002", "taken_time": "2024-07-20 09:15:00", "address": "Japan, Tokyo, Shibuya"}

Where photo_id matches the filename (without extension), and taken_time/address are used by the Agent's metadata tools.

  1. Build a FAISS index and deploy the Retriever for your photos (the most complex step; see the ImageSeeker README)
  2. Configure .env accordingly and run uvicorn demo.server:app --port 8000

Environment Variables Reference

Variable Required Default Description
DEMO_IMAGE_DIR Yes - Image directory path (e.g., DISBench/images/15352839@N00)
DEMO_MOCK No false Set to 1 to enable Mock mode (no external services needed)
DEMO_METADATA_PATH Live mode - Metadata JSONL file path
DEMO_RETRIEVER Live mode - Retriever model name (e.g., Qwen/Qwen3-VL-Embedding-8B)
DEMO_AGENT No openai/gpt-4o Agent LLM model
DEMO_MEMORYMANAGER No openai/gpt-4o-mini Memory compression model
DEMO_MAX_TURNS No 30 Maximum Agent iteration turns
DEMO_MAX_CONTEXT_TOKENS No 65536 Context token budget
DEMO_WITHOUT No "" Comma-separated list of tools to exclude
DEMO_USER_ID No basename of DEMO_IMAGE_DIR Override the user_id inferred from DEMO_IMAGE_DIR

CLI parameter mapping:

CLI Flag Environment Variable Notes
--dataset-path N/A Web Demo splits this into DEMO_IMAGE_DIR + DEMO_METADATA_PATH
--user-id N/A Web Demo loads the entire directory; no user-id needed
--retriever DEMO_RETRIEVER Same meaning
--agent DEMO_AGENT Same meaning
--query N/A Web Demo accepts queries in the browser
--max-turns DEMO_MAX_TURNS Same meaning
--mock DEMO_MOCK Web Demo only; no CLI equivalent

Frontend Development Mode

To modify the frontend with hot-reload:

# Terminal 1: Start the backend (serves API)
DEMO_IMAGE_DIR=my_photos DEMO_MOCK=1 uvicorn demo.server:app --port 8000

# Terminal 2: Start the frontend dev server (Vite with HMR)
cd demo/frontend
npm run dev
# Visit http://localhost:5173 (auto-proxies /api requests to port 8000)

Directory Structure

demo/
├── server.py                 # FastAPI backend (SSE search stream, image serving, library pagination API)
├── mock.py                   # Mock Agent (simulates a 3-turn reasoning flow, no external services)
├── requirements.txt          # Python deps: fastapi, uvicorn, sse-starlette, langchain-core, langchain-openai, langgraph
├── __init__.py
└── frontend/                 # React frontend
    ├── index.html
    ├── package.json          # React 19, Zustand 5, Framer Motion 11, Tailwind CSS 4
    ├── vite.config.ts        # Dev proxy: /api → localhost:8000
    └── src/
        ├── main.tsx          # Entry point
        ├── App.tsx           # Root component
        ├── index.css         # Design tokens + light/dark themes + animations
        ├── api/
        │   ├── types.ts      # SSEEvent, Step, DateGroup, etc.
        │   └── sse.ts        # SSE client
        ├── stores/
        │   ├── galleryStore.ts   # Gallery state (groups, pagination, search toggle)
        │   └── searchStore.ts    # Search state (steps, results, theme)
        ├── lib/
        │   ├── animations.ts     # Framer Motion animation variants
        │   └── cn.ts
        └── components/
            ├── NavBar.tsx            # Navigation bar (branding + AI search entry)
            ├── ThemeToggle.tsx       # Light/dark theme toggle
            ├── SearchOverlay.tsx     # Fullscreen search overlay
            ├── SuggestionPills.tsx   # Search suggestions (8 preset queries)
            ├── SearchView.tsx        # Search results (left: steps, right: photos)
            ├── StepTimeline.tsx      # Agent step timeline
            ├── StepItem.tsx          # Single step (purple=reasoning, cyan=tool, green=photos)
            ├── ResultsHero.tsx       # Search results display
            ├── GalleryView.tsx       # Photo gallery (infinite scroll)
            ├── DateSection.tsx       # Date group (adaptive grid)
            ├── DateHeader.tsx        # Date header bar (date + location)
            ├── GalleryTile.tsx       # Gallery photo thumbnail
            ├── GridDensityControl.tsx # Grid density control
            ├── PhotoCanvas.tsx       # Photo canvas renderer
            ├── PhotoCard.tsx         # Search result photo card (with match score)
            ├── PhotoGrid.tsx         # Photo grid layout
            └── Lightbox.tsx          # Fullscreen photo lightbox

Design Notes

UI Design

  • Tech-forward theme: indigo/purple/cyan gradients, frosted-glass nav bar, dark mode by default
  • AI search prominence: large centered search entry in the nav bar with a gradient "AI" badge
  • Reasoning visualization: each Agent step rendered in a three-color scheme (purple = reasoning, cyan = tool call, green = photos found); photos display match scores and are clickable for full-size viewing
  • Responsive: adapts to mobile, tablet, desktop, and ultrawide screens
  • Light/dark toggle: global theme switch via CSS custom properties

Backend Architecture

  • SSE streaming: each reasoning step (reasoning → tool_call → tool_result → photos → result) is pushed to the frontend in real time
  • Library pagination: /api/library returns date-grouped photos in reverse chronological order with cursor pagination; the frontend uses IntersectionObserver for infinite scrolling
  • Image serving: /api/photos/{id} serves local image files directly with cache headers
  • Live mode: calls ImageSeeker/src/agent/graph.py's build_graph() + graph.astream() directly, sharing the exact same Agent core as the CLI