VLLM OSS Proxy Service

A production-ready proxy service that bridges OpenAI API compatibility with VLLM (vLLM) servers running large language models, specifically designed to support tool calling and function invocation workflows.

🎯 Purpose

This proxy solves compatibility issues between OpenAI-format API requests (used by n8n workflows and other applications) and VLLM's native response format, enabling seamless integration of large language models in production environments.

Key Problems Solved

Tool Calling Compatibility: Converts OpenAI tool calling format to VLLM's expected format
Multi-turn Conversations: Maintains conversation context across multiple requests
Streaming Support: Provides real-time Server-Sent Events for responsive applications
Production Reliability: Includes rate limiting, authentication, monitoring, and error handling

🚀 Features

Core Functionality

✅ Full OpenAI API v1 Compatibility - /v1/chat/completions, /v1/models, /v1/completions
✅ Tool Calling Support - Function calling with proper format conversion
✅ Multi-turn Dialogs - Conversation state management with Redis
✅ Streaming Responses - Server-Sent Events (SSE) support
✅ Request/Response Logging - Comprehensive activity tracking

Production Features

🛡️ Authentication - API key validation middleware
⚡ Rate Limiting - Configurable request throttling
📊 Monitoring - Prometheus metrics and Grafana dashboards
🔄 Health Checks - Kubernetes/Docker ready endpoints
🚧 Error Handling - Graceful degradation and retry logic
💾 State Management - Redis-backed conversation persistence

🏗️ Architecture

graph TB
    subgraph "Client Applications"
        A[n8n Workflows]
        B[OpenAI-compatible Apps]
    end

    subgraph "Proxy Service"
        C[FastAPI Router]
        D[Format Converter]
        E[State Manager]
        F[Redis Cache]
    end

    subgraph "VLLM Backend"
        G[VLLM Server]
        H[LLM Model]
    end

    A --> C
    B --> C
    C --> D
    D --> E
    E --> F
    D --> G
    G --> H

📋 Requirements

Python 3.11+
Redis (for conversation state)
Docker & Docker Compose (recommended deployment)
VLLM Server running with OpenAI-compatible endpoints

⚡ Quick Start

1. Clone and Setup

git clone https://github.com/miolamio/oss-vllm-proxy.git
cd oss-vllm-proxy

# Copy environment template
cp .env.example .env

2. Configure Environment

Edit .env file with your settings:

# VLLM Configuration
VLLM_BASE_URL=http://your-vllm-server:8000
VLLM_API_KEY=your-vllm-api-key

# Security
PROXY_API_KEY=your-secure-proxy-key

# Redis
REDIS_URL=redis://localhost:6379

3. Run with Docker Compose

docker-compose -f docker/docker-compose.yml up -d

4. Test the Service

curl -X POST http://localhost:5000/v1/chat/completions \
  -H "Authorization: Bearer your-proxy-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

🛠️ Development Setup

Local Development

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Run development server
make dev

Run Tests

# Unit tests
pytest tests/

# Integration tests
pytest tests/integration/

# With coverage
make test-coverage

📊 Monitoring

Access monitoring dashboards:

Prometheus: http://localhost:9090
Grafana: http://localhost:3000 (admin/password from .env)
Proxy Health: http://localhost:5000/health/live

🔧 Configuration

Environment Variables

Variable	Description	Default
`VLLM_BASE_URL`	VLLM server URL	`http://localhost:8000`
`VLLM_API_KEY`	VLLM API key	`None`
`PROXY_API_KEY`	Proxy authentication key	`None`
`REDIS_URL`	Redis connection string	`redis://localhost:6379`
`LOG_LEVEL`	Logging level	`INFO`
`ENABLE_METRICS`	Enable Prometheus metrics	`true`
`RATE_LIMIT_REQUESTS`	Rate limit per window	`1000`
`RATE_LIMIT_WINDOW`	Rate limit window (seconds)	`60`

Advanced Configuration

See src/utils/config.py for all configuration options.

🔀 API Conversion Examples

Tool Calling Format Conversion

Input (OpenAI format):

{
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get weather info",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {"type": "string"}
        }
      }
    }
  }]
}

Output (VLLM format):

{
  "tools": [{
    "type": "function",
    "name": "get_weather",
    "description": "Get weather info",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {"type": "string"}
      }
    }
  }]
}

🐳 Deployment

Docker Compose (Recommended)

# Production deployment
docker-compose -f docker/docker-compose.yml up -d

# Check logs
docker-compose logs -f proxy

Kubernetes

Kubernetes manifests available in k8s/ directory:

kubectl apply -f k8s/

🧪 Testing

Test Tool Calling

python scripts/test_tool_calls.py

Performance Benchmarking

python scripts/benchmark.py

🤝 n8n Integration

This proxy is optimized for n8n workflows:

Add HTTP Request Node
Set URL: http://your-proxy-url:5000/v1/chat/completions
Add Authorization Header: Bearer your-proxy-api-key
Configure Tool Functions in request body

See docs/n8n-integration.md for detailed setup.

📚 Documentation

🔍 Troubleshooting

Common Issues

Tool calls not working: Check VLLM model supports function calling
Connection refused: Verify VLLM server is running and accessible
Authentication errors: Confirm API keys are correctly set

Debug Mode

# Enable debug logging
export LOG_LEVEL=DEBUG
docker-compose up

Health Checks

# Check proxy health
curl http://localhost:5000/health/live

# Check VLLM connection
curl http://localhost:5000/health/vllm

📈 Performance

Latency Overhead: <50ms
Concurrent Requests: 100+ supported
Memory Usage: <512MB typical
Throughput: Depends on VLLM backend performance

🛡️ Security

✅ API key authentication
✅ Rate limiting protection
✅ Input validation and sanitization
✅ No credential exposure in logs
✅ Secure defaults configuration

🤝 Contributing

Fork the repository
Create feature branch (git checkout -b feature/amazing-feature)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open Pull Request

Development Guidelines

Follow PEP 8 style guide
Add tests for new features
Update documentation
Ensure all tests pass

📄 License

This project is licensed under the MIT License - see LICENSE file for details.

🙏 Acknowledgments

VLLM Team for the excellent inference server
FastAPI for the web framework
n8n Community for workflow automation inspiration

📞 Support

GitHub Issues: Create an issue
Documentation: Check the docs/ directory
Community: Join discussions in GitHub Discussions

Made with ❤️ for the open source AI community

Name		Name	Last commit message	Last commit date
config		config
docker		docker
docs		docs
scripts		scripts
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
SECURITY_CLEANUP_REPORT.md		SECURITY_CLEANUP_REPORT.md
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation