This repository contains cookbooks and examples demonstrating how to monitor and evaluate AI systems for hallucinations, retrieval quality, and other reliability issues using Quotient AI.
- Build and Monitor AI Agents: Track LangChain agents in real-time
- Evaluate Search Quality: Automatically detect when AI search results contain unsupported claims
- Improve AI Reliability: Get insights into common failure patterns and how to fix them
- Production Monitoring: Set up automated monitoring for your AI applications
Visit app.quotientai.co, sign up for a free account, and grab an API key from Settings. Quotient is completely free to get started! Check out the pricing page for details on free tier limits and paid plans.
| Notebook | Description | Open | Resources |
|---|---|---|---|
| Build a LIve Web Documentation Q&A Agent with Qdrant | This notebook demonstrates how to build a documentation QA system using Qdrant for vector storage, Tavily for web crawling, and Quotient for monitoring answer quality and hallucinations. | Open Notebook | Qdrant, Tavily, OpenAI, Langchain, Quotient |
| Evaluate AI Search Quality with Tavily | This notebook demonstrates how to use Quotient to detect hallucinations and document relevancy in search results using Tavily. | Open Notebook | Tavily, Quotient |
| Build a Company Research Tool with Linkup | This notebook demonstrates how to use Linkup's AI search capabilities to research companies while monitoring result quality with Quotient. | Open Notebook | Linkup, Quotient |
| Evaluate AI Search Quality with Exa | This notebook demonstrates how to use Quotient to detect hallucinations and document relevancy in search results using Exa /answer. |
Open Notebook | Exa, Quotient |
| Build a RAG Pipeline with Exa Search & OpenAI | This notebook demonstrates how to use Exa for web search, OpenAI for generating answers from search results, and Quotient for monitoring search quality and detecting hallucinations. | Open Notebook | Exa, OpenAI, Quotient |
| Build and Monitor a Web Research Agent | This notebook demonstrates how to use Quotient to monitor and evaluate a research agent that browses the web and answers questions using the Tavily API. | Open Notebook | Langchain, Tavily, OpenAI, Quotient |
| Build and Monitor an Exa Research Agent | This notebook demonstrates how to use Quotient to monitor and evaluate a research agent that leverages Exa's Python SDK for advanced web search and document retrieval. | Open Notebook | Langgraph, Exa, Anthropic, Quotient |
| Build a Multi-Agent Financial Research System with OpenAI & Quotient Tracing | This notebook demonstrates how to build a financial research system using multiple specialized agents with the OpenAI Agents SDK. The system is monitored using Quotient Tracing to provide visibility into the multi-agent workflow. | README | OpenAI, Quotient |
| Build an OSS Search Engine with Firecrawl, Groq & Quotient | A fork of Fireplexity enhanced with Quotient monitoring to detect hallucinations and evaluate context relevance in AI search results. | README | Firecrawl, Groq, Quotient |
| Build a Financial Analysis Agent with Quotient Traces | A production-ready financial analysis agent that demonstrates real-time stock data analysis with comprehensive tracing, hallucination detection, and document relevance monitoring. | README | LangChain, OpenAI, Quotient |
| Evaluate Tool Use with Limbic | This notebook demonstrates how to evaluate Language Models' ability to use tools correctly using the Limbic Tool-Use Benchmark dataset. | Open Notebook | Together AI, Limbic, Hugging Face |
This repository contains research and examples for AI reliability. Feel free to:
- Run the notebooks and share your results
- Report issues or suggest improvements
- Contribute new examples or use cases
You can reach the Quotient team at [email protected]