Skip to content
This repository was archived by the owner on Sep 18, 2026. It is now read-only.

Commit 45f9702

Browse files
committed
Initial Commit
0 parents  commit 45f9702

859 files changed

Lines changed: 865651 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
.git
2+
.gitignore
3+
.venv
4+
venv
5+
__pycache__/
6+
*.py[cod]
7+
.pytest_cache
8+
.ruff_cache
9+
.pyright
10+
htmlcov
11+
.coverage
12+
.DS_Store
13+
dist
14+
build
15+
*.egg-info
16+
.env
17+
.cache
18+
.rapidocr_models
19+
.docker
20+
dataset
21+
data
22+
tests
23+
.sdd
24+
.deepeval

‎.env.example‎

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
# Copy this file to `.env` for the shared local baseline.
2+
# Use `.env.local` for machine-specific overrides; it overrides `.env`.
3+
# In Docker Compose, service containers also load `.env` then `.env.local`.
4+
# If you need a one-off Compose override file, set `ENV_FILE=/abs/path/to/file`
5+
# and it will be layered after `.env.local`.
6+
7+
# =========================
8+
# Competition Platform
9+
# =========================
10+
EVAL_API_KEY=
11+
EVAL_BASE_URL=https://platform.agentic-challenge.ai/api/v1
12+
EVAL_PHASE=warmup
13+
EVAL_WORK_DIR=platform_runs
14+
EVAL_DOCUMENTS_DIRNAME=documents
15+
EVAL_QUESTIONS_FILENAME=questions.json
16+
EVAL_SUBMISSION_FILENAME=submission.json
17+
EVAL_CODE_ARCHIVE_FILENAME=code_archive.zip
18+
EVAL_ARCHIVE_ALLOWLIST_PATH=src/rag_challenge/submission/archive_allowlist.json
19+
EVAL_COLLECTION_PREFIX=legal_chunks_platform
20+
EVAL_QUERY_CONCURRENCY=2
21+
EVAL_POLL_INTERVAL_S=10.0
22+
EVAL_POLL_TIMEOUT_S=1800.0
23+
EVAL_ARCHITECTURE_SUMMARY=Async legal RAG with OCR-aware ingestion, clause-aware chunking, hybrid dense+BM25 retrieval in Qdrant, reranking, answer-type routing, and page-level grounded telemetry over phase-isolated corpora.
24+
25+
# =========================
26+
# LLM (OpenAI-compatible)
27+
# =========================
28+
LLM_PROVIDER=openai_compatible
29+
LLM_BASE_URL=https://api.openai.com/v1
30+
# Preferred generic key for any OpenAI-compatible provider
31+
LLM_API_KEY=
32+
# Backward-compat aliases (optional): OPENAI_API_KEY / OPENROUTER_API_KEY
33+
LLM_OPENROUTER_REFERER=
34+
LLM_OPENROUTER_TITLE=
35+
ANTHROPIC_API_KEY=
36+
37+
# Per-role models
38+
LLM_SIMPLE_MODEL=gpt-4.1-mini
39+
LLM_COMPLEX_MODEL=gpt-4.1
40+
LLM_STRICT_MODEL=gpt-4.1-mini
41+
LLM_SUMMARY_MODEL=gpt-4.1-mini
42+
LLM_FALLBACK_MODEL=gpt-4.1-mini
43+
44+
# LLM runtime knobs
45+
LLM_SIMPLE_MAX_TOKENS=300
46+
LLM_COMPLEX_MAX_TOKENS=900
47+
LLM_STRICT_MAX_TOKENS=150
48+
LLM_TEMPERATURE=0.0
49+
LLM_TIMEOUT_S=60.0
50+
LLM_CONNECT_TIMEOUT_S=10.0
51+
LLM_MAX_CONTEXT_TOKENS=2500
52+
LLM_STREAM_INCLUDE_USAGE=true
53+
LLM_HTTP2=true
54+
LLM_MAX_CONNECTIONS=50
55+
LLM_MAX_KEEPALIVE_CONNECTIONS=20
56+
# OpenRouter routing hints (optional, OpenRouter base_url only)
57+
LLM_OPENROUTER_PROVIDER_ORDER=
58+
LLM_OPENROUTER_ALLOW_FALLBACKS=true
59+
LLM_CIRCUIT_FAILURE_THRESHOLD=3
60+
LLM_CIRCUIT_RESET_TIMEOUT_S=60.0
61+
LLM_COMPLEX_MIN_LENGTH=150
62+
LLM_COMPLEX_MIN_ENTITIES=2
63+
# LLM_COMPLEX_KEYWORDS=compare,difference,exception,notwithstanding,distinguish,analyze,evaluate,contrast,jurisdiction,precedent
64+
65+
# =========================
66+
# LLM-as-Judge (OpenAI-compatible, optional)
67+
# =========================
68+
JUDGE_ENABLED=false
69+
JUDGE_BASE_URL=https://openrouter.ai/api/v1
70+
JUDGE_API_KEY=
71+
JUDGE_MODEL=google/gemini-2.5-flash-lite
72+
JUDGE_TEMPERATURE=0.0
73+
JUDGE_MAX_TOKENS=900
74+
JUDGE_TIMEOUT_S=60.0
75+
JUDGE_CONNECT_TIMEOUT_S=10.0
76+
JUDGE_HTTP2=true
77+
JUDGE_MAX_CONNECTIONS=20
78+
JUDGE_MAX_KEEPALIVE_CONNECTIONS=10
79+
JUDGE_CONCURRENCY=2
80+
JUDGE_DOCS_DIR=dataset/dataset_documents
81+
JUDGE_SOURCES_MAX_PAGES=6
82+
JUDGE_SOURCES_MAX_CHARS_PER_PAGE=20000
83+
JUDGE_SOURCES_MAX_CHARS_TOTAL=60000
84+
# OpenRouter routing hints (optional, OpenRouter base_url only)
85+
JUDGE_OPENROUTER_PROVIDER_ORDER=
86+
JUDGE_OPENROUTER_ALLOW_FALLBACKS=true
87+
88+
# =========================
89+
# Embedding
90+
# =========================
91+
ISAACUS_API_KEY=
92+
EMBED_MODEL=kanon-2-embedder
93+
EMBED_API_URL=https://api.isaacus.com/v1/embeddings
94+
EMBED_DIMENSIONS=1024
95+
EMBED_BATCH_SIZE=128
96+
EMBED_CONCURRENCY=8
97+
EMBED_TIMEOUT_S=30.0
98+
EMBED_CONNECT_TIMEOUT_S=10.0
99+
EMBED_RETRY_ATTEMPTS=6
100+
EMBED_RETRY_BASE_DELAY_S=0.5
101+
EMBED_RETRY_MAX_DELAY_S=16.0
102+
EMBED_RETRY_JITTER_S=2.0
103+
EMBED_CIRCUIT_FAILURE_THRESHOLD=3
104+
EMBED_CIRCUIT_RESET_TIMEOUT_S=60.0
105+
106+
# =========================
107+
# Reranker
108+
# =========================
109+
ZEROENTROPY_API_KEY=
110+
COHERE_API_KEY=
111+
RERANK_PRIMARY_MODEL=zerank-2
112+
RERANK_PRIMARY_API_URL=https://api.zeroentropy.dev/v1/models/rerank
113+
RERANK_PRIMARY_BATCH_SIZE=50
114+
RERANK_PRIMARY_LATENCY_MODE=fast
115+
RERANK_PRIMARY_TIMEOUT_S=30.0
116+
RERANK_PRIMARY_MAX_CONNECTIONS=20
117+
RERANK_PRIMARY_CONNECT_TIMEOUT_S=10.0
118+
RERANK_FALLBACK_MODEL=rerank-v4.0-fast
119+
RERANK_FALLBACK_TIMEOUT_S=30.0
120+
RERANK_TOP_N=6
121+
RERANK_RERANK_CANDIDATES=80
122+
RERANK_RETRY_ATTEMPTS=4
123+
RERANK_RETRY_BASE_DELAY_S=0.5
124+
RERANK_RETRY_MAX_DELAY_S=8.0
125+
RERANK_RETRY_JITTER_S=1.0
126+
RERANK_CIRCUIT_FAILURE_THRESHOLD=3
127+
RERANK_CIRCUIT_RESET_TIMEOUT_S=60.0
128+
129+
# =========================
130+
# Qdrant
131+
# =========================
132+
# Cloud example:
133+
# QDRANT_URL=https://<cluster-id>.<region>.aws.cloud.qdrant.io
134+
# QDRANT_API_KEY=<qdrant_cloud_api_key>
135+
# QDRANT_USE_CLOUD_INFERENCE=true
136+
#
137+
# Local example:
138+
# QDRANT_URL=http://localhost:6333
139+
# QDRANT_API_KEY=
140+
# QDRANT_USE_CLOUD_INFERENCE=false
141+
#
142+
# Docker Compose services pin their in-container Qdrant host to:
143+
# QDRANT_URL=http://qdrant:6333
144+
# Keep the host-local value here as `localhost`.
145+
QDRANT_URL=http://localhost:6333
146+
QDRANT_API_KEY=
147+
QDRANT_COLLECTION=legal_chunks
148+
QDRANT_POOL_SIZE=20
149+
QDRANT_TIMEOUT_S=30.0
150+
QDRANT_PREFETCH_DENSE=60
151+
QDRANT_PREFETCH_SPARSE=60
152+
QDRANT_USE_CLOUD_INFERENCE=false
153+
QDRANT_ENABLE_SPARSE_BM25=true
154+
QDRANT_SPARSE_MODEL=Qdrant/bm25
155+
QDRANT_FASTEMBED_CACHE_DIR=
156+
QDRANT_SPARSE_THREADS=
157+
QDRANT_CHECK_COMPATIBILITY=false
158+
QDRANT_FUSION_METHOD=RRF
159+
QDRANT_CIRCUIT_FAILURE_THRESHOLD=3
160+
QDRANT_CIRCUIT_RESET_TIMEOUT_S=60.0
161+
162+
# =========================
163+
# Pipeline
164+
# =========================
165+
PIPELINE_CONFIDENCE_THRESHOLD=0.3
166+
PIPELINE_MAX_RETRIES=1
167+
PIPELINE_RETRY_SPARSE_BIAS=90
168+
PIPELINE_RETRY_DENSE_BIAS=40
169+
PIPELINE_DOC_REF_CASE_LAW_FILTER=true
170+
PIPELINE_DOC_REF_PREFETCH_DENSE=20
171+
PIPELINE_DOC_REF_PREFETCH_SPARSE=120
172+
PIPELINE_DOC_REF_SPARSE_ONLY=true
173+
PIPELINE_DOC_REF_MULTI_RETRIEVE=true
174+
PIPELINE_DOC_REF_MULTI_TOP_K_PER_REF=30
175+
PIPELINE_PREMISE_GUARD_ENABLED=true
176+
PIPELINE_PREMISE_GUARD_TERMS=jury,miranda,parole,plea bargain,plea
177+
PIPELINE_RETRY_QUERY_MAX_ANCHORS=3
178+
PIPELINE_MAX_ANSWER_WORDS=350
179+
PIPELINE_RERANK_ENABLED_STRICT_TYPES=true
180+
PIPELINE_RERANK_ENABLED_BOOLEAN=false
181+
PIPELINE_RERANK_MAX_CANDIDATES_STRICT_TYPES=20
182+
PIPELINE_BOOLEAN_RERANK_CANDIDATES_CAP=12
183+
PIPELINE_STRICT_TYPES_CONTEXT_TOP_N=3
184+
PIPELINE_STRICT_TYPES_EXTRACTION_ENABLED=true
185+
PIPELINE_STRICT_TYPES_EXTRACTION_MAX_CHUNKS=4
186+
PIPELINE_STRICT_TYPES_APPEND_CITATIONS=false
187+
PIPELINE_STRICT_REPAIR_ENABLED=true
188+
PIPELINE_BOOLEAN_CONTEXT_TOP_N=2
189+
PIPELINE_BOOLEAN_MULTI_REF_TOP_N=3
190+
PIPELINE_BOOLEAN_MAX_TOKENS=96
191+
PIPELINE_BOOLEAN_PREFETCH_DENSE=40
192+
PIPELINE_BOOLEAN_PREFETCH_SPARSE=40
193+
PIPELINE_CONTEXT_TOKEN_BUDGET_BOOLEAN=900
194+
PIPELINE_CONTEXT_TOKEN_BUDGET_STRICT=1100
195+
PIPELINE_CONTEXT_TOKEN_BUDGET_FREE_TEXT_SIMPLE=1800
196+
PIPELINE_CONTEXT_TOKEN_BUDGET_FREE_TEXT_COMPLEX=2400
197+
PIPELINE_RERANK_SKIP_ON_SINGLE_DOC_REF=true
198+
PIPELINE_RERANK_DOC_MAX_CHARS=1500
199+
PIPELINE_USE_FAST_RERANKER_FOR_SIMPLE=false
200+
PIPELINE_ENABLE_CONFLICT_DETECTION=false
201+
PIPELINE_CONFLICT_DETECTION_ENABLED=false
202+
PIPELINE_CONFLICT_MAX_CHUNKS=8
203+
PIPELINE_ENABLE_MULTI_HOP=false
204+
PIPELINE_DECOMPOSE_ENABLED=false
205+
PIPELINE_MULTI_HOP_MAX_SUBQUERIES=3
206+
PIPELINE_TTFT_EARLY_EMPTY_TOKEN_ENABLED=false
207+
208+
# =========================
209+
# Ingestion
210+
# =========================
211+
INGEST_CHUNK_SIZE_TOKENS=450
212+
INGEST_CHUNK_OVERLAP_TOKENS=45
213+
INGEST_SAC_SUMMARY_MAX_TOKENS=150
214+
INGEST_SAC_SUMMARY_MIN_TOKENS=80
215+
INGEST_SAC_CONCURRENCY=4
216+
INGEST_SAC_DOC_EXCERPT_CHARS=3000
217+
INGEST_UPSERT_BATCH_SIZE=200
218+
INGEST_INGEST_VERSION=v2
219+
INGEST_PARSER_PDF_TEXT_MIN_CHARS=400
220+
INGEST_PARSER_PDF_TEXT_MIN_WORDS=80
221+
INGEST_MANIFEST_HASH_CHUNK_SIZE_BYTES=1048576
222+
INGEST_MANIFEST_DIR=
223+
INGEST_MANIFEST_FILENAME=.rag_challenge_ingestion_manifest.json
224+
INGEST_MANIFEST_SCHEMA_VERSION=1
225+
226+
# =========================
227+
# Verifier
228+
# =========================
229+
VERIFY_ENABLED=true
230+
VERIFY_MAX_TOKENS=500
231+
VERIFY_TEMPERATURE=0.0
232+
233+
# =========================
234+
# Application
235+
# =========================
236+
APP_LOG_LEVEL=INFO
237+
APP_LOG_FORMAT=json
238+
APP_ENVIRONMENT=development
239+
APP_HOST=0.0.0.0
240+
APP_PORT=8000
241+
APP_WARMUP_ENABLED=false
242+
APP_WARMUP_EMBED=false
243+
APP_WARMUP_LLM=false
244+
APP_WARMUP_TIMEOUT_S=5.0
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: Bug Report
2+
description: Something isn't working
3+
labels: ["bug"]
4+
body:
5+
- type: textarea
6+
id: what-happened
7+
attributes:
8+
label: What happened?
9+
description: Describe the bug. What did you expect vs what actually happened?
10+
placeholder: "I ran `make test` and got..."
11+
validations:
12+
required: true
13+
- type: textarea
14+
id: reproduce
15+
attributes:
16+
label: Steps to reproduce
17+
description: Minimal steps to reproduce the issue
18+
value: |
19+
1.
20+
2.
21+
3.
22+
validations:
23+
required: true
24+
- type: textarea
25+
id: env
26+
attributes:
27+
label: Environment
28+
description: OS, Python version, relevant service versions
29+
value: |
30+
- OS:
31+
- Python:
32+
- Qdrant:
33+
- Docker Compose:
34+
validations:
35+
required: false
36+
- type: textarea
37+
id: logs
38+
attributes:
39+
label: Logs
40+
description: Paste relevant logs or error output
41+
render: shell
42+
validations:
43+
required: false

‎.github/ISSUE_TEMPLATE/config.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
blank_issues_enabled: false
2+
contact_links:
3+
- name: Questions & Ideas
4+
url: https://github.com/chernistry/shafi/discussions
5+
about: Ask questions and share ideas in Discussions
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Feature Request
2+
description: Suggest an idea
3+
labels: ["enhancement"]
4+
body:
5+
- type: textarea
6+
id: problem
7+
attributes:
8+
label: Problem
9+
description: What problem does this solve? What's frustrating today?
10+
placeholder: "When I try to ... I have to ..."
11+
validations:
12+
required: true
13+
- type: textarea
14+
id: solution
15+
attributes:
16+
label: Proposed solution
17+
description: How should it work? Examples welcome.
18+
placeholder: "It would be great if `make eval` could..."
19+
validations:
20+
required: false
21+
- type: dropdown
22+
id: area
23+
attributes:
24+
label: Area
25+
options:
26+
- Retrieval pipeline
27+
- Ingestion / embedding
28+
- Evaluation harness
29+
- API / server
30+
- Infrastructure / Docker
31+
- CI / tooling
32+
- Documentation
33+
- Other
34+
validations:
35+
required: true

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## Summary
2+
3+
<!-- One-liner: what does this PR do and why? -->
4+
5+
## Type of change
6+
7+
- [ ] Bug fix (non-breaking change that fixes an issue)
8+
- [ ] New feature (non-breaking change that adds functionality)
9+
- [ ] Refactor (no functional changes)
10+
- [ ] Documentation update
11+
- [ ] CI / infrastructure change
12+
13+
## Testing
14+
15+
- [ ] `make lint` passes
16+
- [ ] `make typecheck` passes
17+
- [ ] `make test` passes
18+
- [ ] New code has type hints
19+
- [ ] Docs updated if needed

‎.github/dependabot.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: pip
4+
directory: /
5+
schedule:
6+
interval: weekly
7+
groups:
8+
minor-and-patch:
9+
update-types:
10+
- minor
11+
- patch
12+
ignore:
13+
- dependency-name: "*"
14+
update-types: ["version-update:semver-major"]
15+
16+
- package-ecosystem: github-actions
17+
directory: /
18+
schedule:
19+
interval: monthly

0 commit comments

Comments
 (0)