Ali Zarinkolah
5c0a5938f8
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Talie chatbot service
Architecture decisions live in docs/adr. The first implementation
milestone is documented in the ingestion vertical-slice plan.
Local Langfuse
This repo includes a root-level development Compose file for Langfuse:
Start Langfuse locally:
cp .env.langfuse.example .env.langfuse
# edit .env.langfuse and replace CHANGE_ME values
docker compose --env-file .env.langfuse -f docker-compose.langfuse.yml up -d
Open:
http://localhost:3000
If the chatbot app runs on your host machine, configure it with:
LANGFUSE_HOST=http://localhost:3000
If the chatbot app later runs inside the same Compose project/network as Langfuse, configure it with:
LANGFUSE_HOST=http://langfuse-web:3000
A future app stack can be launched together with Langfuse using multiple Compose files:
docker compose \
-f docker-compose.yml \
-f docker-compose.langfuse.yml \
--env-file .env \
--env-file .env.langfuse \
up -d