Ali Zarinkolah
5e935e5895
Why: - ADR-0002's keyword search needs a full-text index on content, which collection.py deliberately deferred to plan 002. is_active and chunk_index were unindexed while ingestion was the only reader; every /v1/points read path filters on them. Changes: - content gets a TEXT index with the multilingual tokenizer, which segments Persian correctly where the word tokenizer mishandles ZWNJ-joined compounds. No stemmer or stopword list: content is already letter-folded by normalize_persian_text at ingest, and the ranked Farsi lexical path is the benchmarked BM25 sparse vector, not this index. - Tests assert content is TEXT rather than KEYWORD -- a keyword index would only match an entire chunk verbatim, which never happens and fails silently. - Adds a test that a missing index is added to an already-live collection. Impact: - Requires re-running `python -m src.cli.qdrant_bootstrap`. Payload indexes are additive, so no collection rebuild and no re-embedding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Talie chatbot service
Architecture decisions live in docs/adr. The first implementation
milestone is documented in the ingestion vertical-slice plan.
Day-to-day operation — tuning the ingestion bounds, the proxy timeout
requirement, and how to investigate or retry a failed upload — is the
operator runbook.
Provisioning the datastores
Both schema steps run as explicit deployment steps. The application performs no DDL at startup — not for Postgres (ADR-0009) and not for Qdrant (ADR-0001, "Collection provisioning").
docker compose up -d # Postgres, MinIO, Qdrant
uv run alembic upgrade head # Postgres schema
uv run python -m src.cli.qdrant_bootstrap # the `chunks` collection
uv run fastapi dev src/main.py
Nothing over HTTP can create the first tenant — every /v1 route needs an API
key, and a key cannot exist before its tenant. One command issues both, plus any
domains, printing the key once (only its hash is stored):
uv run python -m src.cli.provision_tenant --slug acme --domain fire
Before a tenant can upload, its domains must be registered — POST /v1/files
rejects an unregistered or disabled domain with 400. The calling backend
manages them over /v1/domains using a key with the domains:write scope:
curl -X POST http://localhost:8000/v1/domains \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{"domain": "fire", "display_name": "Fire insurance"}'
Both bootstrap commands are idempotent and safe to re-run. qdrant_bootstrap verifies an
existing collection against the pinned schema and exits non-zero on a mismatch,
rather than leaving a silently degraded sparse index in place.
./scripts/smoke.sh verifies the whole path — Compose up, both deployment
steps, provisioning, an upload through the running web process to indexed Qdrant
points. See the runbook.
Local Langfuse
This repo includes a root-level development Compose file for Langfuse:
Start Langfuse locally:
cp .env.langfuse.example .env.langfuse
# edit .env.langfuse and replace CHANGE_ME values
docker compose --env-file .env.langfuse -f docker-compose.langfuse.yml up -d
Open:
http://localhost:3000
If the chatbot app runs on your host machine, configure it with:
LANGFUSE_HOST=http://localhost:3000
If the chatbot app later runs inside the same Compose project/network as Langfuse, configure it with:
LANGFUSE_HOST=http://langfuse-web:3000
A future app stack can be launched together with Langfuse using multiple Compose files:
docker compose \
-f docker-compose.yml \
-f docker-compose.langfuse.yml \
--env-file .env \
--env-file .env.langfuse \
up -d