Ali Zarinkolah
5c0a5938f8
Why: - Plan 001 Phase 4 needs batched, concurrency-bounded embedding wired into the inline upload path, with process-wide capacity/timeout/chunk-limit guards (ADR-0017). - The BM25 analyzer and dense-model config are ported from the `emet` evaluation lab, which benchmarked them against the real Farsi corpus (bm25-fa-norm-stop; nomic-embed-text-v2-moe at 768-dim; text-embedding-3-large at native 3072-dim), closing open items in ADR-0001/ADR-0005. Changes: - New: embedding ports, orchestration (embed_chunks), request-bounds helpers, and dense/sparse adapters (analyzers.py, bm25.py, openai_compatible.py). - upload.py now parses/chunks/embeds inline behind INGESTION_MAX_CONCURRENCY (503), INGESTION_TIMEOUT_SECONDS (504), and the chunk-count ceiling (413); every failure path still writes a terminal job row. - Lifespan builds and warms both dense embedders at startup (fail-soft) and creates the sparse embedder and concurrency semaphore. - httpx moves from dev to main dependencies (adapters use it directly). Impact: - Qdrant point upserts are still Phase 5 -- chunks_indexed stays 0. - New EMBEDDING_* env vars documented in .env.example; safe defaults. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
48 lines
1.6 KiB
Python
48 lines
1.6 KiB
Python
from collections.abc import AsyncIterator
|
|
|
|
import pytest
|
|
import pytest_asyncio
|
|
from asgi_lifespan import LifespanManager
|
|
from fastapi import FastAPI
|
|
from httpx import ASGITransport, AsyncClient
|
|
|
|
from src.config import Settings
|
|
from src.main import create_app
|
|
|
|
|
|
@pytest.fixture
|
|
def settings() -> Settings:
|
|
# Every external dependency points at a closed port so unit tests never
|
|
# reach real infrastructure. For the embedders this matters twice over:
|
|
# the real defaults are a colleague's Ollama box and OpenAI's paid API,
|
|
# and ADR-0016 forbids routine runs calling either. Connection-refused is
|
|
# immediate, so the lifespan's fail-soft warm-up costs nothing here --
|
|
# and these tests passing at all is what proves it is fail-soft.
|
|
return Settings(
|
|
postgres={"host": "127.0.0.1", "port": 1},
|
|
minio={"endpoint": "127.0.0.1:1"},
|
|
ingestion={"timeout_seconds": 1.0},
|
|
qdrant={"url": "http://127.0.0.1:1"},
|
|
app={"readiness_check_timeout_seconds": 0.5},
|
|
embedding={
|
|
"nomic": {"base_url": "http://127.0.0.1:1/v1", "timeout_seconds": 0.5},
|
|
"openai": {"base_url": "http://127.0.0.1:1/v1", "timeout_seconds": 0.5},
|
|
},
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def app(settings: Settings) -> FastAPI:
|
|
return create_app(settings)
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def client(app: FastAPI) -> AsyncIterator[AsyncClient]:
|
|
async with (
|
|
LifespanManager(app) as manager,
|
|
AsyncClient(
|
|
transport=ASGITransport(app=manager.app), base_url="http://test"
|
|
) as async_client,
|
|
):
|
|
yield async_client
|