# Documentation Index

Complete documentation for the RAG Chunking Benchmarker.

---

## Quick Start

1. **New to the project?** Start with [HLD](HLD.md) (system design) then [LLD](LLD.md) (module detail)
2. **Domain language?** Read [CONTEXT.md](../CONTEXT.md)
3. **Want to understand strategies?** Read [Strategy Technical Details](strategy-technical-details.md)
4. **Need to use the API?** Check [API Reference](api-reference.md)
5. **Configuring the system?** See [Configuration Guide](configuration.md)
6. **Understanding results?** Read [Evaluation Metrics](evaluation-metrics.md)

---

## Documentation Files

| File | Purpose | Audience |
|------|---------|----------|
| [HLD.md](HLD.md) | High-level design: context, components, flows | Architects, new team members |
| [LLD.md](LLD.md) | Low-level design: schemas, algorithms, APIs | Implementers |
| [architecture.md](architecture.md) | Legacy overview (prefer HLD) | New team members |
| [strategy-technical-details.md](strategy-technical-details.md) | Deep dive into each strategy | Engineers |
| [api-reference.md](api-reference.md) | All endpoints documented | Developers |
| [configuration.md](configuration.md) | Settings and environment variables | DevOps |
| [evaluation-metrics.md](evaluation-metrics.md) | How scoring works | Data scientists |
| [data-flow.md](data-flow.md) | How data moves through the system | Engineers |
| [chunking_strategies.md](chunking_strategies.md) | High-level strategy overview | Everyone |
| [final-chunking-strategy-decision.md](final-chunking-strategy-decision.md) | Final Strategy family decision (`fixed_size`) + charts | Managers, operators |
| [adr/](adr/) | Architectural Decision Records | Everyone |
| [phases.md](phases.md) | Implementation phases | Project managers |
| [tasks.md](tasks.md) | Task tracking | Developers |

---

## Architecture at a Glance

```
┌─────────────────────────────────────────────────────────────────┐
│                    RAG Chunking Benchmarker                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Documents → Parser → Chunking → Embedding → Qdrant           │
│                                                                 │
│  Questions → Query Pipeline → LLM → Answers                    │
│                                                                 │
│  Answers → Evaluation → Metrics → Reports                      │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

---

## Key Concepts

### Chunking Strategies

| Strategy | Splitting | Best For |
|----------|-----------|----------|
| fixed_size | Token count | Baseline comparison |
| recursive | Structural boundaries | Structured documents |
| semantic | Similarity thresholds | Topic shifts |
| contextual_retrieval | LLM-enriched tokens | Retrieval quality |
| semantic_parent_child | Paragraph clusters | Context needed |

### Evaluation Metrics

| Metric | What It Measures |
|--------|------------------|
| Context Relevance | Did we find the right information? |
| Answer Similarity | Did we produce the right answer? |
| Faithfulness | Is the answer trustworthy? |
| Hallucination | Did we invent information? |

---

## API Endpoints

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/documents` | GET/POST | Manage documents |
| `/documents/{id}/process` | POST | Run chunking strategies |
| `/queries` | POST | Ask questions |
| `/benchmarks` | POST | Run comparisons |
| `/benchmarks/{id}/report` | GET | View results |
| `/experiments` | GET | List experiments |

---

## Configuration

All settings via `.env` file. See [Configuration Guide](configuration.md) for details.

Key settings:
- `OPENAI_API_KEY` - Required for embeddings and LLM
- `CHUNK_SIZE` - Target tokens per chunk (default: 512)
- `SEMANTIC_THRESHOLD` - Similarity threshold (default: 0.5)

---

## Further Reading

- [Phases](phases.md) - Implementation roadmap
- [Tasks](tasks.md) - Task tracking
- [ADRs](adr/) - Architectural Decision Records