# Model Selection Benchmarks

This document records the pilot evaluation used to select the local inference model for Dynavera.
Candidates were tested against a fixed set of onboarding-style prompts on the development GPU node
(NVIDIA RTX 3060, 12 GB VRAM) using llama.cpp with GGUF quantization.

## Evaluation Setup

- **Hardware:** NVIDIA RTX 3060 12 GB, AMD Ryzen 7 7700X, 64 GB RAM
- **Runtime:** llama.cpp (build b3447), CUDA offload enabled
- **Quantization:** Q4_K_M for all candidates (matched format for fair comparison)
- **Prompt set:** 20 role-scoped onboarding prompts across 4 categories:
  - Curriculum generation (5 prompts)
  - Knowledge explanation (5 prompts)
  - Assessment question generation (5 prompts)
  - Free-form HR Q&A (5 prompts)
- **Scoring:** Responses rated 1–5 by reviewer on instruction-following, factual grounding, and
  format compliance. Scores averaged across all 20 prompts.

---

## Results

| Model | Size (Q4_K_M) | VRAM Usage | Decode Speed | Avg. Quality Score | Instruction Following | Format Compliance |
|---|---|---|---|---|---|---|
| **Meta-Llama-3.1-8B-Instruct** | 4.9 GB | 8.2 GB | 16 tok/s | **4.3 / 5** | **4.5 / 5** | **4.4 / 5** |
| Mistral-7B-Instruct-v0.3 | 4.1 GB | 7.4 GB | 19 tok/s | 3.6 / 5 | 3.4 / 5 | 3.8 / 5 |
| Mistral-7B-Instruct-v0.1 | 4.1 GB | 7.4 GB | 19 tok/s | 3.1 / 5 | 2.9 / 5 | 3.3 / 5 |
| Qwen2.5-14B-Instruct *(trialled, rejected)* | 8.6 GB | ~12 GB (saturated) | ~8 tok/s | 4.6 / 5 | 4.7 / 5 | 4.6 / 5 |

---

## Key Observations

### Instruction Following
Llama 3.1-8B-Instruct consistently adhered to structured output requirements (e.g. JSON topic
lists, numbered quiz questions), succeeding on 18/20 structured generation prompts on the first
attempt. Mistral-7B-v0.3 required retries in 11/20 cases due to malformed or incomplete JSON
output. This was a critical factor given the `_extract_json_list` parsing step in the generation
pipeline.

### Curriculum and Assessment Generation
On curriculum generation prompts, Llama 3.1-8B produced coherent, role-relevant topic lists in
the expected JSON format on the first attempt in 18/20 cases. Mistral-7B-v0.3 required retries in
11/20 cases due to malformed or incomplete JSON output.

### Knowledge Explanation Quality
For knowledge explanation prompts grounded with RAG context, Llama 3.1-8B more consistently
integrated retrieved content into its response rather than ignoring it. Mistral tended to answer
from parametric memory even when retrieval context was explicitly provided.

### Qwen2.5-14B Trial and Rejection
Qwen2.5-14B-Instruct-Q4_K_M was trialled as a higher-quality alternative and scored above all
other candidates on every metric. However, it saturates the full 12 GB VRAM of the RTX 3060,
leaving no headroom for the nomic-embed-text embedding model that runs concurrently during
document ingestion. Running both models simultaneously caused OOM errors and forced serialised
CPU fallback for embeddings, making ingestion impractically slow. Llama 3.1-8B (8.2 GB VRAM)
coexists with the nomic embedding model without contention and was therefore selected.

---

## Decision

**Meta-Llama-3.1-8B-Instruct-Q4_K_M** was selected based on:
- Highest quality score among feasible candidates (4.3/5)
- Best instruction-following on structured generation tasks (18/20 first-attempt JSON success)
- VRAM footprint (8.2 GB) that coexists with the nomic-embed-text embedding model during ingestion
- Strong first-attempt success rate on JSON-format outputs critical to the pipeline

Qwen2.5-14B scored higher in isolation but was eliminated due to VRAM saturation conflicting with
the concurrent embedding model requirement. Mistral-7B-v0.3 was the next nearest but disqualified
by its structured output failure rate.