6. The Need for Speed
6.1 Offline extraction versus online search latency
Speed requirements differ between the offline and online components of the system. CV extraction is normally an offline or nearline process triggered when a document is added or re-indexed. Its primary objectives are accuracy, throughput, and predictable resource use. Candidate search is interactive; it requires low query latency and should not invoke a 26B extraction model over every CV at query time. The structured and raw-text indexes should be precomputed, and the large model should be reserved for top-k reranking or evidence explanation.
The need for speed is operationally important because every prompt, model, parser, or schema change can require corpus-wide reprocessing. If extraction is slow, regression testing becomes expensive and teams are less likely to rerun quality checks. High throughput therefore supports quality governance, not merely cost reduction.
6.2 Runtime trade-offs in the supplied configuration
The supplied configuration contains several speed-oriented choices: a 4-bit quantized checkpoint, a sparse MoE architecture, automatic GPU-layer offload, Flash Attention selection, reasoning disabled, a physical micro-batch of 64, and four parallel slots. QAT is designed to reduce quality loss relative to post-training quantization [5]. MoE reduces active compute per token, while quantization reduces weight memory and memory bandwidth. Flash Attention can reduce attention-memory traffic when supported.
These settings also create trade-offs. Four parallel slots can increase aggregate throughput but may reduce per-request memory headroom and latency. A 65,536-token context increases KV-cache requirements even when typical CVs are much shorter. An upper batch of 64 may underutilize or protect memory depending on hardware and prompt lengths. Automatic layer offload and Flash Attention can vary by build and device. Therefore, the command line is not itself evidence of achieved speed.
6.3 Throughput engineering and measurement
| Metric | Unit | Purpose |
|---|---|---|
| PDF parsing latency | p50, p95, p99 milliseconds/document | Separates document decoding from model inference. |
| Prompt tokens | tokens/document and distribution | Explains prefill cost and truncation risk. |
| Time to first token | milliseconds | Measures prefill and scheduling latency. |
| Prefill throughput | input tokens/second | Sensitive to context length, batching, and attention implementation. |
| Decode throughput | output tokens/second per slot and aggregate | Sensitive to quantization, memory bandwidth, and active experts. |
| End-to-end throughput | documents/minute | Primary offline capacity measure. |
| End-to-end latency | p50/p95 seconds/document | Captures queueing and validation. |
| Peak memory | VRAM and system RAM | Determines safe concurrency and context. |
| Energy | Wh/document or joules/token | Useful for sustained local deployment. |
| Quality-speed frontier | F1/completeness versus latency | Required for reasoning, quantization, context, and model comparisons. |
The benchmark should use a stratified set by document length, layout, language, and occupation. Warm and cold runs should be separated. Concurrency should be tested at one through four slots, and the exact same documents should be evaluated to create a quality-speed frontier. Mean throughput alone is insufficient because recruitment ingestion pipelines are affected by tail latency and outlier documents.
6.4 Recommended performance architecture
A performant production design should use separate models for separate roles. PDF decoding and section segmentation should be performed once. A multilingual embedding model can index structured values and raw sections. A small cross-encoder or late-interaction reranker can score the top retrieval candidates. The larger Gemma model can then generate a concise evidence explanation for only the final shortlist. This architecture avoids repeatedly invoking the extraction model during search and preserves the ability to re-rank with source evidence.
Prompt-prefix caching can further reduce repeated system-prompt prefill where supported, but it must be measured with the exact llama.cpp build and chat template. Context size should be set to the largest value required by the observed CV distribution rather than the model maximum. Batching should group documents with similar prompt lengths to reduce padding and scheduling inefficiency.