RAG Retrieval Benchmark — Academic Report

Measured results

Common-cutoff quality and latency

Every plotted value comes from the current aggregate artifacts and was reconciled with the 208 case rows.

Active comparisonnDCG@10

LuceneRAG, no rerankRAG + rerank

LUC

0.619

STD-WHOLE-RR

0.666

STD-CRIT-NO-RR

0.548

STD-CRIT-RR

0.697

SEL-WHOLE-NO-RR

0.568

SEL-WHOLE-RR

0.681

SEL-CRIT-NO-RR

0.527

SEL-CRIT-RR

0.689

Table 1. Current macro results for the eight admitted experiments.
Condition	P@10	R@10	nDCG@10	MRR	Mean ms	p95 ms
LuceneLUC	0.523	0.407	0.619	0.837	197	386
RAG Standard Whole + rerankSTD-WHOLE-RR	0.550	0.409	0.666	0.942	14,291	17,576
RAG Standard Criteria, no rerankSTD-CRIT-NO-RR	0.454	0.363	0.548	0.799	1,932	2,141
RAG Standard Criteria + rerankSTD-CRIT-RR	0.577	0.446	0.697	0.942	14,469	19,823
RAG Selective Whole, no rerankSEL-WHOLE-NO-RR	0.496	0.381	0.568	0.788	1,667	1,824
RAG Selective Whole + rerankSEL-WHOLE-RR	0.565	0.429	0.681	0.962	14,093	18,719
RAG Selective Criteria, no rerankSEL-CRIT-NO-RR	0.465	0.355	0.527	0.770	1,776	2,162
RAG Selective Criteria + rerankSEL-CRIT-RR	0.573	0.437	0.689	0.923	14,289	20,243

Matched reranker effects

Standard criteria

+0.149 nDCG@10

Case-bootstrap 95% CI: [0.0557, 0.2526]
Wins / ties / losses: 16 / 2 / 8
Mean latency: +12.54 s

Selective criteria

+0.162 nDCG@10

Case-bootstrap 95% CI: [0.0707, 0.2599]
Wins / ties / losses: 18 / 1 / 7
Mean latency: +12.51 s

Selective whole

+0.113 nDCG@10

Case-bootstrap 95% CI: [0.0508, 0.1782]
Wins / ties / losses: 19 / 1 / 6
Mean latency: +12.43 s

Interpret carefully.

The intervals resample benchmark cases. They do not represent repeated service executions, GPU variability, cache state, or network variance.

Retrieval methods

Comparable evidence, deliberately different mechanisms

Raw scores are not commensurable. The valid comparison uses common cases, judgments and rank metrics.

Semantic lane

RAG

PDF → Markdown → structured JSON
Jina 1,024-D chunk embeddings
1,000 chunk candidates
Aggregation to 60 documents
Optional Qwen reranking to top 25

Strength hypothesis: paraphrase and distributed evidence. Counter-hypothesis: semantic drift, chunk loss, or candidate truncation.

Lexical lane

Lucene

Parsed and normalized text
Language-specific index
Authored syntax and field boosts
Fuzzy/multilingual expansion
SquareSimilarity, filters, top 100

Strength hypothesis: rare exact terms and phrases. Counter-hypothesis: expansion drift or weak paraphrase coverage.

**RAG query and reranking path**Dense candidate generation, aggregation and bounded reranking.

**Lucene query path**Lexical query construction, scoring, filtering and ranking.

**Controlled RAG and Lucene mechanisms**Shared corpus and judgments do not imply identical score construction.

Mathematical model

Equations expose assumptions

Implemented definitions, explanatory abstractions and admissible comparisons are kept distinct.

Lucene term frequencytf(f) = √f

Diminishing returns for repeated terms; boilerplate can still accumulate evidence.

Lucene inverse document frequencyidf(t) = ln((N+1)/(df(t)+1)) + 1

Rare terms gain weight; rare OCR errors may also dominate.

Dense similaritys(q,c) = (e_q · e_c) / (‖e_q‖ ‖e_c‖)

Requires an identical model, prefix, pooling, dimension and normalization contract.

Section aggregationAⱼ(d) = 0.7 max(sⱼ) + 0.3 mean(top₃ sⱼ)

Balances one strong passage with repeated evidence; 0.7/0.3 is configured, not estimated here.

Criteria scoreSᶜʳⁱᵗ(d) = Σᵢ(wᵢ vᵢ sᵢ(d)) / Σᵢwᵢ

Frozen-criteria quality, section mapping and skipped criteria remain alternative explanations.

Section-aware rerankr_d = 0.7 max(pₘ·importanceₘ) + 0.3 mean(top₃)

The returned score is an ordering signal, not assumed to be a calibrated probability.

Candidate survivalR@k_final ≤ R@60_first-stage

The reranker cannot recover a relevant document omitted before the 60-document pool.

Paired quality differenceΔM_c = M_c(RAG) − M_c(Lucene)

A benchmark-case observation, not a universal model property.

Latency accountingΔT = T_embed + T_vector + T_agg + I_rrT_rerank + T_other − T_Lucene

An accounting identity, not a hardware-independent complexity result.

No raw-score shortcut.

SquareSimilarity, cosine, aggregation and reranker scores occupy different spaces. Direct engine comparison begins only after rankings are evaluated against the same judgments.

Model analysis

Capability is not benchmark evidence

Published specifications, local serving choices and observed outcomes are treated as separate evidence layers.

Bi-encoder

Jina v5 small retrieval

Corpus-scale reusable representations

Model/profilejina-embeddings-v5-text-small-retrieval / jina-v5-small-vllm

Pin the exact local snapshot and hash.

Published capability677M; 1,024-D; LAST; 32,768 context; Matryoshka

Capability is not CV-domain effectiveness evidence.

Local servicevLLM pooling; FP16; 8,192 tokens; GPU target 0.3

Backend, batching and contention were not ablated.

NormalizationResolved normalize=true; not explicit in startup pooler config

Verify the effective service/application path.

Chunking600 tokens / 60 overlap

Overlap protects boundaries but creates correlated candidates.

Query intentFind the most suitable job applicants for these requirements:

Prefix sensitivity is not isolated.

Cross-encoder

Qwen3 Reranker 4B

Bounded joint query–evidence scoring

Model/profileQwen3-Reranker-4B / no-instruction profile

An observed gain belongs to this checkpoint and pool.

Published capability4B; 32K; 100+ languages; optional instructions

Published breadth does not prove recruitment ranking quality.

Local serviceBF16; 8,192 tokens; GPU target 0.5

ROCm, vLLM, hardware and contention may affect results.

ArchitectureQwen3ForSequenceClassification; no/yes tokens; Jinja template

Template and override changes are model changes.

Candidate/output60 input documents / top 25

First-stage omissions are irreversible.

Section planner2,000-token chunk; 3,000 atomic; 8,000 sequence

Splitting and importance weights confound pure checkpoint effects.

Architecture critique.

A smaller bi-encoder followed by a larger bounded cross-encoder is computationally coherent, but coherence is not proof of optimality. Pool size, normalization, instruction choice, templates, splitting, another checkpoint, or a lexical–dense hybrid remain credible alternatives.

Critical analysis

What else could explain the pattern?

An academic report challenges causal stories even when the aggregate result supports the system.

Reranked quality lead

May reflect the Qwen checkpoint, candidate composition, section formatting, or top-25 policy—not dense retrieval alone.

Required: pre-rerank rankings and candidate recall.

Lucene latency lead

May combine algorithmic efficiency with date-specific cache, host, database, queue and service conditions.

Required: repeated interleaved controlled runs.

Selective policy

Changes scope and weights together; whole mode can require all selected section types.

Required: scope-only and weight-only ablations.

Embedding contract

Resolved normalization is true, while the startup pooler configuration does not state normalization explicitly.

Required: verify effective output norms and index provenance.

Subgroup results

Group sizes are 9, 13, 3 and 1. A one-case group cannot support a stable subgroup generalization.

Required: case distributions and broader samples.

Operational variance

One measured repetition per case cannot estimate run-to-run reliability or tail latency stability.

Required: replicated runs with stage timing.

Figure atlas

31 report figures

Method diagrams are separated from measured-result and diagnostic figures. Select any figure to enlarge it.

Showing 31 of 31 figures.

Validated deliverables

Download the complete evidence package

The web publication is an analytical companion. The Word report and Excel workbook remain the authoritative formatted deliverables.

DOCX · academic report29 pages · 31 figures XLSX · result workbook22 sheets · 8 experiment pages JSON · validation recordEvidence and package checks

Measured evaluation of a RAG retrieval benchmark

Common-cutoff quality and latency

Matched reranker effects

Comparable evidence, deliberately different mechanisms

RAG

Lucene

Equations expose assumptions

Capability is not benchmark evidence

Jina v5 small retrieval

Qwen3 Reranker 4B

What else could explain the pattern?

31 report figures

Download the complete evidence package