Overview and abstract

Scope statement: The analysis covers one validated evaluation report for each of 24 predefined occupational categories and 2,484 source CV PDFs. CV_Local_LLM_Extraction_Evaluation(7).xlsx is outside the set and is excluded. The valid Advocate category is represented by ADVOCATE-CV_Local_LLM_Extraction_Evaluation.xlsx; the mislabeled detailed duplicate is not used in the aggregate.

Abstract

This study evaluates a locally hosted, quantized Gemma 4 26B A4B mixture-of-experts instruction model used to extract structured information from curriculum vitae (CV) documents for downstream retrieval-augmented generation (RAG). The corrected corpus contains 2,484 original CVs in PDF format distributed across 24 predefined occupational categories, with one validated evaluation report per category. The original PDFs are treated as the primary evidentiary source; a separate reference extraction is used only for diagnostic disagreement analysis. Performance is assessed at pipeline, document, segment, and atomic-fact levels using strict and partial-credit precision, recall, F1, completeness, unsupported-content rate, schema completeness, critical-claim rate, and operational document outcomes. The pooled weighted precision is 94.8%, weighted recall is 65.5%, weighted F1 is 77.4%, completeness is 68.2%, and unsupported content is 1.23%. The technical pipeline produced 2,482 evaluable matched records from 2,484 PDFs (99.92%). Only 201 documents (8.1%) satisfy the joint success thresholds; 1,989 are partially successful and 294 fail. The dominant failure mode is omission rather than hallucination. Experience and profile fields perform comparatively well, while category classification, normalized fields, seniority, and document metadata are unsuitable as authoritative retrieval filters. CV structure and information density vary substantially, and the semantic boundary between skills and experience is often non-exclusive; the same source statement may validly function as both demonstrated experience and skill evidence. The report formalizes the evaluation mathematics, analyzes the supplied inference configuration, examines the speed-quality trade-off associated with reasoning-disabled inference, and critiques the extraction and evaluation prompts. The principal deployment conclusion is that the extracted JSON can serve as a high-precision indexing layer and source of positive evidence, but it must be combined with raw-section fallback, field-compatible retrieval, evidence citation, quality-aware ranking, and human review. Missing structured values cannot safely be interpreted as absence in the source CV.

Keywords: curriculum vitae; information extraction; retrieval-augmented generation; local language model; mixture of experts; precision-recall trade-off; multilingual retrieval; structured search; llama.cpp; prompt engineering

Data and materials. The empirical basis consists of 24 category-specific evaluation workbooks covering 2,484 source CV PDFs, prompt2(5), aiextract.sys.prompt, and aiextract.user.prompt. The original Advocate workbook is used for the Advocate category; the generic outside-scope workbook and the mislabeled detailed Advocate duplicate are excluded. The coverage supplement separates source absence from extraction failure and shows materially broader apparent extraction coverage for ChatGPT than for the reasoning-disabled Gemma configuration.