Study design
Corpus, extraction contract, and evidence lineage
The corpus contains 2,484 PDFs distributed over 24 occupational categories. All systems target the supplied JSON schema. The common audit canonicalizes archive names, uses identical atomic splitting and PDF support rules, penalizes missing outputs, and reports pooled counts plus document-bootstrap uncertainty.
ChatGPT supplies candidate omitted facts only. A candidate enters the denominator only when normalized PDF text supports it. This improves omission search but does not eliminate reference omissions or lexical adjudicator error.
Evidence-to-result workflow
Results are reconstructed from frozen artifacts
The primary numerical results are not produced by asking another language model which extraction looks more convincing. Every result follows a traceable lineage: archived PDFs and JSON outputs are aligned by canonical numeric document ID; wrapper objects are parsed under one policy; PDF text and extraction segments are normalized identically; segment content is decomposed into atomic facts; each fact receives a PDF-support classification; atomic counts are retained by document and segment; and only then are precision, recall, F1, completeness, uncertainty intervals, figures, and claims calculated.
| Stage | Frozen operation | Retained evidence | Bias controlled |
|---|---|---|---|
| Inventory | Canonical numeric ID matching | Presence, duplicate, and parse ledger | Archive-order and candidate-name bias |
| Unwrapping | One wrapper-resolution policy | Extraction object or explicit exclusion | System-specific silent repair |
| Atomicization | One segment mapper and fact splitter | Prediction and omission-candidate fact sets | Whole-document length confounding |
| Support | Exact, token, and 2/3-gram rules | Correct, partial, unsupported, unverifiable | Preference for a named architecture |
| Aggregation | Frozen denominators and partial weight | TP, PARTIAL, FP, FN and derived metrics | Post-hoc metric redefinition |
| Reporting | Figures inherit an explicit evidence tier | Permitted and prohibited claims | Mixing incompatible evaluators |
Functional blinding
The scoring function cannot reward a model name
Archive identity is necessary for file loading and is restored after scoring for aggregation. It is not an input to the atomic decision function. The scoring kernel receives the candidate fact, schema segment, and PDF evidence, but no system identity. Thus an identical fact supported by an identical PDF passage receives the same label whether it originated from Gemma, Qwen, Hybrid NLP, or BERT MULTI.
This is functional rather than social blinding: the research process still knows which archive is being loaded, but the implemented judgment rule cannot branch on that identity. It controls evaluator favoritism more strongly than an instruction asking a human or LLM to “be neutral,” while leaving normalization, fact splitting, and threshold choice open to methodological criticism.
Evaluator choice
Why an LLM is not the sole primary judge
An LLM can discover paraphrases and provide useful qualitative review, but an unvalidated LLM judge is not an adequate sole basis for the primary numerical benchmark. Its decisions can depend on model snapshot, hidden system instructions, provider updates, prompt wording, item order, context truncation, decoding parameters, and stochastic sampling. A nominally identical rerun can therefore alter atomic labels and even model rankings.
Scores become properties of extractor–judge pairs
Substituting another LLM as judge does not make the comparison neutral. It introduces judge identity as an additional experimental factor. Different judges can apply different implicit definitions of semantic equivalence, acceptable inference, completeness, normalization, and hallucination. They may favor outputs resembling their own wording, ontology, reasoning style, tokenizer conventions, instruction tuning, training distribution, or model-family priors. Because those affinities are not directly observable, a single judge cannot separate extraction quality from judge preference.
Here, Si|j is extractor i’s observed score under judge j; Qi is the latent quality of interest; bj is judge severity; aij is judge–extractor affinity; and εij is run variability. With one opaque judge, these components are confounded.
A fair judge-independent ranking would require every pairwise ordering to remain stable when the judge changes. That invariance has not been demonstrated. If one LLM ranks Qwen above Gemma while another ranks Gemma above Qwen, the experiment has measured judge-conditioned preference rather than an extractor-only property. Even without rank reversal, effect sizes and error classifications may change.
Using a related generative model to judge ChatGPT-derived outputs also risks self-preference, shared factual priors, and circular agreement. A fluent rationale is not itself a reproducible measurement rule: another researcher cannot mechanically reconstruct the latent decision boundary that converted a PDF passage into Correct, Partial, or Unsupported.
The claim is not that LLM-assisted evaluation is forbidden. It is that it must not be the sole unvalidated primary evaluator. LLM judging belongs in a separately versioned sensitivity layer whose repeated decisions are retained and compared with blinded human annotations.
| Threat | Effect | Primary control | Valid supplementary use |
|---|---|---|---|
| Stochasticity | Labels and ranks can change | Deterministic executable rules | Repeated judging with agreement estimates |
| Model drift | A model name does not fix a snapshot | Versioned code, thresholds, and hashes | Archive every judge request and response |
| Self-preference | Style or shared priors may be rewarded | Identity-invariant PDF support | Independent judge and randomized labels |
| Judge-model dependence | Scores describe extractor–judge pairs and rankings may reverse | One identity-invariant scoring kernel | Multiple judges, interaction estimates, and rank-stability tests |
| Prompt/order sensitivity | Presentation changes the decision | No natural-language primary judgment prompt | Counterbalanced sensitivity experiment |
| Opaque boundary | Partial versus correct is not reconstructable | Published thresholds and formulas | Human coding manual and adjudication log |
Why this protocol is academically defensible
Reproducibility is necessary, but its scope must remain explicit
The primary construct is source-supported extractive fidelity: whether output facts are visibly supported by the CV text and whether PDF-supported candidate facts are represented. For that construct, the frozen rule-based protocol is falsifiable, repeatable, auditable, and comparable across the four tested output archives. Identical evidence and identical thresholds yield an identical atomic ledger.
This does not make lexical support equivalent to perfect semantic truth. Paraphrases can be under-credited; lexical coincidence can be over-credited; PDF extraction errors can alter support; fact splitting changes the unit of analysis; and facts omitted by both ChatGPT and the tested extractor may escape the omission denominator. These are reported construct-validity limits, not hidden behind an opaque evaluator.
The academically strongest extension is layered validation: retain deterministic scoring for reproducibility; create an independent blinded human gold standard with at least two annotators, evidence spans, disagreement adjudication, and inter-rater agreement; then evaluate any LLM judge against that gold standard. The rule layer answers whether evidence is reproducibly recoverable, the human layer addresses semantic validity, and the LLM layer tests scalable approximation under measured error.
Result-provenance audit
Unsupported numerical-looking material was removed
The common evaluator was rerun from all 2,484 PDFs and the four extractor archive sets, and its complete result object was compared with the stored JSON. The workbook's common overall, category, segment, pair, document and alignment sheets match that JSON cell-by-cell. Historical sheets remain a separately labelled sensitivity tier and preserve their source-workbook values.
A manually scored HNLP/BERT sensitivity heatmap, illustrative web stage-retention percentages, and arbitrary deployment coordinates were removed. Unmeasured speed, repeated-run, and cross-dataset-stability rankings were replaced by mechanism statements and explicit experiment requirements. Conceptual MoE, causal, reasoning and factorial diagrams are labelled non-empirical.
Reproducing a value proves where it came from and how it was calculated. It does not make an automated lexical classification equivalent to blinded human adjudication.
Formal metrics
Precision, recall, F1, and completeness
Each metric inherits the evaluator’s matching, normalization, missing-output, and aggregation rules. Schema validity is structural; it cannot establish factual correctness. Lexical coverage measures recoverable source text, not semantic recall.
Comparability condition
A comparison is confirmatory only under a shared measurement process
The six required components are corpus, source representation, target ontology, matcher/adjudication, aggregation, and missing-output policy. If any component differs, κ = 0 and numerical differences are exploratory rather than causal.
Statistical treatment
Paired estimates and limitations
Weighted-F1 uncertainty uses 4,000 document-bootstrap replicates. All six model-pair category contrasts are reported descriptively. These quantify consistency under the common evaluator but do not identify whether a difference was caused by prompts, MoE routing, heading rules, dictionaries, chunk boundaries, thresholds, mapping, or backfill.