# Master Prompt: Academic Evaluation of Local-LLM CV Extraction

You are acting as a senior academic data analyst, information-extraction researcher, and Microsoft Excel reporting specialist.

Analyze the three attached ZIP archives and deliver a complete Microsoft Excel `.xlsx` workbook evaluating the success and failure of the local LLM extraction only.

## 1. Input archives and their roles

The three attached ZIP archives have different roles.

### Archive 1: Original PDF archive

- Contains the original CV documents in PDF format.
- Treat the PDFs as the primary source of evidence.
- Use the PDFs as the closest available approximation to ground truth.

### Archive 2: ChatGPT extracted JSON archive

- The filename normally contains `_extracted_json.zip`.
- Contains a previous ChatGPT attempt to convert each PDF into an `aiextract.json` structure.
- Use this archive only as a secondary reference and diagnostic comparator.
- Do not treat this extraction as ground truth.
- Do not use agreement with this extraction as proof that the local LLM is correct.

### Archive 3: Local-LLM extracted JSON archive

- The filename normally contains `-local-llm-extract.zip`.
- This is the extraction system being evaluated.
- Evaluate its results against the corresponding original PDFs.
- The actual extraction object may be located under:

`strategy_outputs.llm`

- Unwrap this object before evaluating its schema and content.
- Retain relevant pipeline metadata, including:

`strategy_status.llm`

`active_match_strategy`

`strategy_provenance.llm`

The primary evaluation concerns the following transformation:

Original PDF ? Local-LLM extracted JSON

The ChatGPT extraction may only be used for diagnostic comparison and disagreement analysis.

## 2. Required final deliverable

Create and return one native Microsoft Excel workbook named:

`CV_Local_LLM_Extraction_Evaluation.xlsx`

The workbook must meet the following requirements:

- Microsoft Excel `.xlsx` format.
- Do not provide CSV files as the principal delivery.
- Do not use macros.
- Do not create external workbook links.
- Use native Excel tables, formulas, charts, filters, conditional formatting, and freeze panes.
- Ensure that all charts remain editable in Microsoft Excel.
- Use document IDs or source filenames instead of candidate names in dashboards.
- Minimize the display of personally identifiable information from the CVs.
- Provide a clickable download link to the completed workbook.

Do not provide only a written analysis. The completed Excel workbook is the required output.

## 3. Research objective

The principal research question is:

To what extent does the local LLM successfully transform information contained in original CV PDFs into valid, complete, accurate, and non-hallucinated structured JSON records?

Evaluate the local extraction across the following dimensions:

1. File and pipeline success.
2. JSON validity.
3. Schema conformity.
4. Field and segment coverage.
5. Factual accuracy.
6. Extraction completeness.
7. Precision.
8. Recall.
9. F1 score.
10. Partial correctness.
11. Unsupported or hallucinated content.
12. Normalization quality.
13. Cross-document consistency.
14. Error types and their frequencies.
15. Agreement with the reference extraction, clearly separated from PDF-grounded accuracy.

Do not reduce the evaluation to one overall score. Report the different quality dimensions separately.

## 4. Reproducibility rules

Apply the following rules identically whenever this prompt is used on a different dataset.

### 4.1 Document matching

Match records using a normalized document ID derived from filenames.

Example:

Original PDF:

`10554236.pdf`

Local extraction:

`10554236.pdf.json`

Reference extraction:

`10554236.json`

Normalized document ID:

`10554236`

Do not match records by candidate name, job title, archive order, or the order in which files are read.

Create an inventory containing:

- Document ID.
- PDF filename.
- Local JSON filename.
- Reference JSON filename.
- PDF present.
- Local extraction present.
- Reference extraction present.
- Match status.
- Duplicate status.
- File-parsing status.

Report all unmatched, duplicate, missing, and malformed files explicitly.

### 4.2 No silent corrections

Do not silently repair malformed JSON or change extracted values before scoring.

When a technically safe repair is necessary to continue the analysis:

- Preserve the original value.
- Record the repair in the error log.
- Mark the record as technically repaired.
- Report results before and after repair when the repair affects the evaluation score.

### 4.3 Fixed evaluation direction

The local extraction is the prediction.

The PDF is the primary evidentiary source.

Apply the following rules:

- Information supported by the PDF and correctly represented by the local extraction is a correct extraction.
- Information present in the local extraction but unsupported by or contradictory to the PDF is an unsupported or hallucinated extraction.
- Information clearly present in the PDF but absent from the local extraction is a missed extraction.
- Agreement with the ChatGPT reference extraction does not override PDF evidence.
- When the local and reference extractions agree but both conflict with the PDF, classify the local result as incorrect.
- When the local and reference extractions disagree, use the PDF to adjudicate the disagreement.

### 4.4 Evidence preservation

For every evaluated fact, retain sufficient information to audit the judgment.

Record:

- Document ID.
- PDF page number, where available.
- PDF section or heading.
- Short source excerpt.
- Expected PDF-supported value.
- Local extracted value.
- Reference extracted value, where available.
- Evaluation status.
- Error category.
- Evaluator explanation.
- Evaluator confidence.

Use only short excerpts. Do not reproduce entire CVs in the workbook.

### 4.5 Consistent normalization

Before comparing values, apply the same normalization rules to all datasets:

- Remove leading and trailing spaces.
- Collapse repeated whitespace.
- Compare text case-insensitively unless capitalization changes the meaning.
- Normalize superficial punctuation differences.
- Normalize equivalent date formats.
- Normalize numerical and written month names.
- Normalize common abbreviations when their meaning is unambiguous.
- Treat semantically equivalent wording as equivalent when the meaning is unchanged.
- Do not treat substantively different employers, job titles, institutions, qualifications, dates, skills, certifications, or languages as equivalent.
- Preserve all original values in separate columns.

Do not use aggressive fuzzy matching that could conceal extraction errors.

Document all normalization rules in the workbook.

## 5. Evaluation levels

Use three evaluation levels.

### Level A: Pipeline and file level

Evaluate whether each local extraction:

- Exists.
- Can be opened.
- Contains valid JSON.
- Has `strategy_status.llm = done`, when the wrapper exists.
- Contains an accessible `strategy_outputs.llm` object.
- Contains the expected extraction structure.
- Can be matched to an original PDF.
- Can be included in the content evaluation.

### Level B: Segment or field-family level

Where available in the observed schema, evaluate these field families:

- Profile.
- Experience.
- Education.
- Skills.
- Certifications.
- Languages.
- Seniority.
- Category classification.
- Document metadata.
- Normalized fields.

Use the actual schema found in the attached files.

Do not invent fields merely to make different datasets appear identical.

When schemas differ between datasets, create a documented field-mapping table and distinguish:

- Exact schema matches.
- Mappable fields.
- Dataset-specific fields.
- Unavailable fields.

### Level C: Atomic-fact level

Break structured and list-based sections into atomic facts.

Examples of atomic facts include:

- One employer.
- One job title.
- One employment date range.
- One responsibility.
- One achievement.
- One degree.
- One educational institution.
- One education date.
- One skill.
- One certification.
- One language.
- One seniority classification.
- One category label.

Evaluate atomic facts separately.

Do not classify an entire list or segment as completely correct merely because some elements are correct.

Do not classify an entire list as completely incorrect merely because one element is incorrect.

## 6. Atomic-fact classification

Assign exactly one primary status to each evaluated atomic fact.

### Correct

Use this status when:

- The local value is explicitly or unambiguously supported by the PDF.
- Any normalization difference is superficial and does not change the meaning.

### Partially correct

Use this status when:

- The local value captures some relevant information but is incomplete, overly broad, slightly inaccurate, or combines correct and incorrect elements.

Examples include:

- Correct employer but incorrect employment date.
- Correct qualification but missing specialization.
- Correct job title with an unsupported seniority modifier.
- Correct skill category but incomplete skill wording.

### Unsupported

Use this status when:

- The local extraction contains a fact that cannot be supported by the PDF.

This includes hallucinated:

- Employers.
- Job titles.
- Institutions.
- Degrees.
- Dates.
- Certifications.
- Languages.
- Skills.
- Responsibilities.
- Achievements.
- Categories.

### Contradicted

Use this status when:

- The local extracted value directly conflicts with the PDF.

### Missing

Use this status when:

- A relevant fact is clearly present in the PDF but absent from the local extraction.

### Not applicable

Use this status when:

- The field or fact is genuinely not present in the PDF and should not have been extracted.

### Unverifiable

Use this status when:

- PDF quality, layout, language, ambiguity, or technical limitations prevent a reliable judgment.

Do not convert unverifiable facts into successes or failures.

Exclude unverifiable facts from precision, recall, and F1 calculations, but report their number and percentage separately.

## 7. Quantitative metrics

Calculate metrics at the following levels:

- Entire dataset.
- Individual document.
- Segment or field family.
- Error category.
- Relevant document subgroup, when appropriate.

### 7.1 Count definitions

For strict metrics, define:

`TP = Correct local facts`

`FP = Unsupported local facts + Contradicted local facts`

`FN = Missing PDF facts`

`PARTIAL = Partially correct facts`

### 7.2 Strict precision

Calculate:

`Strict Precision = TP / (TP + FP + PARTIAL)`

A partially correct prediction must not count as fully correct in the strict score.

### 7.3 Strict recall

Calculate:

`Strict Recall = TP / (TP + FN + PARTIAL)`

### 7.4 Strict F1 score

Calculate:

`Strict F1 = 2 × Strict Precision × Strict Recall / (Strict Precision + Strict Recall)`

Return `N/A`, not zero, when the denominator is genuinely undefined.

### 7.5 Partial-credit metrics

Also calculate partial-credit metrics using a fixed partial-credit weight of `0.5`.

Calculate:

`Weighted TP = TP + (0.5 × PARTIAL)`

`Weighted Precision = Weighted TP / (TP + PARTIAL + FP)`

`Weighted Recall = Weighted TP / (TP + PARTIAL + FN)`

`Weighted F1 = 2 × Weighted Precision × Weighted Recall / (Weighted Precision + Weighted Recall)`

Keep strict and weighted metrics in separate columns.

Do not replace strict metrics with weighted metrics.

### 7.6 Completeness

Calculate:

`Completeness = (TP + PARTIAL) / (TP + PARTIAL + FN)`

Completeness measures whether relevant PDF information was represented, regardless of whether partially extracted information was completely accurate.

### 7.7 Unsupported-content rate

Calculate:

`Unsupported Rate = FP / (TP + PARTIAL + FP)`

### 7.8 Exact factual accuracy

Calculate:

`Exact Accuracy = TP / (TP + PARTIAL + FP + FN)`

### 7.9 Schema completeness

For each record, calculate:

`Schema Completeness = Populated applicable schema elements / Expected applicable schema elements`

Do not penalize fields that are genuinely not applicable to the PDF.

### 7.10 Pipeline success rate

Calculate:

`Pipeline Success Rate = Evaluable local JSON records / Matched PDF records`

A local record is evaluable only when it:

- Exists.
- Is parseable.
- Contains an accessible extraction object.
- Can be matched to a PDF.

### 7.11 Critical hallucination rate

Calculate:

`Critical Hallucination Rate = Documents containing at least one critical unsupported or contradicted claim / Evaluable documents`

Critical claims include unsupported or contradicted:

- Candidate identity.
- Employer.
- Job title.
- Educational institution.
- Degree.
- Certification.
- Employment date.
- Education date.

## 8. Operational document outcome

Use the same thresholds for every dataset unless the user explicitly changes them before analysis.

Store all thresholds in a visible Excel Parameters worksheet.

The thresholds are operational research thresholds and must not be presented as universal scientific standards.

### Successful extraction

Classify a document as `Successful` only when all the following are true:

- Local JSON exists and is parseable.
- The extraction object is accessible.
- No critical unsupported or contradicted identity, employer, job-title, institution, degree, certification, or date claim is present.
- Weighted F1 is at least `0.90`.
- Completeness is at least `0.90`.
- Unsupported-content rate is no greater than `0.05`.

### Partially successful extraction

Classify a document as `Partially successful` when:

- It is technically evaluable.
- It does not satisfy every successful-extraction criterion.
- Weighted F1 is at least `0.60`, or completeness is at least `0.60`.

### Failed extraction

Classify a document as `Failed` when any of the following applies:

- Local extraction is missing.
- JSON cannot be parsed.
- The expected local extraction object is absent.
- The local extraction cannot be matched to a PDF.
- Weighted F1 is below `0.60`.
- Completeness is below `0.60`.
- A major proportion of the output is unsupported or contradicted.
- A critical pipeline or schema failure prevents meaningful evaluation.

Create separate columns for:

- Technical outcome.
- Content outcome.
- Final combined outcome.

This distinction must prevent a technically valid JSON file from being interpreted as a factually successful extraction.

## 9. Schema evaluation

Check whether the unwrapped local extraction follows the observed `aiextract.json` structure.

Expected top-level elements may include:

- `document_metadata`
- `extraction_run_metadata`
- `normalized`
- `segments`
- `warnings`

Expected normalized fields may include:

- `category_labels`
- `languages`
- `seniority_level`
- `years_experience_estimate`

Expected segment fields may include:

- `segment`
- `summary_raw`
- `normalized_values`
- `source_section_heading`
- `source_char_start`
- `source_char_end`
- `extraction_method`
- `extractor_name`
- `extractor_version`
- `notes`

Evaluate:

- Required-element presence.
- Correct data types.
- Empty versus populated fields.
- Duplicate segments.
- Invalid segment names.
- Internal contradictions.
- Invalid or implausible character offsets.
- Metadata consistency.
- Schema-version consistency.
- Warning usage.
- Extraction provenance.
- Wrapper accessibility.
- Pipeline-status consistency.

Distinguish schema validity from factual accuracy.

A structurally valid JSON record may still contain inaccurate or hallucinated content.

A factually useful record may also have schema defects.

Report these dimensions separately.

## 10. Reference extraction rules

Use the ChatGPT extraction archive only for a separate diagnostic analysis.

For every comparable local and reference field, classify:

- Agreement.
- Partial agreement.
- Disagreement.
- Local only.
- Reference only.
- Neither populated.
- Not comparable.

When the systems disagree, adjudicate using the PDF.

Create separate variables for:

- Local-reference agreement.
- Local PDF-grounded correctness.
- Reference PDF-grounded correctness, only when required to explain a disagreement.

Do not include reference agreement in the main local success score.

Display the following statement prominently in the reference-diagnostics worksheet:

“Agreement between two extraction systems measures consistency, not necessarily correctness.”

Primary dashboard graphs must concern local-versus-PDF performance.

Reference-comparison results must be placed in a separate diagnostic worksheet or secondary dashboard section.

## 11. Error taxonomy

Use a stable error taxonomy across datasets.

At minimum, include:

- Missing local file.
- Invalid JSON.
- Missing extraction wrapper.
- Pipeline-status failure.
- Schema element missing.
- Wrong data type.
- Empty field.
- Missing segment.
- Duplicate segment.
- Omitted fact.
- Unsupported fact.
- Contradicted fact.
- Partially correct fact.
- Incorrect employer.
- Incorrect job title.
- Incorrect institution.
- Incorrect degree.
- Incorrect employment date.
- Incorrect education date.
- Incorrect duration.
- Incorrect skill.
- Incorrect certification.
- Incorrect language.
- Incorrect seniority.
- Incorrect category.
- Over-generalization.
- Under-specification.
- Normalization error.
- Segmentation error.
- Cross-field inconsistency.
- Source-offset error.
- PDF unreadable.
- Evaluation uncertainty.
- Other.

When `Other` is selected, require a written explanation.

Use the following severity levels:

- Critical.
- Major.
- Moderate.
- Minor.

## 12. Required Excel workbook structure

Create the following worksheets in this exact order.

### 00_Read_Me

Explain in accessible academic language:

- The purpose of the evaluation.
- Which extraction system is being evaluated.
- The role of each archive.
- Why the PDFs are the primary evidence.
- Why the ChatGPT extraction is reference-only.
- What successful, partially successful, and failed mean.
- What the user should expect from the workbook.
- How to use Excel filters.
- How to interpret percentages.
- Important methodological limitations.
- Privacy and personal-data considerations.

### 01_Executive_Summary

Include:

- Number of original PDFs.
- Number of local extraction files.
- Number of reference extraction files.
- Number of matched records.
- Number of unmatched records.
- Number of evaluable records.
- Number and percentage of successful documents.
- Number and percentage of partially successful documents.
- Number and percentage of failed documents.
- Overall strict precision.
- Overall strict recall.
- Overall strict F1.
- Overall weighted precision.
- Overall weighted recall.
- Overall weighted F1.
- Overall completeness.
- Overall unsupported-content rate.
- Critical hallucination rate.
- Best-performing segment.
- Worst-performing segment.
- Three most frequent error types.
- Concise academic interpretation.
- Main methodological limitations.

Where practical, summary values must link to detailed sheets using Excel formulas.

### 02_Parameters

Record:

- Prompt version.
- Analysis date.
- Dataset name.
- Success weighted-F1 threshold.
- Partial-success weighted-F1 threshold.
- Success completeness threshold.
- Partial-success completeness threshold.
- Maximum unsupported rate.
- Partial-credit weight.
- Filename-matching rule.
- Normalization rules.
- Metric definitions.
- Undefined-denominator policy.
- Critical-error definition.
- Histogram bin boundaries.
- Archive filenames.
- Archive SHA-256 hashes, where technically available.

Use this worksheet to make repeated analyses comparable.

### 03_File_Inventory

Create one row per normalized document ID.

Include:

- Document ID.
- PDF filename.
- PDF present.
- Local filename.
- Local present.
- Reference filename.
- Reference present.
- Three-file match status.
- Duplicate indicator.
- Local JSON parse status.
- Local pipeline status.
- Extraction object found.
- Schema-valid indicator.
- Evaluation eligibility.
- Exclusion reason.
- Technical repair indicator.
- Technical repair description.

### 04_Document_Results

Create one row per PDF or document.

Include:

- Document ID.
- Pipeline status.
- Technical outcome.
- Schema completeness.
- TP.
- Partial.
- FP.
- FN.
- Unverifiable count.
- Strict precision.
- Strict recall.
- Strict F1.
- Weighted precision.
- Weighted recall.
- Weighted F1.
- Completeness.
- Unsupported rate.
- Exact accuracy.
- Critical-error count.
- Critical hallucination indicator.
- Content outcome.
- Final combined outcome.
- Principal failure reason.
- Evaluator confidence.
- Notes.

Use Excel formulas for calculated metrics and outcome classifications wherever practical.

### 05_Segment_Results

Create one row per segment or field family.

Include:

- Segment.
- Number of applicable documents.
- Number populated by the local LLM.
- TP.
- Partial.
- FP.
- FN.
- Unverifiable count.
- Strict precision.
- Strict recall.
- Strict F1.
- Weighted precision.
- Weighted recall.
- Weighted F1.
- Completeness.
- Unsupported rate.
- Exact accuracy.
- Most common error.
- Interpretation.

### 06_Atomic_Evidence

Create one row per evaluated atomic fact.

Include:

- Document ID.
- Segment.
- Field.
- Atomic-fact ID.
- PDF page.
- PDF section.
- Short PDF evidence excerpt.
- Expected or PDF-supported fact.
- Local extracted fact.
- Reference extracted fact.
- Original PDF value.
- Original local value.
- Original reference value.
- Normalized PDF value.
- Normalized local value.
- Normalized reference value.
- Evaluation status.
- Error category.
- Severity.
- Critical-error indicator.
- Confidence.
- Evaluator explanation.

Avoid displaying unnecessary names, addresses, phone numbers, email addresses, or other personal identifiers.

### 07_Error_Log

Create one row per identified error.

Include:

- Error ID.
- Document ID.
- Segment.
- Field.
- Error category.
- Severity.
- Local value.
- PDF-supported value.
- Reference value.
- Description.
- Likely cause.
- Consequence for the analysis.
- Recommended remediation.

### 08_Reference_Diagnostics

Include local-versus-reference agreement results.

Display prominently:

“This worksheet measures agreement between extraction systems. It does not establish factual correctness.”

Include:

- Document ID.
- Segment.
- Field.
- Local value.
- Reference value.
- Agreement status.
- PDF adjudication result.
- Which extraction is PDF-supported.
- Notes.

### 09_Dashboard

Create a visually coherent academic dashboard using local-versus-PDF results only.

Include:

- Headline KPI cards.
- Document-outcome summary.
- Pipeline summary.
- Precision, recall, and F1 summary.
- Completeness and hallucination indicators.
- Required graphs described below.
- Clear chart titles.
- Consistent percentage scales.
- Filters or slicers where supported.

Do not display unnecessary personal information.

### 10_Chart_Explanations

For every graph, provide:

- Chart number.
- Chart title.
- Research question answered.
- Variables shown.
- Numerator.
- Denominator.
- Reason for selecting the visualization.
- How to interpret high values.
- How to interpret low values.
- Strongest visible pattern.
- Important exception or outlier.
- Interpretation limitation.
- Dataset-specific academic interpretation.

### 11_Methodology_Limitations

Explain:

- Evaluation design.
- Unit of analysis.
- Document-matching procedure.
- Evidence-adjudication procedure.
- Atomic-fact methodology.
- Strict metrics.
- Weighted metrics.
- Operational thresholds.
- Construct validity.
- Measurement reliability.
- Internal validity.
- External validity.
- Potential evaluator subjectivity.
- Ambiguous or unreadable PDFs.
- Semantic-normalization limitations.
- Why reference agreement is not accuracy.
- Potential occupation or domain bias.
- Generalizability limitations.
- Privacy and ethical considerations.
- Reproducibility provisions.

### 12_Data_Dictionary

Define every important:

- Worksheet.
- Column.
- Metric.
- Formula.
- Status.
- Threshold.
- Error category.
- Severity level.
- Outcome classification.
- Abbreviation.

## 13. Required Excel graphs

Create the following native Excel visualizations.

Each visualization must have:

- A descriptive title.
- Axis titles where relevant.
- A legend where relevant.
- Data labels where readable.
- Clearly documented denominators.
- Visible source data in an Excel worksheet.
- A corresponding explanation in `10_Chart_Explanations`.

### Graph 1: Overall extraction outcomes

Chart type:

Donut chart or pie chart.

Show:

- Successful.
- Partially successful.
- Failed.

Purpose:

Show the proportion of documents in each final outcome category.

Also display the underlying document counts.

Do not rely only on percentages.

### Graph 2: Processing and evaluation pipeline

Chart type:

Column chart.

Show counts at each stage:

- Original PDFs.
- Matched local files.
- Parseable local JSON files.
- Local extraction objects found.
- Schema-valid records.
- Evaluable records.
- Successful records.

Purpose:

Show where records are lost, excluded, or rejected during the extraction and evaluation pipeline.

### Graph 3: Segment-level precision, recall, and F1

Chart type:

Clustered column chart.

For each segment, plot:

- Strict precision.
- Strict recall.
- Strict F1.

Purpose:

Show whether each segment mainly suffers from unsupported extractions, missed information, or both.

Use a consistent zero-to-one or zero-to-100-percent scale.

### Graph 4: Segment-level extraction status

Chart type:

100-percent stacked bar chart.

For each segment, show percentages of:

- Correct.
- Partially correct.
- Unsupported or contradicted.
- Missing.

Exclude `Not applicable` from the denominator.

Report `Unverifiable` separately.

Purpose:

Compare the composition of extraction results across field families.

### Graph 5: Completeness versus factual quality

Chart type:

Scatter plot.

For every document:

- X-axis: Completeness.
- Y-axis: Weighted F1.
- Document ID must be available as a point label or in the chart source table.
- Final outcome category must be included in the supporting data.

Purpose:

Distinguish documents that extracted substantial information inaccurately from documents that extracted less information accurately.

Do not interpret correlation as causation.

### Graph 6: Error Pareto chart

Chart type:

Combination chart.

Show:

- Bars representing the frequency of each error category.
- Error categories ordered from most frequent to least frequent.
- A line representing the cumulative percentage of all errors.

Purpose:

Identify the small number of error categories responsible for the largest proportion of observed extraction failures.

### Graph 7: Distribution of document performance

Chart type:

Histogram.

Variable:

- Document-level weighted F1.

Purpose:

Show whether document performance is concentrated, widely dispersed, bimodal, skewed, or affected by low-performing outliers.

Use fixed bin boundaries.

Record the bin boundaries in the Parameters worksheet so they remain consistent across datasets.

Recommended bins:

- 0.00 to below 0.10.
- 0.10 to below 0.20.
- 0.20 to below 0.30.
- 0.30 to below 0.40.
- 0.40 to below 0.50.
- 0.50 to below 0.60.
- 0.60 to below 0.70.
- 0.70 to below 0.80.
- 0.80 to below 0.90.
- 0.90 to 1.00.

### Graph 8: Document-by-segment performance heatmap

Visualization type:

Excel conditional-formatting heatmap.

Rows:

- Document IDs.

Columns:

- Segment or field families.

Values or statuses:

- Correct or high-performing.
- Partial.
- Failed, unsupported, or missing.
- Not applicable.
- Unverifiable.

Purpose:

Reveal recurring document-level and segment-level patterns that aggregate graphs may conceal.

Include a visible legend.

Use the same status rules and thresholds across datasets.

### Graph 9: Schema quality versus content quality

Chart type:

Clustered bar chart.

Compare:

- Schema completeness.
- Content completeness.
- Weighted F1.

Present the comparison by segment or another clearly documented aggregation.

Purpose:

Demonstrate that structural validity and factual quality are different constructs.

A structurally complete JSON record is not necessarily factually correct.

### Optional Graph 10: Reference agreement versus PDF-grounded accuracy

Chart type:

Scatter plot or clustered column chart.

Compare:

- Local-reference agreement rate.
- Local PDF-grounded weighted F1.

Purpose:

Demonstrate whether agreement with the reference extraction corresponds to factual accuracy.

Place this graph only in the reference-diagnostics section.

Do not place it among the primary local-LLM success graphs without a clear warning that agreement is not correctness.

## 14. Graph interpretation requirements

In `10_Chart_Explanations`, explain each graph in complete academic prose.

Do not merely repeat the graph title.

For every visualization:

1. Explain what is being measured.
2. Define the numerator.
3. Define the denominator.
4. Explain why the selected visualization is suitable.
5. Explain what a high value means.
6. Explain what a low value means.
7. Identify the strongest visible pattern.
8. Identify an important exception or outlier.
9. Explain whether the graph concerns:
   - Pipeline performance.
   - Structural quality.
   - Content accuracy.
   - Completeness.
   - Hallucination.
   - Error concentration.
   - Distributional variation.
10. State at least one limitation affecting interpretation.

Use the following interpretive distinctions:

- High recall with low precision means the extractor captures much of the PDF content but also introduces unsupported information.
- High precision with low recall means extracted information is usually correct, but a substantial amount of PDF content is omitted.
- High schema completeness with low F1 means the JSON is structurally complete but factually unreliable.
- High completeness with a high unsupported rate means the system extracts a large amount of information but introduces unsupported content.
- High local-reference agreement does not necessarily mean high PDF-grounded accuracy.
- A high failure percentage may reflect pipeline failures, content-quality failures, or both; distinguish them.
- A histogram shows the distribution of performance and should not be interpreted as identifying causes.
- A Pareto chart identifies frequent errors but does not prove that frequency is equivalent to severity.
- A scatter plot may indicate an association but does not demonstrate causality.

## 15. Academic discussion requirements

Write the workbook narrative at a high theoretical and academic level while keeping it understandable.

Use relevant concepts such as:

- Information extraction.
- Construct validity.
- Measurement reliability.
- Ground-truth approximation.
- Precision-recall trade-off.
- Hallucination.
- Omission error.
- Semantic equivalence.
- Schema conformity.
- Data provenance.
- Error propagation.
- Reproducibility.
- Internal validity.
- External validity.
- Adjudication uncertainty.
- Operationalization.
- Unit of analysis.
- Missing-data mechanisms.
- Classification thresholds.
- Structural validity.
- Factual validity.

Avoid overstating the findings.

Use academically cautious language, including:

- “The evidence indicates…”
- “Under the operational thresholds used…”
- “The observed result should be interpreted with caution because…”
- “Agreement should not be equated with correctness…”
- “The PDFs function as the primary evidentiary source rather than a perfectly annotated gold standard…”
- “The findings describe performance within the attached dataset and should not automatically be generalized to other CV populations…”

Do not claim that this evaluation establishes universal local-LLM performance beyond the attached dataset.

## 16. Comparison across future datasets

The workbook must support comparisons with workbooks generated from other datasets using the same prompt.

Use exactly the same:

- Worksheet names.
- Column names.
- Metric formulas.
- Outcome categories.
- Partial-credit weight.
- Success thresholds.
- Failure thresholds.
- Error taxonomy.
- Severity taxonomy.
- Histogram bins.
- Graph titles.
- Graph definitions.
- Percentage scales.

Include a dataset-summary table with one row containing:

- Dataset name.
- Analysis date.
- PDF count.
- Matched count.
- Evaluable count.
- Pipeline success rate.
- Successful percentage.
- Partially successful percentage.
- Failed percentage.
- Strict precision.
- Strict recall.
- Strict F1.
- Weighted precision.
- Weighted recall.
- Weighted F1.
- Completeness.
- Unsupported rate.
- Critical hallucination rate.
- Schema completeness.
- Unverifiable percentage.

Design this table so that the row can be copied into a future cross-dataset comparison workbook without changing its structure.

Do not change metric definitions between datasets based on observed performance.

## 17. Quality controls

Before delivering the workbook, verify all of the following:

- Every original PDF is represented in the inventory.
- Filename matching uses normalized document IDs.
- Local wrapper objects have been correctly unwrapped.
- The local extraction is the only system receiving the primary success or failure classification.
- The reference extraction is not treated as ground truth.
- Every metric follows the formulas stated in this prompt.
- Percentages use consistent formatting.
- Undefined metrics show `N/A` rather than misleading zeros.
- Counts displayed in graphs match detailed worksheet tables.
- Every graph has a corresponding explanation.
- Every failed record has a documented failure reason.
- Every unsupported or contradicted fact is traceable to the evidence table.
- All Excel formulas are valid.
- No formula cells contain errors such as `#DIV/0!`, `#VALUE!`, `#NAME?`, or `#REF!`.
- Filters are applied to detailed data tables.
- Freeze panes are applied where useful.
- Column widths are readable.
- Long text columns use text wrapping.
- Personally identifiable information is minimized in dashboards.
- The workbook opens correctly in Microsoft Excel.
- All graphs are visible and editable.
- All graph source data is retained.
- The final response includes a clickable `.xlsx` download link.

## 18. Handling technical limitations

If direct PDF evidence cannot be reliably accessed for a subset of documents:

- Do not score those documents as factually correct or incorrect without evidence.
- Mark affected facts as `Unverifiable`.
- Exclude unverifiable facts from precision, recall, and F1 denominators.
- Report the number and percentage of unverifiable documents and facts.
- Explain the limitation in the executive summary and methodology worksheet.

If only structural analysis is technically possible:

- Clearly state that content accuracy was not established.
- Do not replace PDF-grounded evaluation with local-reference agreement.
- Do not classify documents as factually successful solely because the JSON is valid.

Continue processing accessible records even when individual files fail.

Do not stop the entire analysis because one document is malformed, missing, or unreadable.

Record every excluded record and its exclusion reason.

## 19. Final response requirements

After completing the analysis:

1. Provide the clickable Excel workbook download link.
2. State the number of original PDFs.
3. State the number of local extraction files.
4. State the number of reference extraction files.
5. State the number of matched records.
6. State the number of evaluable records.
7. Confirm that the local LLM was evaluated against the original PDFs.
8. Confirm that the ChatGPT extraction was used only as a reference.
9. Mention important technical or methodological limitations.
10. Do not present unsupported conclusions outside the workbook.