QG_1 — Documentation Review & Results-Integrity Audit

Operational S2S extreme-heat-day (EHD) forecast for Vietnam · audit run 2026-09-15 · scope: docs/RESULTS_REPORT.md, docs/OPERATIONAL_GUIDE.md, cross-checked against the evidence-pack CSVs and outputs/ehd_skill_grid.nc.

Bottom line. Every headline and stratified number in the two documents was re-derived from its source file and reproduces exactly — headline block scores, bootstrap CIs, the independent-validator re-derivation, the FDR cell counts (recomputed from the NetCDF), the season/severity strata, and the v2/§6 ensemble & vote tables. The leakage rule holds (validator PASS). Five review findings remain, none of which invalidate the load-bearing v1 result; the most important is a tension between §10 and §11 (the "ensemble beats every single member" claim excludes the CNN, which its own §11 data show is stronger at W3/W5/W6), and an unverified secondary product (the heat-wave target is defined and coded but never scored).

1. Integrity audit — numbers vs. source files

Each documented value was recomputed or read directly from the cited file. "Exact" means the document figure equals the source to the printed precision.

The two documents are also mutually consistent: the Vietnamese operational guide and the English report tell the same horizon story (skilful through W3; region/season-dependent; MAM and the south longest; long-lead flag over-forecasts) with identical numbers.

2. Review findings

Documented claim	Doc location	Source of truth	Verdict
Headline BSS / AUC / BSS-vs-raw / base rate, all 6 blocks (`hgb_recal`)	Report §4	`cv_scores_by_block.csv`	EXACT
Bootstrap 95% CI + significance flags, 6 blocks	Report §4	`cv_bootstrap_ci.csv`	EXACT
Independent re-derivation matches stage-08 (Brier/BSS/AUC/CI)	Report §7	`validator_headline_rederivation.csv`	PASS
FDR-significant cell counts 539/504/192/116/68/30 of 541	Report §5.1	recomputed from `outputs/ehd_skill_grid.nc`	EXACT
Season stratification, 24 cells (DJF/MAM/JJA/SON × 6)	Report §5.2	`cv_scores_by_season_block.csv`	EXACT
Base-rate severity stratification, 18 cells (Low/Mod/High × 6)	Report §5.3	`cv_scores_by_severity_block.csv`	EXACT
v2 best single-member SEDI per block (hgb/mlp/mlp_sev/hgb_sev)	Report §10	`v2_best_model_by_block.csv`	EXACT
v2 `ens_stack` SEDI/BSS/AUC + `ens_mean` W6	Report §10	`v2_scores_with_ensemble.csv`	EXACT
§6 vote POD/POFD/FAR/Peirce/SEDI + P₀, 6 blocks	Report §6.2–6.3	`v6_vote_scores_by_block.csv`	EXACT
§6 regional W1 Peirce 0.65 / SEDI 0.80; W5 P_thres→0.10	Report §6.4	`v6_regional_by_week.csv`	EXACT
Region names + land-cell total (Σ = 541)	Report §5.1/§12	`regional_partition.csv`, `region_centroids.csv`	EXACT
Operational-guide headline table (BSS + significance, 6 blocks)	Guide §1	`cv_scores_by_block.csv`	EXACT

F1 MEDIUM §10 "ensemble beats the best single member at every lead" is not substantiated

Files: docs/RESULTS_REPORT.md §10 · results/tables/v2_scores_with_ensemble.csv · results/tables/v6_member_scores_by_block.csv

§10 states the 9-member ensemble "lifts skill over the best single member at every lead," and its table quotes the best single member as hgb/mlp/mlp_sev/hgb_sev. But that baseline is drawn from v2_scores_with_ensemble.csv, which scores only the 7 sklearn zoo members — the CNN and deep-ANN, though members of the ensemble, are not scored as single members there. §11 and v6_member_scores_by_block.csv show the CNN single member equals or beats ens_stack SEDI at W3, W5 and W6:

block	cnn_c1 SEDI (§11 data)	ens_stack SEDI (§10)
W3	0.296	0.291	CNN higher
W5	0.216	0.203	CNN higher
W6	0.187	0.170 (ens_mean 0.175)	CNN higher

Caveat (why MEDIUM, not HIGH): the CNN SEDI comes from the stage-17 (§6) scoring pipeline, which applies each member's own per-region×block threshold; the §10 ens_stack SEDI comes from the stage-14 pipeline. The two are not a perfectly like-for-like binary comparison, so the exact crossings could shift. Regardless, §10 and §11 are internally in tension: §11 explicitly calls the CNN "the strongest single member at long lead," which contradicts §10's blanket "every lead" claim.

Recommendation: either (a) qualify §10 to "beats the best sklearn-zoo member, and is competitive with the CNN at long lead," or (b) add cnn/ann single-member rows to the stage-14 scoring so §10's baseline covers the full 9-member pool on one consistent pipeline.

F2 MEDIUM Secondary product (heat-wave, ≥3 consecutive EHDs) is defined but never verified

Files: docs/RESULTS_REPORT.md §1 · docs/OPERATIONAL_GUIDE.md §0 · src/ehd/targets.py

Both the report (§1) and the project constraints list a secondary product P(heat-wave = ≥3 consecutive EHDs) with block-boundary straddling, and the logic is coded in src/ehd/targets.py. But no verification table, figure, or skill number for the heat-wave target exists anywhere in results/, and the operational NetCDF carries only prob_ehd/ehd_flag (the primary target). A reader is led to expect a secondary deliverable that is not actually scored or shipped.

Recommendation: either verify the secondary target (even a single pooled BSS/SEDI table) and add it to §4/§8, or state plainly in §1 that the heat-wave target is defined and implemented but not yet verified/produced in this release.

F3 LOW v1 binary-score inventory is partial vs. the scoring spec

Files: docs/RESULTS_REPORT.md §8 · cv_scores_by_region_block.csv

The scoring spec (project CLAUDE.md) requires the full binary inventory POD, POFD, FAR, EDI, SEDI, CSI, ETS, HSS, Peirce, frequency bias. The v1 tables (§8, and cv_scores_by_region_block.csv) report only POD, FAR, CSI, Peirce, HSS, SEDI, freq bias — POFD, EDI and ETS are omitted for v1. (The v2 and §6 tables do include all ten.) §8 honestly lists only what it computes, so this is a completeness gap rather than a misstatement.

Recommendation: add POFD/EDI/ETS to the v1 binary tables for parity, or note in §8 that the full contingency inventory is reported for v2/§6 while v1 reports the reduced set.

F4 LOW §5.1 regional range slightly understated at W4

Files: docs/RESULTS_REPORT.md §5.1 · cv_scores_by_region_block.csv

§5.1 says the south (R6/R7) is "still ≈ +0.04–0.06 at W4–W6." The actual R6/R7 W4 BSS values are 0.070 and 0.082 — above the stated 0.06 ceiling. The true W4–W6 range for R6/R7 is ≈ 0.04–0.08. Minor; the qualitative point (south retains positive skill at long lead) is correct.

Recommendation: change "≈ +0.04–0.06" to "≈ +0.04–0.08 (highest at W4)".

F5 INFO Scope notes — Stage 2 plan & operational archive

Files: docs/STAGE2_PLAN.md · outputs/operational/

STAGE2_PLAN.md (newest doc) is a plan for a not-yet-run "Stage 2" (prompt_2, member registry / pruning / expansion / real-time case) and explicitly awaits a "go". It is not part of the current results and introduces no inconsistency — flagged only so it isn't mistaken for delivered work.
The operational archive under outputs/operational/ is a 3-init 2022 demo (20220102/05/09), not a full archive. The docs correctly describe the produce command without claiming a complete archive, so this is consistent — just confirming the known "optional remaining" item.

3. Leakage-rule check

The load-bearing constraint holds. The independent validator (11_independent_validator.py) re-reads the raw LOYO NPZ, recomputes the headline block scores in plain numpy, re-checks fold coverage, and its output (validator_headline_rederivation.csv) agrees with stage-08 to the printed precision on every block (Brier, BSS, AUC identical; CIs within bootstrap tolerance). All 6 blocks reproduce and the significance verdicts match (W1–W3 significant, W4–W6 not).

4. Verdict

The documents are numerically trustworthy: no fabricated or drifted numbers were found across 11 checked claim-groups. Before the paper/handoff, address F1 (qualify or re-score the §10 "beats every single member" claim so it is consistent with §11's CNN result) and F2 (verify or explicitly de-scope the heat-wave secondary product). F3–F4 are polish; F5 is informational.