# MEMORY.md — lessons, decisions, confirmations, dead ends

One entry per lesson. Update entries rather than duplicating; delete an entry once shown
wrong. Do not restate what the repository/report already records. Newest decisions on top.

## PROJECT STATUS (2026-09-14) — resume point
**v2 (severity + cascade zoo) is RUN and VERIFIED** — see L-12. 20-fold LOYO on SLURM →
`outputs/loyo_pred_v2/pred_v2_*.npz`; pooled 2003–2022 OOS binary skill in
`results/tables/v2_cv_scores_by_model_block.csv` + `v2_best_model_by_block.csv`. Headline
unchanged (skilful to ~W3). Remaining v2 optional: cnn family (not yet run) + write-up into
`docs/RESULTS_REPORT.md`. v1 status below unchanged.

## PROJECT STATUS (2026-09-13) — v1 resume point
Pipeline complete end-to-end through cross-validated verification + regional products.
DONE: data inventory & reconciliation; climatology/Tx90p/EHD index + threshold diagnostic;
gridded-vs-station reconciliation; forecast daily-Tx extraction (all 2080 inits, 20 yr);
calibration families (single-member MOS: raw/climat/logistic/hgb); isotonic recalibration;
per-region×block decision thresholds; **full 20-fold LOYO** (`outputs/loyo_pred/pred_*.npz`);
pooled CV scores + whole-year block bootstrap; **per-cell FDR field significance**; regional
partition table + publication maps + `outputs/ehd_skill_grid.nc`. Tests 10/10 pass.
HEADLINE: significant Brier skill vs climatology to **W3 (day 24)**; none at/after W4 (L-10).
Spatial gradient: south skilful longest (L-11).
REMAINING: (a) season/severity stratification in `08_verify.py`; (b) operational per-init
gridded CF-NetCDF product from a PRODUCTION model trained on all years (real-time path);
(c) written deliverables — operational guide (VN+EN), results report, evidence pack, and an
independent validator re-derivation of headline scores. Full run guide + resume detail in
CLAUDE.md "CURRENT STATUS". Read the CSVs in results/tables/ for numbers; do not recall them.

## Summary (one line each)
- L-01 **No ensemble**: reforecast is control-only (1 member) → calibration re-framed as single-member distributional MOS. [confirmed]
- L-02 **Forecast already on 0.25° obs grid** → downscaling stage dropped; upstream interp documented as limitation. [confirmed]
- L-03 **Forecast Tx is 6-hourly mx2t6** → daily Tx = daily max of 4 values; day 0 = first 24 h after init. [confirmed]
- L-04 **Obs calendar 365_day** (no 29 Feb) vs forecast gregorian → match on (month, day), drop 29 Feb from forecast Tx. [confirmed]
- L-05 **Region ids 1–7 available gridded** (`reg` in mask) and per-station (header field 4) → zonal stats via gridded mask; shapefile only for map. [confirmed]
- L-06 **Station folder is multi-variable**: Tx_ (200) is the target; R_ is rainfall, not Tx. [confirmed]
- L-07 **Gridded = calibration target, station = independent validation**; stations run ~+1.5–2.2°C warmer → station EHD freq 2–5× gridded. [confirmed]
- L-08 **Forecast daily Tx extracted** for all 2080 inits (20 yr) to outputs/forecast_daily_tx/; day d = max of 4 mx2t6, valid=init+d. [confirmed]
- L-09 **Single-fold dev result (holdout 2022)**: superseded by L-10 full CV. [superseded]
- L-10 **HEADLINE CV result (20-fold LOYO)**: significant Brier skill vs climatology to W3 (day 24); no skill at/after W4. Calibration beats raw at every lead. [confirmed]
- L-11 **Spatial/regional skill gradient**: FDR-significant cells 539/504/192/116/68/30 (W1..W6 of 541); tropical south (R6,R7) keeps skill longest, north loses it by W3. [confirmed]
- L-12 **v2 severity+cascade zoo run**: 20-fold LOYO + pooled 2003–2022 out-of-sample binary skill done on SLURM; best-by-SEDI W1 hgb 0.678, W2 mlp 0.414, W3 mlp 0.256, decaying to W6 0.153; LV1 easy-negative screening (cutoff 0.90/0.99/1.0) is ~neutral for skill. [confirmed]
- L-13 **v2 EHD_ens + maps + HTML (Phase A)**: ens_stack (short lead) + ens_mean (long lead) beat best single member; per-cell skill maps (`outputs/v2_skill_maps.nc`) + `results/v2_report.html`. CNN/ANN = Phase B pending. [confirmed]
- L-14 **Stage-2 member registry + nested pruning**: registry v1 = 23 members (9 families); nested leakage-safe pruning is a NULL result except at W2 (6 linear + ann_torch consensus-pruned, ensemble PSS 0.303→0.309); W3–W6 keep the full pool; optimism gap ≤0.004 (CV validated). Expansion v3 (+7 tele/bal members) submitted to SLURM (job 1401305). [confirmed]

---

### L-14 — Stage 2: member registry (v1/v2) + nested leakage-safe pruning
**Prompt-2 items 1 & the member_definition.** A MEMBER = one (family, configuration); registry v1 =
9 families × configs = **23 members** (7 sklearn × 3 cascade cutoffs + cnn_c1 + ann_torch_c1). Built
by `scripts/18_member_registry.py` → `outputs/member_registry/registry_v1.json` (+ .csv). Uniform
vote rule: each member's calibrated P(yEHDE) ≥ its leakage-safe per-region×block threshold → 1 vote.
**Pruning (item 1)** `scripts/19_prune_members.py`: selection is NESTED (per outer LOYO fold, ranked
on the other 19 years' out-of-sample votes), PER LEAD WEEK, criterion = PSS (POD−POFD) over W2–W6
pooled across regions with a FAR ceiling and an **ensemble-effect gate** (reject a prune if the
train vote-ensemble PSS falls). **Result = mostly NULL** (pool is highly redundant, cf. §6 LOMO):
consensus prune (≥50% of folds) only at **W2**, dropping the 6 linear members (logistic×3,
ridge_sev×3) + ann_torch_c1; dropping them *improves* W2 ensemble PSS 0.3033→0.3085. W3–W6 keep all
23. **Optimism gap ≤0.0044 PSS** at every week — the nested selection barely differs from an all-year
selection, confirming the CV is not inflated. Registry v2 stores a per-week active mask
(`results/tables/v2_active_members_by_week.csv`); downstream items 4/5/7 use the pruned pool per week.
**Why it matters:** individual linear members rank lowest but are decorrelated enough that the gate
keeps them except at W2 — a publishable "counting is robust, pruning barely helps" result.
**Expansion (item 2)** `scripts/20_expand_members.py` + `src/ehd/{features_ext,expand_models}.py`:
adds 7 new members (registry v3) = `{logistic,hgb,rf,mlp,hgb_sev}_tele_c1` (Stage-1 features
AUGMENTED with ENSO NINO3.4 + MJO OMI PC1/PC2/amp + antecedent-persistence obs-EHD-freq/sev) and
`{logistic,rf}_bal_c1` (class_weight balanced). Control-only reforecast ⇒ **no ensemble-spread
predictors in training** (documented, [[qg1-hpc-slurm]] rule → SLURM job 1401305, `submit_stage2_loyo.slurm`).
Downstream loader = `src/ehd/members_v3.py` (merges v2 + cnn + v3, applies per-week active mask).

### L-13 — EHD_ens ensemble + per-cell skill maps + HTML report (v2 Phase A)
**Request (user 2026-09-14):** (1) stack/ensemble all AI members → EHD_ens; (2) per-week per-cell
maps of POD/POFD/FAR/CSI/EDI/SEDI…; (3) add CNN+ANN and repeat; (4) all results in HTML;
(5) note everything. User delegated all design choices to the agent.
**Phase A done (7 tabular members; CNN/ANN = Phase B, pending):**
- `src/ehd/ensemble.py` + `scripts/14_ensemble_v2.py`: members = the 7 zoo families at cascade
  cutoff 1.0. **ens_mean** = unweighted mean; **ens_stack** = logistic meta-learner trained
  LEAVE-ONE-YEAR-OUT at the meta level (leakage-safe — member probs for year y came from base
  models that never saw y, and the stacker for y is fit on the other 19 yrs). Ensemble decision
  threshold = per-region Peirce-optimal fit on the fit-years' ensemble probs, applied to held-out yr.
- **Result:** ens_stack ≥ best single member at short lead (W1 SEDI 0.691 vs 0.676; W2 0.429 vs
  0.414), ens_mean best at long lead (W3–W6, e.g. W6 SEDI 0.168 vs best member 0.150). So the
  ensemble helps modestly and is the recommended product; headline horizon unchanged (~W3).
- `scripts/15_skill_maps_v2.py`: per-cell 2×2 counts pooled over 20 LOYO years (each sample uses
  its own fold/region threshold) → `outputs/v2_skill_maps.nc` (model×block×lat×lon, all metrics)
  + `results/figures/v2_skillmap_{ens_stack,ens_mean,hgb,mlp}.png` (metric-rows × week-cols). Maps
  confirm the north→south, short→long-lead skill decay (matches L-11).
- `scripts/16_report_v2_html.py` → `results/v2_report.html` (self-contained, base64 figures).
- Tables: `results/tables/v2_scores_with_ensemble.csv`.
**Perf note:** stage 14 stacking took ~9.5 min on the LOGIN node (120 logistic fits on ~1M×7).
Borderline vs the SLURM rule ([[qg1-hpc-slurm]]); route the post-CNN re-run through SLURM.
**How to apply:** ens_stack (short lead) / ens_mean (long lead) is the v2 product; report both.

**Phase B (CNN + deep ANN) — code done, SLURM run submitted (job 1401269, 2026-09-14):**
- `src/ehd/gridmodels.py` (torch-CPU): **TorchANN** (deep MLP [128,64,32]+dropout, tabular) and
  **TorchCNN** (fully-convolutional: reshapes the per-cell tabular rows back onto the
  (init,channel,lat,lon) grid — cell k↔(iy[k],ix[k]) — 3×3 convs so each cell uses neighbours;
  reads land-cell values back in the SAME sample order as the tabular members). Both isotonic-
  recalibrated on TRAIN. CNN uses the FULL field (no train-cap); ANN train-capped 200k.
- `scripts/12b_run_cnn_ann.py` → `outputs/loyo_pred_v2_cnn/pred_v2cnn_<yr>.npz` (cols
  p_cnn_c1, p_ann_torch_c1 + per-region decthr). Single-level (cutoff 1.0), mirrors stage-12
  assembly so rows align 1:1 with pred_v2 (verified: cell & y match exactly for 2010/W1).
- `submit_cnn_ann.slurm` (5 nodes×4 tasks, same layout as loyo_v2) runs 20 folds then re-runs
  stages 13→14→15→16. Stages 13/14/15/16 now auto-detect the cnn dir and fold cnn+ann_torch in
  (graceful fallback if absent). Torch needs `LD_LIBRARY_PATH=$CONDA/lib` (CXXABI, see slurm).
- Smoke: CNN AUC 0.786 on W1 with only 4 train yrs/5 epochs → competitive; ~25 min/fold est.
**REQUIRES torch:** `import numpy`/`torch` fails on login node WITHOUT LD_LIBRARY_PATH set to the
conda lib (system /lib64/libstdc++ lacks CXXABI_1.3.9).

### L-12 — v2 severity predictand + two-level cascade model zoo (20-fold LOYO, pooled OOS)
**Result:** the queued v2 method (predictand `sev = Tx − min(Tx90p,35)`; expanded zoo
logistic/hgb/rf/mlp classifiers + ridge_sev/hgb_sev/mlp_sev regressors; two-level cascade with
LV1 easy-negative screen) ran to completion on SLURM (`submit_loyo_v2.slurm`, 5 nodes × 4 tasks,
~30 min wall). All 20 folds → `outputs/loyo_pred_v2/pred_v2_*.npz`; stage 13 pooled the 20
held-out years (leakage-safe "whole dataset" = 2003–2022 OOS) into the full binary inventory
(POD/POFD/FAR/CSI/ETS/HSS/Peirce/EDI/SEDI/freq_bias + Brier/BSS/AUC).
**Best model per block (by SEDI):**
| block | family | cutoff | SEDI | EDI | POD | FAR | BSS_clim | AUC |
|---|---|---|---|---|---|---|---|---|
| W1 | hgb     | 0.90 | 0.678 | 0.682 | 0.768 | 0.298 | 0.351 | 0.844 |
| W2 | mlp     | 1.00 | 0.414 | 0.417 | 0.659 | 0.527 | 0.119 | 0.707 |
| W3 | mlp     | 0.90 | 0.256 | 0.258 | 0.600 | 0.586 | 0.046 | 0.628 |
| W4 | mlp     | 0.90 | 0.214 | 0.217 | 0.591 | 0.602 | 0.031 | 0.607 |
| W5 | mlp_sev | 0.90 | 0.173 | 0.171 | 0.550 | 0.612 | 0.012 | 0.574 |
| W6 | hgb_sev | 0.99 | 0.153 | 0.151 | 0.544 | 0.619 | 0.002 | 0.563 |
**Findings:** (1) v2 reproduces the v1 horizon story (L-10) — strong W1, real skill through
~W3, climatology-equivalent by W4+ (BSS_clim → 0). (2) The **LV1 easy-negative cascade screen
is essentially neutral**: sweeping the "certain-nEHDE" cutoff 0.90→0.99→1.0 (=no screen) moves
SEDI/BSS in the 3rd decimal — it does not hurt, and gives at most a marginal W1 gain (SEDI
0.678 @0.90 vs 0.676 @1.0). So the cascade is safe to keep for interpretability but is not a
skill driver on this dataset. (3) freq_bias 1.08–1.49 confirms long-lead over-forecasting
(consistent with L-11). (4) MLP wins the mid blocks; hgb/hgb_sev win the ends — no single
family dominates, justifying the zoo.
**How to apply:** report v2 as a like-for-like extension of v1 (same EHDE event, leakage rule,
six blocks). Headline unchanged: skilful to ~W3. Evidence:
`results/tables/v2_cv_scores_by_model_block.csv` (126 rows = 7 families × 3 cutoffs × 6 blocks),
`v2_best_model_by_block.csv`, `results/figures/v2_binary_skill_by_lead.png`.
**TODO (optional):** cnn family (planned as follow-on, not in this run — verify it was included
or is still pending); write v2 into `docs/RESULTS_REPORT.md`.

### L-11 — Field significance (FDR) and the north–south predictability gradient
**Result:** per-cell Brier skill vs climatology (hgb_recal) with Benjamini-Hochberg FDR
(q=0.05) over a whole-year bootstrap of yearly Brier differences: significant cells =
539/504/192/116/68/30 of 541 for W1..W6. Regional partition (BSS by region x block) shows a
clear gradient: at W1 all regions 0.25–0.41; the tropical south **R6 Tay Nguyen / R7 Nam Bo**
retain skill much longer (BSS ~0.18–0.19 at W2, ~0.10–0.12 at W3, still +0.04 at W6), while
the north (R1–R3) loses skill by W3 and goes slightly negative W3–W6.
**Why:** tropical heat is more persistent/less synoptically driven than the north.
**How to apply:** in the report and operational guide, state the skilful horizon is
lead- AND region-dependent (north ~W2, south ~W3+). Evidence: `outputs/ehd_skill_grid.nc`
(CF NetCDF: bss_vs_clim, fdr_significant, mean_prob_ehd per block), `regional_partition.csv`,
`results/figures/{region_map,cv_skill_map_W1,regional_partition}.png`.
**Note:** the regional_flag column (positive_ratio>=0.5) reflects predicted positive
predominance under the Peirce-optimal threshold; at long lead the decision threshold
over-forecasts (freq_bias 1.2–1.6) so the flag must be read WITH the verification scores,
not alone. **Data note:** region shapefile `7,NB_region.shp` is corrupt; maps use the gridded
`reg` mask + cartopy coastlines.

### L-10 — Headline cross-validated skill (20-fold LOYO, primary target, hgb_recal)
**Result** (recalibrated gradient boosting, ~1.12M out-of-sample forecasts/block, vs a
per-fold base-rate climatology; 95% whole-year block bootstrap):
| block | days | BSS vs clim [95% CI] | AUC | BSS vs raw | significant |
|---|---|---|---|---|---|
| W1 | 0–10  | 0.356 [0.332, 0.383] | 0.845 | 0.517 | yes |
| W2 | 11–17 | 0.118 [0.092, 0.144] | 0.710 | 0.310 | yes |
| W3 | 18–24 | 0.037 [0.009, 0.063] | 0.630 | 0.288 | yes (marginal) |
| W4 | 25–31 | 0.014 [−0.012, 0.039]| 0.603 | 0.281 | no |
| W5 | 32–38 | 0.004 [−0.021, 0.027]| 0.591 | 0.280 | no |
| W6 | 39–45 | −0.006 [−0.028, 0.018]|0.579 | 0.278 | no |
**Defensible headline:** *significant skill over climatology out to W3 (day 24); at and
beyond W4 (day 25+) the product is statistically indistinguishable from climatology.* AUC>0.5
at all leads (weak discrimination persists) but calibrated probability skill (BSS) reaches
zero by W4. Calibration always beats the raw forecast (BSS_raw +0.28…+0.52) because raw
block-exceedance probability is badly miscalibrated (flat reliability; see cv_reliability.png).
**Why it matters:** matches the brief's preference for an honest "no skill beyond W3/W4".
**How to apply:** report W1–W3 as the skilful range; present W4–W6 as climatology-equivalent.
hgb_recal is the product model. Evidence: `results/tables/cv_scores_by_block.csv`,
`cv_bootstrap_ci.csv`, `cv_scores_by_region_block.csv`, `results/figures/cv_skill_by_lead.png`,
`cv_reliability.png`. **Still TODO in verification:** per-cell FDR field significance and
season/severity stratification (year-block bootstrap already done).

### L-09 — Calibration core works; skill is short-lead and decays by ~W4 (single fold)
**Decision:** Direct single-member MOS classifiers (logistic baseline, HGB) predict
P(>=1 EHD in block) from block-aggregated forecast features (features.py). Development fold
(holdout 2022, 9 train yrs, no recalibration yet) gives, vs a base-rate climatology:
W1 HGB BSS +0.32 / AUC 0.81; W2 +0.13 / 0.70; W3 +0.04; W4-W6 ~0 (HGB) with logistic
marginally positive (~+0.03) and better-behaved at long lead. Raw block-exceedance
probability is miscalibrated (flat reliability, BSS vs clim negative) -> calibration is
essential, recalibration (task 7) still to add.
**Why it matters:** matches the brief's preference for a defensible "limited skill beyond
~W3-W4" over a decorated one. **How to apply:** carry logistic + HGB (+ more families) into
the full 20-fold LOYO (stage 06) with nested tuning; expect HGB to need regularisation at
long lead where it currently overfits weak signal. Evidence:
`results/tables/singlefold_2022_scores.csv`, `results/figures/singlefold_2022_reliability.png`.
**Caveats:** ONE fold; global base-rate climatology reference; no isotonic recalibration;
decision threshold from train only (correct). Not a headline number.

### L-08 — Forecast daily-Tx backbone built
**Decision:** `scripts/03_build_forecast_targets.py` stacks all inits per year into
`outputs/forecast_daily_tx/fc_daily_tx_<year>.nc` (init, lead_day 0–45, lat, lon; °C;
lat sorted ascending; valid_date coord). 20 files, 2080 inits, ~635 MB.
**How to apply:** this is the deterministic predictor; observed EHD targets + per-fold Tx90p
threshold are joined inside CV (never here) to keep the leakage rule. Lead-block reductions
live in `src/ehd/targets.py` (primary = ≥1 EHD; secondary = ≥3-run heatwave, straddle rule =
run detected on continuous 0–45 series then attributed to every block it overlaps).

### L-07 — Gridded vs station reconciliation → target choice
**Decision:** Calibrate and verify against **gridded Tx** (spatially complete, aligned to the
forecast grid); use the **200 station Tx** records as **independent point validation**.
**Why:** over 2002–2023, 189 matched stations are on average **+1.5 to +2.2°C warmer** than
their 0.25° cell (corr 0.83–0.98, so systematic offset not timing), making station EHD
frequency **0.31–0.59 vs gridded ~0.10**. The gridded field is the only complete target;
stations bound how far point-scale EHD risk can exceed the gridded product.
**How to apply:** report the offset as a limitation; never impute station −99.0; match on
intersecting dates only. Evidence: `results/tables/obs_reconciliation_*.csv`,
`results/figures/obs_reconciliation.png`, `docs/OBS_RECONCILIATION.md`.

---

### L-01 — The reforecast has no ensemble; calibrate the single deterministic member
**Decision:** Treat calibration as single-member MOS / distributional regression — fit a
predictive distribution (or directly P(EHD in block)) from the control forecast plus
covariates, rather than any ensemble statistic.
**Why:** `find` over the whole EC tree returns only `_cf_` (control) files; 2080 Tx inits =
104/yr × 20 yr, one member each; no `_pf_` anywhere. Ensemble mean/spread, EMOS, CRPS-ensemble
calibration, rank histograms, and raw exceedance-fraction P(EHD) are undefined with n=1.
**How to apply:** Predictors = deterministic Tx (and neighbours), lead, calendar harmonics,
lat/lon, elevation, and other control S2S fields. ML families (quantile regression forests,
gradient boosting with pinball loss, distributional NN) output a predictive distribution;
P(EHD) = P(predictive Tx ≥ threshold). Surfaced to user as reconciliation R-01.

### L-02 — No downscaling needed; forecast grid == obs grid
**Decision:** Omit the downscale stage. **Why:** forecast lon/lat coordinate values are
identical to the obs grid (41×65, 0.25°); the archive was interpolated to 0.25° upstream.
**How to apply:** Document upstream interpolation as a limitation (the "regrid before
calibrate" failure mode is already embedded and cannot be undone from this data). Surfaced
to user as reconciliation R-02.

### L-03 — Daily Tx from 6-hourly mx2t6
**Decision:** daily Tx(d) = max over the four `mx2t6` values in day d. **Why:** header shows
`mx2t6` = "Maximum temperature in the last 6 hours", 184 steps × 6 h, first stamp init+06:00.
**How to apply:** day 0 covers init+00→24 h; 46 days → days 0–45 → lead blocks W1–W6 exactly.

### L-04 — Calendar mismatch obs(365_day) vs forecast(gregorian)
**Decision:** Match forecast and obs on (month, day); drop 29 Feb from forecast-derived Tx.
**Why:** gridded obs built with `-del29feb`, calendar `365_day`; forecast is gregorian.

### L-05 — Region ids available gridded and per-station
**Decision:** Use gridded `reg` (1–7) for all numeric zonal products; shapefile only for the
publication map outline. **Why:** mask carries `reg`; DBF has no names; station header field
4 is the region code (SAPA=2, HANOI=3, ALUOI=4, DANANG=5, PLEIKU=6, CAMAU=7). Region names
to be derived from `reg` centroid positions + station cross-check, not from the DBF.

### L-06 — Station directory is multi-variable
**Decision:** Use `Tx_*.txt` (200 stations) as station Tx; ignore `R_*`(rainfall),
`Tm_*`(Tmin), `T2m_*`(Tmean) for the Tx target. **Why:** `R_ALUOI` values are precipitation
(mm, many zeros, −99 missing); `Tx_ALUOI` values are realistic °C. File format: name / lon
lat elev region / per-year 31×12 day×month matrix, missing −99.0.