# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

**S2S-FM-VN** — a leakage-safe, reproducible, HPC-ready pipeline that maps the ECMWF
S2S **control** reforecast (coarse 1.5° grid) to daily/weekly **extreme-heat** products
on the 0.25° mainland-Vietnam observational grid (VnGAD), over leads **LT0–LT45**. It
complements ECMWF's dynamical integration; it does not replace it. This is a standalone
project — sibling S2S heat projects on this machine use different lead-week/threshold/
member conventions; **none of theirs apply here.** Only `config/targets.yaml` governs.

## Non-negotiable rules (read before any change)

- **HPC filesystem is read-only to you** (see parent `../CLAUDE.md`): never delete, move,
  or modify any file/dir outside this project tree. Observational originals in `data/`
  (symlink → `/work/.../FM_data`) and `/work/.../OBS/` are read **in place**. Derived data
  goes only under `data/processed/` and `data/raw/`. `outputs/` is a symlink to `/work/.../FM_out`.
- **Locked definitions.** `config/targets.yaml` (P90, EHD threshold, weeks W1–W7, episodes,
  cross-week accounting) is LOCKED by the project prompt. Do not reopen or "improve" it; a
  labelled sensitivity experiment is allowed but never replaces the primary definition.
- **Leakage rule** (README.md "Leakage rule", `config/verification.yaml:leakage_rule`):
  *Category A* = fixed predictand definitions (P90, EHD_THRESHOLD) computed once from obs
  2003–2022, identical in every fold — not leakage. *Category B* = everything the forecasting
  system estimates (climatology, bias correction, calibration, feature selection, all ML
  params, thresholds, pretraining) is fit on **training years only**, per fold. A held-out
  year enters none of it. Never optimise a probability threshold on held-out test data.
- **Don't touch other projects' jobs.** `Dr_GCM/.../download_ecmwf_baseline` runs under the
  same user and may share the ECDS queue — leave it alone.

## Source of truth / where to look first

`docs/decision_log.md` is the running log (one lesson per entry, newest first, L1–L18); read
it before making pipeline decisions. `docs/HANDOFF_2026-09-26.md` has current data/job state.
`docs/canonical_schema.md` is the **FROZEN** GRIB→internal schema. `docs/RETRIEVAL_PLAN.md` is
the full download plan. `My_swers.md` records the user's gate decisions (D1–D7).

## Commands

```bash
# Tests (numpy-only, no GRIB/data needed) — run these after any change to targets/timealign
python3 tests/test_targets.py        # 11 tests, LOCKED target/event definitions
python3 tests/test_timealign.py      # 6 tests, valid-time / UTC↔local alignment
# or: pytest tests/

# Phase 0 (audit, P90, EHD threshold, masks)
python3 scripts/00_phase0_data.py
python3 scripts/02_verify_masks_and_regions.py
python3 scripts/03_build_obs_climatology.py

# Reforecast download (resumable, skip-existing; GATE before any `--dates all`)
python3 scripts/05_download_reforecasts.py --group sfc_tx  --dates 1     # smoke test
python3 scripts/05_download_reforecasts.py --group pl_hi   --dates all   # full group

# Assemble → load/validate → features → train
python3 scripts/08_assemble_cube.py --group all --dates all
python3 scripts/09_load_cube.py
python3 scripts/10_build_features.py --dates all
python3 scripts/11_train_eval.py
```

- Interpreter is `python3` at `/home/hapt/miniconda3/bin/python3` (xarray 0.20.1, sklearn 1.0.1).
  `cdo` and `grib_ls`/`grib_get` (ecCodes CLI) are on PATH from the same conda env. PyTorch is
  **not installed** and the cluster is **CPU-only** (no GPU); all code must stay device-agnostic
  via `src/utils/device.py` (imports torch lazily). Numbered-stage scripts prepend `src/` to
  `sys.path`; run them from the repo root.
- Most scripts take `--dates N` (smoke, first N) or `--dates all`, and are **resumable**
  (skip outputs that already exist). Long downloads/assembles run in the background.

## Pipeline architecture

Numbered `scripts/NN_*.py` are the executable pipeline, in order; `src/` holds the reusable
library they import; `config/*.yaml` holds all locked parameters. Data flows:

1. **GRIB retrieval** (`05_download_reforecasts.py`) — ECDS `s2s-reforecasts`, control forecast
   only, native 1.5°, via `cdsapi` with credentials from `~/.cdsapirc_1` (url `ecds.ecmwf.int/api`;
   **not** `~/.cdsapirc`/legacy MARS). Scheme (L16): EC model-version year 2023; each of the 104
   MM-DD dates in `hdate_2023.txt` is one request bundling hindcast years 2003–2022. Variables are
   split into **groups by native step frequency and domain** so nothing is silently dropped (L18):
   `sfc_tx` (`mx2t6`, 6-hourly, LOCAL), `sfc_dyn` (`msl/10u/10v` + `2t`, daily, LOCAL),
   `sfc_t2m` (daily-*mean* `2t`, pulled separately because `2t` is archived only as 24-h
   averages `0_24..1080_1104`, LOCAL), and `pl_hi` (`t/gh/u/v/q` @ 850/700/500/200 hPa, daily,
   **LARGE domain** — required for ENSO/MJO/monsoon signal, L17). Filenames carry a domain tag
   `_loc`/`_ls`. (`--group` choices: `sfc_tx`, `sfc_dyn`, `sfc_t2m`, `pl_hi`.)
   → `data/raw/ecmwf_s2s_1p5deg/s2s_cf_<group>_<dom>_mv2023-MM-DD_hy2003-2022.grib`.
2. **GRIB → canonical** (`src/data/grib_preprocess.py`, `S2SReforecastPreprocessor`) — per the
   FROZEN schema: `cdo -f nc copy` → NetCDF → xarray; flip latitude to ascending (GRIB is N→S);
   K→°C for temperature; `valid_time = init + endStep` (UTC); predictors stay on native coarse
   grid (never regridded to 0.25° before the encoder, D8). Daily Tx = max of the four `mx2t6`
   6-hour windows per **local UTC+7 calendar day** (`daily_max_from_6h`, `timealign`).
3. **Assembly** (`08_assemble_cube.py`) — standardises each file to daily lead days LT0–LT45 →
   `data/processed/cube/<group>/<group>_mv2023-MM-DD.nc`. Logical cube per variable =
   104 files stacked lazily on `model_version_date`: `(model_version_date[104], hindcast_year[20],
   lead[46], [level], lat, lon)`. `09_load_cube.py` lazily stacks + validates (dask-backed).
4. **Features** (`10_build_features.py`) → one `.npz` per model-version date under
   `data/processed/ml/`, columns tagged `ctx/raw/loc/large` for the ablation.
5. **Train/eval** (`11_train_eval.py`) — dual heads (Tx regression + EHD classification),
   **leave-one-year-out** over the 20 hindcast years, predictor-scope ablation raw→+local→+large,
   Ridge + HistGradientBoosting; baseline = raw ECMWF forecast (the B1 chain) on identical rows.
   `07_prototype_b1.py` is the torch-free end-to-end validation baseline.

## `src/` library

- `src/targets/` — `climatology.py` (fixed P90 + EHD threshold; 365 calendar-day indices, 29 Feb
  pooled with 28 Feb at index 59), `events.py` (daily EHD, exceedance, weekly aggregates, episode
  detection + cross-week accounting), `obs.py` (observed truth/climatology products). **The locked
  distinctions that fail quietly are enforced + unit-tested here** — notably HW2_WEEK (≥2 EHDs
  anywhere in week) vs heat-wave EPISODE (≥2 *consecutive* EHDs); never conflate them.
- `src/utils/` — `timealign.py` (valid-date = init + lead by **actual date, never array index**;
  UTC↔local-day), `device.py` (CPU/CUDA, lazy torch), `config.py`, `logdir.py`.
- `src/data/` — `interfaces.py` (abstract data layer), `grib_preprocess.py` (concrete, schema-frozen),
  `observations.py` (`VnGADObservations`, no GRIB dependency).

## Grids & masks (locked)

Target grid = VnGAD 0.25°, 61×31, lon 102–109.5°E / lat 8.5–23.5°N, 586 valid land cells.
Region mask = `mask7reg_VnGAD_0.25.nc` (`vn_7regions`, codes 1–7) on that same grid — read in
place, never redrawn. 45 valid cells carry no R1–R7 code → `region=0` (national INCLUDE, regional
EXCLUDE, D7). `data/Mask_VN_S2S_EC_0.25.nc` is a **different grid** (65×41) from a sibling project
and is **not used** (L6).