# Phase 0 audit & gate report — S2S-FM-VN

**Session date:** 2026-09-23. **Scope:** Phase 0 (audit → target prototype), then
STOP before any bulk ECMWF download or model training.

## One-line status

The observation data, the fixed P90/EHD-threshold climatology and the full
target/event code are built and verified (11/11 unit tests pass, mean daily EHD
frequency 0.10 as expected); the **only blocker to the next phase is ECMWF access**
— the configured `api.ecmwf.int` token is expired and S2S has migrated to ECDS.

## 1. What was verified (with the file that shows it)

| Area | Result | Evidence |
|---|---|---|
| Compute | Slurm `workq`, 11×20 CPU, **no GPU** (GRES=null); PyTorch absent | `config/training.yaml`, node `sinfo` |
| Python | 3.9.5; numpy/xarray/netCDF4/cdsapi/ecmwf-api-client present; **cfgrib missing** | node inspection |
| Storage | `/work` 35 TB free (bulk target), `/home` 401 GB | `df` |
| Gridded Tx | VnGAD, var `tasmax` degC, 61×31 @0.25°, 1961–2023, 586 valid land cells (time-stable), fill −999, includes 15 Feb-29 days | `metadata/phase0_data_audit.json` |
| Region mask | R1–R7 (codes 1–7) on the VnGAD grid; counts 51/105/37/88/63/96/101 | `metadata/phase0_data_audit.json` |
| Fixed P90 | 365×61×31, cyclic ±15 d, 2003–2022; 17.0–38.4 °C; 6.5% day-cells capped at 35 °C | `data/processed/P90_2003_2022.nc` |
| EHD threshold | min(P90, 35 °C) | `data/processed/EHD_THRESHOLD_2003_2022.nc` |
| Targets | mean daily EHD freq **0.1013** (per region 0.093–0.108) | `outputs/metrics/target_event_frequency.csv` |
| Target code | P90 leap/wrap, EHD/exceedance, weekly aggregates, episode detection, cross-week onset/active/days split, censoring — **11/11 tests pass** | `tests/test_targets.py`, `src/targets/` |
| ECMWF structure | class s2 / dataset s2s / origin ecmf / stream enfh / type cf; native 1.5° (ocean 1.0°); GRIB2; length 34–65 d; years 2015–2026 | `metadata/ecmwf_reforecast_inventory.csv` |
| ECMWF probe | node reached API; **"Access token expired"** | `metadata/ecmwf_minimal_retrieval_status.json` |

The locked definitions are implemented exactly and untouched; every Category B
quantity is deferred to per-fold fitting (nothing fit on held-out data yet); W7 is
kept separate; cross-week events are handled per spec; no original file was
modified, moved or deleted.

## 2. What remains uncertain

- Valid `date`/`hdate` pairs covering 2003–2022 and the **true initializations/yr**.
- **Model-cycle consistency** across 2003–2022 (may straddle cycles).
- Daily-max 2t aggregation window and **UTC-vs-local-day** alignment.
- Requested-vs-retrieved grid / member type — needs a real GRIB header.
- Whether the gridded VnGAD product used the stations (independence of station verification).

## 3. Estimated data volume (order-of-magnitude, to refine after minimal retrieval)

High-priority set: ~8 pressure fields (T850, Z500, U850, V850, U200, V200, q850,
q700) + ~5 surface fields, LT0–45 (46 steps), ~2 inits/week × ~20 hindcast years
≈ ~2,000 initializations, one member (cf), 1.5° over the two predictor domains.
This is on the order of **a few hundred GB to ~1–2 TB** depending on step frequency
and domain crop — comfortably within the 35 TB `/work` budget. **Exact figure to be
computed from the minimal retrieval's per-field byte size** before proposing bulk download.

## 4. Proposals (for approval)

- **Predictor list:** required surface (2t, mx2t, msl, 10u, 10v) + high-priority
  pressure set above; recommended (sst@1.0°, tp, mn2t, sshf, slhf); optional soil/rad.
  See `config/variables.yaml`.
- **Domains:** local/regional 95–120 °E, 5–30 °N; large-scale 40–180 °E, 20 °S–50 °N.
- **Architecture:** dual-branch encoder → cross-scale fusion → lead encoder (Z0..Z45,
  not collapsed) → 0.25° VN decoder; prototype **5–10 M params**. See `config/model.yaml`.
- **CV:** year-grouped; recommend **blocked-year** for the prototype (cheapest
  per-fold pretraining), LOYO/rolling-origin as feasibility allows — effective
  sample count and per-fold pretraining cost to be quantified once initializations/yr
  is known. See `config/training.yaml`.
- **Forecast-episode binarisation:** daily P(EHD) ≥ 0.5, applied identically to every model.

## 5. Decision items needing you (one consolidated list)

| # | Decision | Recommendation | Evidence |
|---|---|---|---|
| **D1** | ECMWF credential expired | Refresh the ECMWF/ECDS token for the account; never paste it into chat/repo — place it in `~/.ecmwfapirc` or the ECDS client config | retrieval probe → "Access token expired" |
| **D2** | S2S migrated to ECDS (2026-05-27) | Confirm the access path: ECDS client / cdsapi vs legacy MARS Web API; then run the ready minimal retrieval | ECMWF migration notice; `docs/ecmwf_s2s_inventory.md` |
| **D3** | Two 0.25° grids | Confirm VnGAD (61×31) as the target grid and `mask7reg_VnGAD` for regions; treat `Mask_VN_S2S_EC` (65×41) as out-of-scope | `config/domains.yaml` |
| **D4** | No GPU on the cluster | Provision a GPU resource **or** approve a CPU-sized prototype before Phase 3 | `sinfo` (GRES=null) |
| **D5** | Station Tx absent | Provide the station Tx file + station→grid provenance (for the independence caveat) | `config/data.yaml` |
| **D6** | `cfgrib` missing | Approve installing `cfgrib`/`eccodes` python (or standardise on cdo/eccodes-CLI + `grib_to_netcdf`) | node inspection |
| **D7** | 45 valid cells unclassified by region | Confirm handling (report separately vs assign nearest region) | audit JSON |

## 6. Gate

**STOPPING before bulk download / training, as required.** Awaiting approval on the
decision items above (D1/D2 are prerequisites for Phase 1). Nothing ran beyond the
Phase 0 scope.