# Decision log — S2S-FM-VN

One lesson per entry; newest first. Update an entry rather than duplicating it;
delete an entry once shown wrong. Summary line on top of each entry.

---

## L18 — S2S surface variables have different native step frequencies
`mx2t6` is 6-hourly (184 steps); `msl` daily (46); `2t`/`10u`/`10v` differ — a single
6-hourly request silently returned only 1 `2t` step. **Why:** bundling mixed-frequency
vars in one leadtime list drops data. **How to apply:** downloader groups split by
frequency -- `sfc_tx` (mx2t6, 6-hourly), `sfc_dyn` (2t/msl/10u/10v, 24-hourly),
`pl_hi` (pressure, 24-hourly, large domain). Assembly standardizes all to daily LT0-45.
Full-set sizes/plan in `docs/RETRIEVAL_PLAN.md` (~17.4 GB, pl_hi large-scale is 162
MB/date). Full count per variable = 104 mv-dates x 20 hyears x 46 leads = 95,680.

## L17 — Pressure/large-scale predictors need the LARGE domain, not the local box
Initial prototype mistakenly used the LOCAL area [30,95,5,120] (95-120E,5-30N) for
the pressure group too. Pressure/circulation fields (Z500, U200/V200, T850, q) are
the LARGE-SCALE branch and must span 40-180E, 20S-50N to capture ENSO/MJO/monsoon/
teleconnections; the local box cannot. **Why:** wrong domain silently starves the
large-scale branch of its signal. **How to apply:** downloader
`scripts/05_download_reforecasts.py` now sets per-group domains -- `sfc_core`
(local, tag `_loc`), `pl_hi` (large-scale [50,40,-20,180], tag `_ls`); see
`config/domains.yaml`. Wrong-domain pl_hi files removed and re-downloaded on the
large domain. sfc_core / mx2t6 stay LOCAL (target-side) -> B1 baseline unaffected.

## L16 — 2003-2022 download scheme via hdate_2023.txt VALIDATED (2026-09-25)
New key in `~/.cdsapirc_1` (url ecds.ecmwf.int/api) clears the throttle. Scheme:
for EC model version 2023, each MM-DD in `hdate_2023.txt` (104 dates) is a
model-version date; one request with `year=2023,month,day` + `hyear=[2003..2022]`
returns the full 20-year reforecast set for that date. Tested (mv 2023-12-28):
`scripts/05_download_reforecasts.py 1` -> 2.9 MB GRIB, **3680 msgs = 20 hyears x
184 steps**, dataDates 2003-12-28..2022-12-28, mars.date 2023-12-28.
`scripts/06_process_reforecast_file.py` splits by dataDate (grib_copy) ->
per-year canonical -> daily-max -> **all 20 years 47/47 aligned to VnGAD obs**.
**Why:** confirms the whole retrieval+preprocess chain end-to-end on real
multi-hindcast data. **How to apply:** downloader `scripts/05_...` (resumable,
skip-existing, one-at-a-time, cred from ~/.cdsapirc_1); multi-init preprocessing
`S2SReforecastPreprocessor.load_canonical_by_init`. Full set = 104 requests; STOP
for a retrieval plan/gate before the full bulk (predictor vars + domains + volume).

## L15 — Real GRIB inspected; canonical schema FROZEN (2026-09-24)
User downloaded a real ECDS `s2s-reforecasts` GRIB (`data/Tx_05-01-2022.grib`, via
`test.py`, `cdsapi.Client(url="https://ecds.ecmwf.int/api")`). Confirmed and frozen
(`docs/canonical_schema.md`, `metadata/grib_inspection.md`): GRIB2 `ecmf` `cf`
`regular_ll` native **1.5deg**; lat **DESCENDING** N->S (flip on load); units **K**
(->degC); variable `mx2t6` (paramId 121, `stepType=max`); **184 six-hourly windows**
`0-6..1098-1104` = 46d x 4/day; init = `dataDate`=`mars.hdate` (2022-01-05),
model-version = `mars.date` (2023-01-05); `valid=init+endStep` (verified). Path (D6):
`cdo -f nc copy` -> CF NetCDF -> xarray (cfgrib not needed). **Why:** the working
`test.py` corrected two guesses -- the daily-max variable is
`maximum_2_m_temperature_in_the_last_6_hours` with 6-hourly leads (not `2_m_temperature`
/`0_24`). **How to apply:** `src/data/grib_preprocess.py`
(`S2SReforecastPreprocessor`, `daily_max_from_6h`); daily Tx = max of the 4 mx2t6
windows per LOCAL UTC+7 day. Preprocessing tested on the sample by
`scripts/04_preprocess_grib_sample.py`.

## L14 — ECDS blocker is a per-user request QUOTA/limit, not the request params
The minimal `s2s-reforecasts` request is now fully valid against LIVE ECDS
constraints (`metadata/ecds_constraints_staged.json`): FC 2015-01-01 (a valid
Jan-2015 Thursday model-version day), hindcast 2003-01-01, `variable=2_m_temperature`
(NAME, not paramId), `leadtime_hour=0_24`, `data_format=grib`, native 1.5deg (no
`grid` key), `area=[30,95,5,120]`. Yet every submit returns `rejected`; the real
reason (forced via the job `/results` endpoint) is:
  "The job has been rejected. The number of queued requests per user for ECDS
   datasets is limited to $value."
Account audit (15 pages): 132 jobs = 128 rejected + 4 successful, **0 queued**.
Purged all 128 rejected (kept 4 successful) -> account at 4 jobs -> **STILL
rejected**, and DELETE calls hit `429 Too Many Requests`. So it is NOT retained
count and NOT live-queue; it is an **account RATE-LIMIT / throttle** (or an ECDS
server bug: the limit value renders as literal `$value`). 4 historical
`s2s-forecasts` successes prove the licence/access is fine. **Why:** ~140+ API
calls on this account in one morning exhausted the rate limit. **How to apply:**
STOP all API submissions; let the account COOL DOWN (hours). Then EITHER retry ONE
`scripts/01_minimal_ecmwf_retrieval.py`, OR (more reliable) download one minimal
GRIB via the ECDS **web form** and drop it at
`data/raw/ecmwf_s2s_1p5deg/s2s_refc_cf_min.grib` for inspection. Consider reporting
the unfilled `$value` limit to ECDS support. Vocabulary lessons still hold: ECDS
`variable` uses NAMES; `data_format`='grib' (not grib2); omit `grid` for native
1.5deg. Job tools: `scripts/ecds_jobs.py`, `scripts/ecds_purge_rejected.py`.

## L13 — Phase 1 preprocessing waits on a real GRIB; only metadata-independent scaffolding now
Do NOT build GRIB->canonical preprocessing around assumed metadata. Sequence:
retrieve minimal GRIB -> inspect -> document archive structure -> FREEZE the
canonical schema (`docs/canonical_schema.md`) -> implement preprocessing -> verify
alignment. **Why:** patching assumed metadata later corrupts temporal alignment
silently. **How to apply:** built now = `src/utils/{timealign,device,logdir,config}.py`,
`src/data/{interfaces,observations}.py` (abstract + placeholders), tests
`tests/test_timealign.py`; concrete `Preprocessor`/`PredictorSource.load_canonical`
raise NotImplementedError until schema freeze. Observational-target work proceeds in
parallel (no GRIB dependency).

## Gate decisions received 2026-09-23 (from My_swers.md)
- **D1/D2:** access path = ECDS -> cdsapi -> `s2s-reforecasts`, cf only, coarse 1.5°;
  credential in `~/.cdsapirc` with url `https://ecds.ecmwf.int/api` (user configures
  the token locally). Legacy Web API / direct MARS NOT used. Reproduce the request
  from ECDS "Show API request code" and verify a minimal one before bulk. **Status:
  `.cdsapirc` still points at old CDS on 2026-09-23 — minimal retrieval waits on the
  ECDS token.** Retrieval script ready: `scripts/01_minimal_ecmwf_retrieval.py`.
- **D3:** VnGAD target grid + `mask7reg_VnGAD` regions — **verified PASS** (see L6).
- **D4:** CPU-sized prototype approved for Phase 0-3; code must be device-agnostic
  (CPU/CUDA), record memory. GPU only needed for full pretraining/CV (see L10).
- **D5:** station Tx not required now; VnGAD primary; station verification is an
  optional later module (see L5).
- **D6:** raw workflow standard = ecCodes CLI (`grib_ls`/`grib_get`) + CDO ->
  NetCDF/Zarr -> xarray; keep original GRIB; cfgrib optional, not a blocker.
- **D7:** unassigned valid cells -> `region=0` (national INCLUDE, R1-R7 EXCLUDE);
  diagnostic done (see L12).
Bulk 2003-2022 download NOT started (per instruction).

---

## L11 — Leakage rule is written down and checked at every gate
Category A (P90, EHD_THRESHOLD) is fixed from 2003-2022 and identical in every
fold — this is a predictand *definition*, not forecasting leakage. Category B
(everything the forecasting system estimates, incl. pretraining) is fit on
TRAINING years only; held-out years enter none of it. Transductive all-years
pretraining is a labelled side experiment only. **Why:** the paper's central
claim is exactly what leakage would fake. **How to apply:** re-read
`config/verification.yaml:leakage_rule` at each gate.

## L10 — CPU-only cluster; a foundation model needs a compute decision
`sinfo`: partition `workq`, 11 nodes x 20 CPU, **GRES=null (no GPU)**; PyTorch not
installed. **Why:** Phase 3+ pretraining/fine-tuning assumes GPU-ish compute.
**How to apply:** either size the prototype for CPU or provision a GPU resource;
resolve before Phase 3. Gate item D4.

## L9 — ECMWF S2S migrated to ECDS; configured token is expired
Old `api.ecmwf.int/v1` Web API token returns "Access token expired"; S2S moved to
ECDS on 2026-05-27. **Why:** blocks the minimal + bulk retrieval. **How to apply:**
refresh credential and confirm the ECDS access path (cdsapi/earthkit/MARS) before
any download; never paste keys into chat/repo. Gate items D1, D2.

## L8 — Do not assume 104 initializations/yr or one model cycle for 2003-2022
Reforecasts are tied to model-version dates; a 20-year span can straddle cycles.
**Why:** silent combination corrupts the training set. **How to apply:** tabulate
per-initialization cycle metadata from live catalogue before combining.

## L7 — UTC vs local-day alignment is a known silent error
Obs Tx is a local (UTC+7) calendar-day maximum; ECMWF maxima are UTC; LT0 may be a
partial day. **Why:** a one-day shift fakes/destroys skill. **How to apply:**
`valid_date = init + lead`, align by actual date; unit-test incl. leap years.

## L12 — Unassigned valid cells handled as region=0 (national include, R1-R7 exclude)
45 of 586 valid VnGAD cells carry no R1-R7 code; 43 are coast/border (heuristic).
Per D7 they get `region=0`: kept in national training/products/maps/verification,
excluded from R1-R7 aggregation until assignment is resolved. **Why:** dropping
valid mainland cells would bias national skill; arbitrary assignment would corrupt
regional stats. **How to apply:** use `data/processed/region_codes_vngad.nc`
(-1 outside, 0 unassigned, 1-7 regions); review `metadata/unassigned_region_cells.csv`
and `outputs/figures/unassigned_region_cells.png`.

## L6 — Two 0.25 grids exist; target grid is VnGAD, region mask must match it — VERIFIED
Gridded Tx (`VnGAD`, 61x31, 102-109.5E/8.5-23.5N) is the target grid. The region
mask `mask7reg_VnGAD_0.25.nc` (codes 1-7) is on that same grid — **programmatically
verified**: lat/lon exact match, dims match, both ascending, 7 codes, missing =
non-finite (`metadata/mask_alignment_report.json`, PASS). The symlinked
`Mask_VN_S2S_EC_0.25.nc` is a DIFFERENT grid (65x41) from a sibling S2S-EC project
and is **not** used. **Why:** mixing grids silently misaligns targets and masks.
**How to apply:** use `mask7reg_VnGAD` for regions; do not redraw boundaries.

## L5 — Station Tx not found; independence caveat pre-noted
No station Tx file in `data/` or the OBS source. **Why:** station verification is
secondary and, if the gridded product used those stations, not independent. **How
to apply:** request station path + station->grid provenance. Gate item D5.

## L4 — 29 Feb pooled with 28 Feb; climatology has 365 fixed day-indices
`day_of_year_noleap` maps every date by (month,day) on a non-leap calendar and
sends 29 Feb to index 59. **Why:** consistent, testable leap handling; VnGAD keeps
its 15 Feb-29 days (23010 days) and they contribute to late-Feb windows. **How to
apply:** `src/targets/climatology.py`; tested in `tests/test_targets.py`.

## L3 — HW2_WEEK and heat-wave EPISODE are different concepts, kept separate
HW2_WEEK = >=2 EHDs anywhere in the week (no consecutive condition). Episode = >=2
CONSECUTIVE EHDs on the continuous daily sequence before weekly aggregation.
**Why:** conflating them is a named failure mode. **How to apply:** separate
functions `weekly_aggregate` vs `detect_episodes`; unit-tested.

## L2 — Cross-week episodes: one onset, split active-days/severity, no duplication
An episode crossing a week boundary is ONE event: onset counted once in its start
week; HW_ACTIVE=1 in every overlapped week; HW_DAYS/HW_SEV_ACTIVE split by day;
full duration/severity belong to the onset week only and never to a right-censored
episode. **Why:** duplicating full duration/severity across weeks is a named
failure mode. **How to apply:** `analyze_forecast_sequence`; unit-tested.

## L1 — Fixed-P90 sanity: mean daily EHD frequency ~0.10
Over 2003-2022 the mean daily EHD frequency is 0.1013 (per region 0.093-0.108),
as expected for a P90-based threshold capped at 35 degC (6.5% of day-cells capped).
**Why:** first evidence the P90/threshold/EHD chain is correct. **How to apply:**
regression check when the pipeline changes.