# S2S-FM-VN — Foundation Model for Extreme-Heat Forecasting over Vietnam — Opus Prompt Package
Built with the Claude Opus Prompt Architect meta-prompt (§1 intake → §3 composition → §4 anti-pattern check → §6 output format), from the user's 73-section project draft. Every scientific definition the draft marks as fixed is carried over unchanged; what changed is structure, the operating mode, and eleven gaps or contradictions closed (listed in §5).
The prompt names the model generically as "Claude Opus" rather than a version string, per §4 of the meta-prompt — select Opus 5.x in the harness.
This is a **separate project** from the Stage 1–3 S2S EHD packages. Its lead weeks (W1 = LT00–06 … W7 = LT42–45), its P90 rule (fixed, not fold-specific) and its predictor set (control forecast only) all differ from Stage 1. The prompt says so explicitly so the agent does not import Stage 1 conventions from project files.
---
## 1. Task classification
| Dimension | Classification |
|---|---|
| Horizon | Very long, multi-session: ECMWF retrieval for 20 years, target construction, pretraining per CV fold, fine-tuning, baselines, ablations, full verification, publication package. |
| Stakes | High. Feeds a paper whose central claim ("pretraining adds out-of-sample value") is exactly the claim leakage would fake. Bulk ECMWF retrieval is also expensive and slow to redo. |
| Ambiguity | Definitions are fixed and precise. Open: ECMWF archive structure and model-cycle consistency, CV fold design under per-fold pretraining cost, forecast-side binarisation for episode detection, UTC-vs-local day alignment. |
| Interactivity | Gated. The user wants to be asked rather than guessed for, and wants a hard stop before bulk download. So: autonomous inside a phase, one consolidated decision report at each gate, no piecemeal questions. |
| Delegation | Yes, from Phase 3 on (targets, baselines, FM training, verification, validator). Minimal in Phase 0. |
| Context volume | High: ECDS metadata, gridded + station Tx, region masks, static fields, large model/verification output. |
| Extended thinking | Yes — leakage categories, per-fold pretraining, cross-week episode accounting and censoring interact. |
| Memory | Yes, across sessions — via `docs/decision_log.md` (the draft routes decisions to README.md and docs/, so no separate MEMORY.md is invented). |
| Domain sensitivity | None. |
## 2. Selected blocks
- **OP-11** — dual mission (paper + later operationalisation) sets the evidence standard.
- **OP-01 (modified)** — outcome-first for architecture and training, which the draft presents as a starting design; the event definitions are locked and not the agent's to redesign.
- **OP-02** — leakage categories, per-fold pretraining and episode accounting must be reconciled before code.
- **OP-04 (gated variant), not OP-03** — the draft's "ask, don't assume" rule plus its explicit STOP before bulk download. Adapted into the two-phase rhythm already used in the Gridded-Obs package: work autonomously inside a phase, raise every decision at the gate in one report.
- **OP-05** — gridded vs station Tx, ECDS metadata vs documentation, model vs baseline disagreements surfaced, not blended.
- **OP-09** — every verified/skill claim points at a file from this session.
- **OP-06** — scope guard: a small foundation model for one domain, not a general framework.
- **OP-07** — delegation, only after the Phase 0 gate; validator independent of the verification code.
- **OP-08** — `docs/decision_log.md` as the cross-session lesson store.
- **OP-13** — validation pass; high stakes.
- **OP-10** — final message as re-grounding.
- **OP-12 (adapted)** — the draft's HTML-in-`outputs/` requirement is the user-facing channel; the prompt points at it plus the harness's message channel.
## 3. Assembled prompt
```text
I am building a research-grade foundation-model system for subseasonal-to-seasonal
(S2S) forecasting of extreme heat over mainland Vietnam, for a hydro-meteorological
research group. It will be written up for peer review and later made operational. The
paper's central claim - that self-supervised pretraining on ECMWF S2S forecast
trajectories adds out-of-sample skill beyond raw ECMWF, bias correction and AI models
trained from scratch - is exactly the claim that leakage or a weak baseline would fake.
So the standard is: a result that survives a hostile reviewer, including a null result.
This is a standalone project. Other S2S heat projects may exist in this environment
with different lead-week, threshold and member definitions. Do not import any of their
conventions; the definitions below are the only ones that apply here.
Act as a senior S2S scientist, an ML researcher in weather/climate foundation models,
a scientific Python engineer, and an independent reviewer hunting for leakage, invalid
assumptions, inconsistent definitions and weak verification - all at once.
- Do not modify, delete, rename or move any existing file or folder without my
explicit permission. You may create new files and folders in the working directory.
- Observed gridded Tx, station Tx and the climatic sub-region information are in the
existing 'data' subdirectory. Read them in place. Put derived data in NEW
subfolders (e.g. data/processed/, data/raw/ecmwf_s2s_1p5deg/); never copy, move or
restructure the original observation files.
- All outputs go under 'outputs'. Everything I am meant to check - audits,
inventories, diagnostics, verification summaries - is ALSO rendered as a
self-contained HTML page in 'outputs' (e.g. outputs/PHASE0_AUDIT.html), in addition
to the Markdown/CSV/NetCDF files named below.
- Read README.md and existing documentation, and inspect the repository, Python
environment, storage, CPU/GPU and any existing code before creating anything.
- ECMWF data-store credentials: use only what is already configured on the machine.
If none is configured, stop and tell me what file or setting is missing. Never ask
me to paste a key into chat and never write one into the repository.
- Never fabricate ECMWF variables, parameter codes, reforecast dates, archive
structure, API parameters, pressure levels, lead times or metadata. Inspect the
CURRENT ECMWF data-store metadata and generated request syntax; a remembered API is
not evidence.
- Paths, domains, variables, levels, model and training hyperparameters, CV folds,
probability thresholds, bootstrap settings and rare-event masks live in config
files, never hard-coded.
- Record every scientific and technical decision, with its reason and evidence, in
README.md / docs/, and one lesson per entry in docs/decision_log.md (one-line
summary on top; update an entry rather than duplicating it; delete it once shown
wrong).
- Never describe 0.25-degree products as 0.25-degree dynamical ECMWF forecasts. They
are statistical/AI downscaling conditioned on coarse S2S information plus fine-scale
static and climatological fields.
The project runs in phases (see ). THIS session does Phase 0 through the
target prototype - exactly the actions in - and then STOPS before any
bulk ECMWF download or model training.
Inside that scope, proceed without asking on anything reversible that follows from
this brief. Where a genuinely scientific choice is unclear, do not guess and do not
stop mid-phase to ask it alone: record it as a decision item with your recommendation,
the evidence behind it and the alternatives, and deliver all decision items together
in the gate report. The only reasons to stop early are: missing credentials, data that
cannot be found or read, or a finding that contradicts a LOCKED definition below.
Every later phase ends the same way: a gate report, one consolidated decision list,
stop. I will tell you when to proceed.
A leakage-safe, reproducible, HPC-ready system, working name S2S-FM-VN, that maps the
ECMWF S2S control reforecast on its coarse grid to daily and weekly extreme-heat
products on the 0.25-degree mainland-Vietnam observational grid, over leads LT0-LT45:
X_S2S(coarse, local + large-scale) -> encoder -> latent Z
-> 0.25-degree decoder (+ static and climatological fields) -> heat products
plus baselines, ablations, comprehensive verification, and a publication package that
answers the questions in . The model complements ECMWF's dynamical
integration; it does not replace it.
These are my decisions. Implement them exactly. Do not reopen, replace or "improve"
them. A sensitivity experiment against an alternative is allowed only if labelled as
such and never replaces the primary definition.
Predictors - ECMWF S2S reforecasts (s2s-reforecasts), originating centre ECMWF,
CONTROL forecast (cf) only. No perturbed members in the primary phase.
Retrieved and kept on the coarse grid, preferably 1.5 x 1.5 degrees,
unless current metadata shows a more appropriate archive-native grid.
Finer grids offered by the data store are remapping, not added
dynamical information, and are not the primary raw dataset.
Target grid - the exact 0.25-degree observational Tx grid over mainland Vietnam. The
product domain is every grid cell with valid observed Tx; do not drop a
coastal or border cell because it extends partly over sea or across the
border.
Obs - gridded Tx is PRIMARY (targets, training, CV, spatial verification);
station Tx is SECONDARY validation and is not described as independent
if the gridded product was built partly from those stations.
Period - 2003-2022 (overlap of reforecasts and observations).
Horizon - LT0-LT45, LT = lead day relative to initialization.
P90 - fixed climatological threshold, computed ONCE from observed Tx
2003-2022, per grid cell s and calendar day d, pooling all years' values
in a centred cyclic 31-day window (d-15 .. d+15) that wraps across the
year boundary. Handle leap years consistently and document exactly how
29 February is treated. Not monthly, seasonal, annual or fold-specific.
Archive: P90_2003_2022.nc
EHD thresh - EHD_THRESHOLD(s,d) = min(P90(s,d), 35.0 degC). min, never max.
Archive: EHD_THRESHOLD_2003_2022.nc
EHD - EHD(s,t) = 1 if Tx(s,t) >= EHD_THRESHOLD(s,d), else 0.
Exceedance - exceedance(s,t) = max(Tx(s,t) - EHD_THRESHOLD(s,d), 0)
Weeks - W1=LT00-06 W2=LT07-13 W3=LT14-20 W4=LT21-27 W5=LT28-34 W6=LT35-41
W7=LT42-45 is a 4-day PARTIAL week: verified separately, labelled as
partial, never pooled with or compared directly to W1-W6 without explicit
normalisation. Headline verification is W1-W6. No overlapping moving
7-day windows as forecast or verification products.
EHD_COUNT - number of EHDs in the week; 0..7 for W1-W6, 0..4 for W7.
EHE_WEEK - 1 if EHD_COUNT >= 1.
HW2_WEEK - 1 if EHD_COUNT >= 2, anywhere in the week, no consecutive-day condition.
HW episode - >= 2 CONSECUTIVE EHDs, detected on the CONTINUOUS daily sequence BEFORE
any weekly aggregation. HW2_WEEK and the episode definition are
different concepts; keep them distinct in code, docs and figures.
Severity - SEV_WEEK(Wk) = sum over EHDs in Wk of (Tx - EHD_THRESHOLD), degC day.
Episode e - D_e = t_end - t_start + 1 (days); I_mean,e = mean exceedance (degC);
I_max,e = max exceedance (degC); Sev_e = summed exceedance (degC day).
Cross-week accounting (an episode crossing a week boundary is ONE event):
HW_ONSET(Wk) / HW_ONSET_COUNT(Wk) - onset counted once, in the week it starts
HW_ACTIVE(Wk) - 1 for every week the episode overlaps
HW_DAYS(Wk) - only the episode days that fall inside Wk
HW_SEV_ACTIVE(Wk) - only the exceedance that falls inside Wk
Full duration and full severity belong to the onset week, for onset-based summaries
(mean, max and P95 duration of episodes starting in Wk). Never duplicate full
duration or severity across overlapping weeks.
Censoring - detect episodes with a labelling buffer of about 6 days before LT0 and 6
after LT45 where data allow; the official horizon stays LT0-LT45. Flag episodes
left_censored / right_censored and store the flags in event tables. A left-censored
episode contributes to HW_ACTIVE and HW_DAYS but is not an onset. A right-censored
episode contributes to onset and active-day statistics but never to full-duration
statistics.
Fixed contingency reference probability threshold: 0.5.
Category A - fixed predictand definitions: P90 and EHD_THRESHOLD. Computed once from
2003-2022, identical in every experiment and fold. Using them is not forecasting-model
leakage: they define WHAT the event is.
Category B - everything the forecasting system estimates, fitted on TRAINING years
only inside every fold: predictor climatology, lead-dependent anomalies, means and
standard deviations, normalisation, bias correction, calibration, feature selection,
PCA or any learned reduction, all ML, foundation-model and downstream parameters,
early-stopping choices, and any optimised probability threshold. For headline
results a held-out year enters none of these - including foundation pretraining.
An all-years (transductive) pretraining experiment is allowed only as a clearly
labelled side experiment, never reported as out-of-sample skill. Never optimise any
threshold on held-out test data. Write this rule into README.md and check it at every
gate.
Before any bulk retrieval, establish from the current data-store metadata, request
syntax and returned files - not from memory:
- valid reforecast/reference-date relationships and which reference dates yield
hindcast years covering 2003-2022; the actual number of initializations per year
- lead-time structure, forecast length, temporal frequencies
- surface and pressure-level variables, valid levels per variable, and how daily
maximum 2-m temperature (if offered) is aggregated and over which time window
- requested vs actually retrieved grid
- how the control forecast is identified
Then make ONE minimal retrieval (one initialization, a few leads, one surface and one
pressure-level variable, cf, coarse grid) and verify dimensions, grid, units, level
types, member type and valid dates from the GRIB metadata.
Reforecast consistency: before combining 2003-2022 samples, tabulate per
initialization - hindcast date, reference date, model cycle/version metadata where
available, forecast length, available leads and variables, any structural change. If
the 2003-2022 set spans more than one model cycle or configuration, document it and
propose defensible alternatives at the gate; never combine silently.
Produce: docs/ecmwf_s2s_inventory.md, metadata/ecmwf_variable_inventory.csv,
metadata/ecmwf_reforecast_inventory.csv (variable, short_name, level_type,
pressure_level, units, temporal_frequency, requested_grid, retrieved_grid,
reference date, hindcast date, lead time, member type, availability, notes). Estimate
storage and I/O for the proposed predictor set before proposing bulk retrieval.
Downloads, when approved, are resumable, logged, chunked and storage-aware.
Two coarse predictor branches; neither is upsampled to 0.25 degrees before the
encoder:
Local/regional - initial candidate 95E-120E, 5N-30N (Vietnam, Indochina, South
China Sea, southern China; regional circulation, moisture
transport, land-atmosphere processes).
Large-scale - initial candidate 40E-180E, 20S-50N (tropical Indian Ocean,
Maritime Continent, western/central Pacific, East Asia; ENSO, MJO,
monsoon, WNP subtropical high, teleconnections).
Domains may be adjusted after a scientific and computational audit; any change is
documented and justified.
Candidate variables, organised as required / recommended / optional in
config/variables.yaml once availability is confirmed: 2-m temperature, daily max (and
min if useful) 2-m temperature, MSLP, 10-m U/V, total precipitation, SST, soil
temperature and moisture, surface sensible and latent heat flux, relevant radiation;
T, Z, U, V, q at 1000/925/850/700/500/300/200 hPa. High-priority first set: T850,
Z500, U850, V850, U200, V200, q850, q700. Do not download everything available;
justify each group by scientific value against storage, I/O and redundancy.
0.25-degree decoder inputs: latitude, longitude, elevation, land mask, distance to
coast if available and justified, Tx climatology, P90 climatology, EHD-threshold
climatology, and region indicators from the R1-R7 information in 'data' - inventory
it and use it if valid; never invent or redraw region boundaries.
Audit the gridded Tx automatically: files, dimensions, coordinate names, longitude
convention, latitude ordering, calendar, units, missing values, time and spatial
coverage, exact grid spacing, valid-data mask. Convert to degC internally if needed;
never alter the originals. Build the valid mask from actual valid data. Locate and
audit station Tx the same way, and establish whether those stations fed the gridded
product.
S2S-FM-VN - starting design. The architecture below is my working hypothesis, not a
locked definition: deviate where evidence says so, and record the evidence.
ECMWF cf, coarse, LT0-LT45
-> local branch and large-scale branch, each: variable/level encoder -> spatial
encoder (small Swin, hierarchical ViT, ConvNeXt+attention or similar)
-> cross-scale fusion -> temporal/lead encoder over Z0..Z45 -> Z_S2S
-> 0.25-degree Vietnam decoder with static/climatology inputs
-> daily and weekly outputs
- Learnable variable, pressure-level, lead-time and calendar/season embeddings;
variables addable/removable through config. ClimaX/Aurora ideas only where useful.
- Do not collapse the whole LT0-LT45 trajectory into one global vector before
decoding.
- One shared backbone. Optional lead-regime adapters (e.g. LT0-10, 11-25, 26-45)
and region embeddings/adapters/heads. No per-lead or per-region backbones.
- Size: prototype ~5-10 M parameters; base ~15-30 M only if validation skill,
stability and compute justify it. Track parameters, GPU memory, train and
inference time, disk and checkpoint size.
Pretraining (self-supervised, coarse predictor space only; never reconstruct
interpolated 0.25-degree ECMWF fields): variable/level masking, spatial patch masking,
lead-time masking (single days and contiguous intervals), and combinations;
L_pretrain = l_var*L_var + l_space*L_space + l_time*L_time. Add a component only if it
shows benefit.
Outputs (multi-task, daily evolution preserved): daily Tx_hat; daily P(EHD); for
W1-W6 the count distribution P(N=0..7); optionally direct P(EHE_WEEK), P(HW2_WEEK).
Derive E[N] = sum k P(N=k), P(EHE_WEEK) = 1 - P(N=0), P(HW2_WEEK) = 1 - P(N=0) -
P(N=1), which guarantees P(HW2) <= P(EHE). For W7 mask classes 5-7 or use a
horizon-aware head. If direct heads exist, compare them with count-derived
probabilities; also diagnose consistency between the daily P(EHD) head and the weekly
count head, and report the inconsistency rather than hiding it.
Fine-tuning loss (candidates): Huber/MAE-type for Tx, BCE/focal for daily EHD,
categorical or ordinal for counts, Brier-type calibration terms where justified,
episode-characteristic terms if they help.
Normalisation: absolute fields and, where appropriate, lead- and calendar-dependent
anomalies and standardised anomalies - all fitted on training years only.
Temporal alignment: valid_date = initialization_date + lead_day, aligned by actual
date, never by array index. Handle leap years, 29 February, missing observations,
missing leads and incomplete forecasts. Unit-test several initializations including
leap years.
Cross-validation: year-grouped only, never random splits. At least blocked-year CV;
leave-one-year-out and rolling-origin if feasible. Because held-out years may not
enter pretraining, pretraining is repeated per fold - estimate the cost of each fold
design and recommend one at the gate, with the effective training-sample count per
fold stated.
Baselines, all under the same folds:
B0 observed climatology
B1 raw ECMWF temperature-based (deterministic from cf; for probabilistic scores,
a simple in-fold calibration of raw exceedance to probability)
B2 bias-corrected ECMWF
B3 conventional ML - ridge/linear, logistic, XGBoost, MLP
B4 deep model from scratch - same or closely comparable downstream architecture,
no self-supervised pretraining
FM1 pretrained local-only encoder
FM2 pretrained dual-scale encoder
Primary comparisons: FM1 vs B4, FM2 vs B4, FM2 vs B3, FM2 vs B1/B2.
Ablations (staged screening if full factorial is too costly): pretraining vs none;
local vs local+large-scale; absolute vs absolute+anomalies; surface vs
surface+pressure levels; with/without SST; with/without soil variables; with/without
temporal encoder; with/without 0.25-degree static/climatological decoder inputs.
Verification is a core module. Headline results use only cross-validated,
out-of-fold predictions. Everything is lead/week-specific, spatially explicit,
characteristic-specific, probabilistic where relevant, and episode-aware.
Forecast-side episodes: episode detection needs a binary daily forecast series.
Derive it by one stated rule applied identically to every model (for example daily
P(EHD) >= 0.5, the fixed reference threshold); propose the rule at the gate. Pre-LT0
context for forecast episodes may use observations available at initialization; the
post-LT45 buffer is for observed labels only.
Continuous (Tx, EHD_COUNT, HW_DAYS, SEV_WEEK): bias, MAE, RMSE, Pearson, Spearman.
Count distribution: additionally RPS and RPSS against climatology - never only via
expected-value errors.
Categorical (EHE_WEEK, HW2_WEEK, HW_ACTIVE, HW_ONSET at 0.5): store H, M, FA, CN,
totals, observed and forecast event counts; compute POD, POFD, FAR, FBI, CSI, ETS,
HSS, TSS/PSS, EDI, SEDI, with a documented finite-sample correction for POD/POFD of 0
or 1 in EDI/SEDI. Optimised thresholds, if any, from training/validation data only.
Probabilistic: Brier score, BSS, ROC AUC, PR AUC, reliability diagrams, calibration
curves, sharpness histograms.
Rare-event control: store n_samples, n_events, n_nonevents for every metric; mask
cells below configurable minima (start at 10 events and 10 non-events; sensitivity
at 5/10/20). Never fill masked or unstable metrics with zeros.
Event matching (for onset, duration, intensity, severity): per grid cell, detect
observed and forecast episodes on continuous daily sequences; match on >= 1 day
overlap, one-to-one, maximising overlap or temporal IoU; unmatched forecast = false
alarm, unmatched observed = miss. Document the algorithm. On matched events: onset
error (days: bias, MAE, RMSE); duration, severity (degC day) and mean/max intensity
(degC) errors (bias, MAE, RMSE, Pearson, Spearman) - complete events only for
duration and severity.
Physical characteristics, OBS and FORECAST per week and grid cell, plus FCST-OBS where
meaningful: mean EHD_COUNT; EHD, EHE_WEEK, HW2_WEEK, onset and active frequency; mean
HW_DAYS; mean, max, P95 duration; mean, max intensity; mean, max, P95 episode
severity; mean and max SEV_WEEK.
Maps: every W1-W6 metric and characteristic above as 0.25-degree maps with consistent
masks, scales and metadata. Distributions: lead-dependent violin/box plots across
valid grid cells (median, Q25/Q75, whiskers, optional P05/P95, spatial-mean marker)
for the skill metrics and the characteristics, paired OBS-vs-FCST violins for the
characteristics. Grid cells are a descriptive spatial distribution, not independent
replicates - say so on the figures. Multi-model: comparison boxplots by week,
detailed violins for selected models, lead curves; never a wall of overlapping
violins. Regional (R1-R7): lead curves and concise tables for EHD_COUNT MAE/CORR,
BSS, POD, FAR, FBI, EDI, SEDI, HW frequency, mean duration, mean severity.
Uncertainty: year-block (or initialization-block) bootstrap only; never resample
grid-days independently. Model-difference maps (e.g. delta MAE, BSS, SEDI for FM vs
XGBoost and FM vs B4) with 95% CI; mark cells whose CI excludes zero; do not
over-read isolated significant cells; apply FDR/field significance where warranted.
Station verification, after the gridded system: station-specific fixed P90 (same
rule), EHD thresholds and characteristics; forecasts sampled to station coordinates
by a documented method; station maps, distributions and lead curves, labelled as not
fully independent if applicable.
Machine-readable first, figures second.
- outputs/metrics/verification_metrics.nc (dims model, week, metric, lat, lon or
equivalent), all variables with units and metadata, including matched, observed
and forecast event counts.
- CSVs: data_inventory, ecmwf_reforecast_inventory, target_event_frequency,
cv_fold_definition, count_metrics_by_week, event_metrics_by_week,
probabilistic_metrics_by_week, metrics_by_region, event_characteristics_by_week,
matched_event_metrics, model_comparison, ablation_results,
bootstrap_skill_differences - each with sample size, event count where relevant,
model, week/lead and CV definition.
- Maps under outputs/verification/W1..W6/{TX, EHD_COUNT, EHE_WEEK, HW2_WEEK,
HW_EPISODE, HW_ACTIVE, HW_ONSET, DURATION, INTENSITY, SEVERITY, PROBABILITY}/ with
names like W3_EHD_COUNT_OBS.png, W3_EHD_COUNT_BIAS.png, W3_HW_DURATION_MAE.png.
W7 in its own clearly labelled supplementary folder.
- Paper figure set: domains; observed climatology (P90, EHD/EHE/HW frequency,
severity); skill vs lead; violin skill vs lead; OBS vs FCST characteristics;
representative W1/W3/W6 maps; FM added value (vs B4, vs XGBoost); large-scale
added value (FM2 minus FM1).
- An HTML summary page in outputs/ for every gate.
Layout (adapt names as needed, keep the separation): README.md; environment file;
config/{paths,data,variables,domains,targets,model,training,verification}.yaml;
src/{data, targets, models, training, verification, plotting, utils}; numbered
scripts/00_environment_check.py .. 11_make_figures.py; scripts/12_independent_validator.py;
tests/; slurm/; outputs/{metrics, maps, figures, models, events, logs}.
Slurm scripts for download, preprocessing, targets, pretraining, per-fold
fine-tuning (job arrays), event detection, verification, bootstrap and figures -
written after inspecting the actual HPC, with checkpoint/restart, mixed precision,
seeds, gradient accumulation, early stopping.
Independent validator: re-derives, by separate code, train/test separation, P90
window/wrap/leap handling, EHD, weekly counts and flags, episodes and cross-week
continuity, onset/active/days, severity, censoring, contingency counts and scores,
Brier, selected MAE/RMSE/CORR, event matching and duration/severity errors. Writes
outputs/metrics/validator_report.csv and outputs/metrics/leakage_audit.txt.
Unit tests for every target and score function, including at minimum:
EHD_COUNT=0 -> EHE=0, HW2=0; =1 -> EHE=1, HW2=0; >=2 -> EHE=1, HW2=1.
EHDs at LT6 and LT7 -> ONE 2-day episode, not two.
Episode spanning W1-W2 -> one onset; HW_ACTIVE=1 in both; HW_DAYS split by days.
P90 window inclusion, cyclic wrap, leap years, 29 Feb; valid-date alignment;
censoring; matching; RPS/RPSS; every categorical score.
0 Audit 1 Minimal ECMWF retrieval 2 Target prototype 3 Small end-to-end prototype
(reduced variables, years, model: download -> preprocess -> pretrain -> fine-tune ->
daily -> weekly -> events -> verification -> maps/violins) 4 Full acquisition
5 Baselines B0-B4 6 Foundation pretraining 7 Cross-validated fine-tuning, strict
out-of-fold predictions 8 Full verification 9 Ablations 10 Publication package
(final tables, figures, archived configs, methods documentation, reproducibility
package). Each phase starts only after the previous gate is approved.
This session - do these, then stop:
1. Inspect the repository, README.md and docs; environment, storage, CPU/GPU.
2. Audit gridded Tx, station Tx and the region information in 'data'.
3. Inspect current ECMWF data-store S2S reforecast metadata and request syntax;
confirm control-forecast and coarse-grid retrieval.
4. Determine the valid 2003-2022 retrieval strategy and the reforecast consistency
table; inventory variables and levels.
5. Estimate storage for the proposed predictor set; propose final local and
large-scale domains and the predictor list.
6. One minimal retrieval; verify its GRIB metadata.
7. Implement fixed P90 (2003-2022, cyclic +/-15 days) and unit-test it including
year wrap and leap years; implement EHD threshold, daily EHD, EHD_COUNT,
EHE_WEEK, HW2_WEEK, continuous episode detection with censoring flags, onset /
active / HW_DAYS, preliminary severity; explicitly test an episode crossing W1/W2.
8. Write docs/PHASE0_AUDIT.md and outputs/PHASE0_AUDIT.html,
metadata/ecmwf_variable_inventory.csv, metadata/ecmwf_reforecast_inventory.csv,
config/variables.yaml, config/domains.yaml, config/targets.yaml,
config/verification.yaml.
9. STOP before bulk download. Gate report: what was verified (with the file that
shows it); what remains uncertain; estimated data volume; proposed predictor
list; proposed architecture and parameter count; proposed CV fold design with
per-fold pretraining cost; compute requirement; and ONE consolidated list of
decisions needing me, each with your recommendation and evidence.
Decide the approach yourself, but handle each of these explicitly and log it in
docs/decision_log.md - they fail quietly:
- Held-out years leaking into pretraining, normalisation, anomaly climatology,
calibration, early stopping or threshold choice. Per-fold pretraining is expensive,
which is exactly the pressure that produces this leak.
- Assuming 104 initializations per year or a single model cycle across 2003-2022
without checking the reference-date structure and model-version metadata.
- Day misalignment: observed Tx is a local (UTC+7) calendar-day maximum, ECMWF daily
or 6-hourly maxima are defined in UTC; LT0 may be a partial day. Define the
mapping explicitly and test it.
- Treating cf-only output as if it carried ensemble spread; every probability here
comes from a learned or in-fold calibrated model.
- Interpolating the predictor branches to 0.25 degrees "to make the shapes fit".
- Detecting episodes after splitting into weeks, or counting a cross-week episode as
two onsets, or giving its full duration/severity to every overlapping week.
- Using right-censored episodes as complete durations.
- Comparing unmatched forecast and observed event durations.
- Pooling W7 with W1-W6.
- Reporting rare-event scores without sample counts, or filling masked cells with 0.
- Bootstrapping grid-days or treating grid cells as independent samples.
- A pretraining set too small (about 20 years x the actual initializations per year,
one member) to show benefit - if FM does not beat B4, that is the finding; report
it rather than tuning until it does.
- Building a general-purpose foundation-model framework instead of this one system.
The project succeeds only if it can answer, with cross-validated evidence: (1) does
ECMWF S2S hold useful extreme-heat skill over Vietnam; (2) how skill degrades W1->W6;
(3) does conventional AI beat raw/bias-corrected ECMWF; (4) does pretraining beat the
same architecture from scratch; (5) does large-scale context beat local-only; (6)
which predictor groups add skill; (7) are spatial distributions of frequency,
duration, severity and intensity reproduced; (8) are onset and active periods
predictable; (9) are cross-week events handled correctly; (10) are gains robust
across years and regions; (11) are probabilities calibrated; (12) at what lead does
skill reach the no-skill level.
Reason step by step in your thinking before writing code: reconcile the leakage
categories, the per-fold pretraining structure, the valid-date alignment and the
cross-week episode accounting against each other, because an error in any one
invalidates the verification. Keep that scratch work out of your messages.
From Phase 3 on, delegate weakly dependent workstreams (targets and events,
baselines, FM pretraining/fine-tuning per fold, verification, the independent
validator) with bounded scope, the fold structure to respect, and the exact output
format. The validator must not share code with the verification module. Synthesise
comparisons yourself; do not let one workstream judge another's results. In Phase 0,
work directly.
Do not add features or abstractions the workflow does not need. Validate at real
boundaries (file I/O, config, the ECMWF API, external data), not everywhere.
Attribute claims to their source. Where ECDS metadata and documentation disagree,
where gridded and station Tx disagree, or where models rank differently under
different scores, say so and state which you weighted more heavily and why.
Before reporting anything as verified, done or skilful, point to the concrete result
from this session - a returned GRIB header, a test run, a score file, a figure. If a
step failed, give the actual error. Never optimise merely to raise verification
skill; any choice that could inflate skill is documented and replaced by a more
conservative alternative where possible.
Surface through the harness's user-facing channel, and as HTML in outputs/, the
deliverables as they are produced - the data audit, the minimal-retrieval check, the
target prototype tests, the gate report. Deliverables, not narration.
Before finishing any phase, check silently: are the locked definitions untouched; is
every Category B quantity fitted on training years only; are W7 and cross-week events
handled as specified; does every metric carry its sample counts; is every claim tied
to a file; were any files moved, modified or deleted without permission; did anything
run beyond the current gate? Fix what is wrong. Do not show the checklist.
Open your final message with one sentence saying what was verified and whether
anything blocks the next phase. Details after. Write it as a re-grounding for someone
who did not watch the run: complete sentences, file names spelled out, none of the
shorthand you invented while working.
```
## 4. Omitted but considered
- **OP-03 (unattended autonomy)** — excluded. The draft's "ask for clarification" rule and the hard STOP before bulk download are both incompatible with "proceed without asking". Replaced by the gated rhythm: autonomous inside a phase, one consolidated decision list per gate.
- **The context-budget reassurance line** — dropped. It exists to keep an unattended agent from quitting early; here the agent is supposed to stop at each gate.
- **A separate MEMORY.md / CLAUDE.md** — not added; the draft routes decisions to README.md and docs/, so the lesson store is `docs/decision_log.md` rather than a second convention.
- **The draft's "Very important notes" as prose preamble** — folded into `` so the file-safety rules sit next to the rules they interact with (e.g. `data/` holds the user's originals).
- **Full verbatim restatement of every map and violin list** (draft §52, §55, §62 repeat the same metric lists three times) — consolidated once in `` and ``. Nothing was removed, only de-duplicated; three copies drift.
- **A fixed CV fold design** — withheld. The draft allows blocked-year, LOYO or rolling-origin; with per-fold pretraining the cost difference is large, so the agent must estimate it and recommend at the gate.
- **Vague intensifiers** — none; OP-02 and OP-13 do that work.
## 5. Changes from the draft, and why
| # | In the draft | In the prompt | Reason |
|---|---|---|---|
| 1 | "Do not make assumptions. Ask before proceeding" + a 24-step first-action list | Gated: autonomous within Phase 0–2, one consolidated decision list at the STOP | Otherwise the agent either stalls on every step or ignores the rule |
| 2 | Proposed `data/raw/…/observations/` | Originals read in place; derived data only in new subfolders | The draft's structure implies moving the user's obs, which the file-safety rule forbids |
| 3 | ECMWF API use, no credential rule | Use pre-configured credentials only; stop if missing; never paste keys | Credential entry is prohibited for the agent; better to say so up front |
| 4 | "Present checks as HTML in outputs" (one line) | HTML page per gate, named in each deliverable list | Made it a concrete deliverable so it is not forgotten |
| 5 | Region masks conditional ("if R1–R7 exist") | "Region info is in `data`: inventory and use it; never redraw" | The draft itself says the sub-region information is supplied |
| 6 | cf only | Stated consequence: no ensemble spread; B1 probabilities via in-fold calibration | Otherwise B1 has no probabilistic scores and comparisons are uneven |
| 7 | Episode matching on forecasts | Explicit rule needed to binarise daily P(EHD) before episode detection; pre-LT0 context from obs only | Forecast episodes are undefined without it |
| 8 | Temporal alignment | Local (UTC+7) day vs UTC daily max; partial LT0 | Silent one-day shift in Tx is a classic S2S verification error |
| 9 | "20 years" assumed consistent | Must check reference-date/model-cycle structure behind 2003–2022 | Reforecasts are tied to a reference date and model cycle; a 20-year span can straddle cycles |
| 10 | Per-fold pretraining implied | Cost estimate + fold recommendation at the gate; FM-vs-B4 null result explicitly reportable | ~2,000 single-member trajectories is small for pretraining; the honest outcome may be "no added value" |
| 11 | Daily and weekly heads | Diagnose daily-P(EHD) vs count-head consistency | Two heads can disagree; the paper should show whether they do |
### Ghi chú cho bạn (không thuộc bản prompt)
Hai điểm tôi **giữ nguyên theo đúng quyết định của bạn** nhưng muốn bạn biết trước khi chạy:
- **P90 cố định 2003–2022 cho mọi fold.** Khác với Stage 1 (ở đó Tx90p phải fit không dùng năm kiểm định). Cách phân loại Category A/B của bạn là hợp lý về mặt định nghĩa, nhưng reviewer có thể vẫn hỏi. Nếu muốn, có thể thêm một thí nghiệm độ nhạy "P90 theo fold" ở Phase 9 để trả lời sẵn — tôi chưa đưa vào prompt vì bạn chưa yêu cầu.
- **Chỉ dùng control forecast.** Không có spread từ ensemble, nên mọi xác suất đều do mô hình học ra; baseline ECMWF thô chỉ có xác suất sau khi hiệu chỉnh trong fold.
Nếu đồng ý thêm thí nghiệm P90-theo-fold, chỉ cần thêm một dòng vào danh sách ablation trong ``.