Please proceed with:

```text
FC_DATE = 2015-01-01
HDATE   = 2003-01-01
GRID    = 1.5/1.5
```

The date pair is valid according to the ECMWF S2S reforecast structure: the reforecast set associated with the 1 January 2015 real-time forecast contains hindcasts initialized on 1 January for the preceding 20 years, including 1 January 2003.

For the **S2S-FM-VN predictor stream**, the decision is now fixed:

```text
ECMWF S2S predictors: 1.5° × 1.5°
Vietnam target/output grid: 0.25° × 0.25°
```

Do not retrieve the primary predictor stream at 0.25°.

The intended architecture remains:

```text
ECMWF S2S cf at 1.5°
        ->
local + large-scale coarse encoders
        ->
spatiotemporal latent representation Z
        ->
0.25° static/climatological Vietnam decoder
        ->
daily Tx / EHD products at 0.25°
        ->
continuous episode detection
        ->
weekly EHD/EHE/HW characteristics
```

Therefore, for the minimal retrieval:

1. Use `FC_DATE=2015-01-01`.
2. Use `HDATE=2003-01-01`.
3. Use `grid=1.5/1.5`.
4. Retrieve only the smallest valid cf sample.
5. Inspect the returned GRIB header.
6. Verify grid, variable, units, level type, member type, reference time, hindcast date, lead/step, and valid time.
7. Document the actual archive structure.
8. Freeze `canonical_schema.md`.
9. Continue immediately with Phase 1 preprocessing and temporal alignment if the retrieval and metadata checks pass.
10. Do not start the full 2003–2022 bulk download yet.

If this officially valid historical pair is unavailable through the current CDS/ECDS interface for an archive/API-specific reason, report the exact API error rather than substituting another guessed FC_DATE/HDATE pair.


######

Request on CDS ECMWF:
I have CDS Key already, and I can sussecsfully download using my script like below:
#!/usr/bin/env python3
import os
import sys
import cdsapi

DATASET = "s2s-reforecasts"

FC_DATE = "yver-mm-dd"
HDATE   = "yyyy-mm-dd"
TARGET  = "Srf_daily_cf_yyyy-mm-dd" #.nc"
TMP_TARGET = TARGET + ".part"

try:
    fc_year, fc_month, fc_day = FC_DATE.split("-")
    hyear, hmonth, hday = HDATE.split("-")
except ValueError as exc:
    raise ValueError(
        f"Invalid date placeholders after sed: FC_DATE={FC_DATE!r}, HDATE={HDATE!r}"
    ) from exc

PARAMS = [
    "167","168","235","59","136","228164","228086","228087","228095","228096",
]
LEADTIME_HOUR = [
    "0_24",   "24_48",  "48_72",  "72_96",  "96_120", "120_144","144_168","168_192","192_216","216_240",
    "240_264","264_288","288_312","312_336","336_360","360_384","384_408","408_432","432_456","456_480",
    "480_504","504_528","528_552","552_576","576_600","600_624","624_648","648_672","672_696","696_720",
    "720_744","744_768","768_792","792_816","816_840","840_864","864_888","888_912","912_936","936_960",
    "960_984","984_1008","1008_1032","1032_1056","1056_1080","1080_1104",
]

request = {
    "origin": "ecmwf",
    "year": fc_year,
    "month": fc_month,
    "day": fc_day,
    "time": "00:00",
    "hyear": [hyear],
    "hmonth": [hmonth],
    "hday": [hday],
    "level_type": "single_level",

    "variable": PARAMS,

    "forecast_type": "control_forecast",
    "leadtime_hour": LEADTIME_HOUR,
    "data_format": "grib2", #"netcdf",
    "area": [24, 101, 8, 111],
    "grid": "0.25/0.25",
}

def main() -> int:
    if os.path.exists(TARGET) and os.path.getsize(TARGET) > 0:
        print(f"File exists, skip: {TARGET}")
        return 0

    if os.path.exists(TMP_TARGET):
        os.remove(TMP_TARGET)

    print(f"Downloading {DATASET}: forecast date {FC_DATE}, hdate {HDATE}")
    print(f"Output: {TARGET}")
    print(f"Params: {'/'.join(PARAMS)}")

    client = cdsapi.Client(url="https://ecds.ecmwf.int/api")
    client.retrieve(DATASET, request).download(TMP_TARGET)

    if not os.path.exists(TMP_TARGET) or os.path.getsize(TMP_TARGET) == 0:
        raise RuntimeError(f"Download failed or empty file created: {TMP_TARGET}")

    os.replace(TMP_TARGET, TARGET)
    print(f"Done: {TARGET} ({os.path.getsize(TARGET)} bytes)")
    return 0


if __name__ == "__main__":
    sys.exit(main())

To here
##############

 Want me to go ahead and draft the Phase 1 preprocessing + temporal-alignment code (device-agnostic, D4) now so it's ready the moment the first
  GRIB lands, or wait until after the minimal retrieval verifies the real archive structure?
Please **wait for the minimal ECMWF retrieval to verify the real archive structure before implementing the full Phase 1 preprocessing and temporal-alignment code**.

The returned GRIB must first confirm:

* actual grid and grid ordering;
* dimensions;
* time/step representation;
* initialization/reference time;
* valid-time encoding;
* lead-time units;
* surface vs pressure-level representation;
* variable short names;
* units;
* member/control-forecast metadata;
* pressure-level coordinate conventions;
* any unexpected archive-specific structure.

I do not want preprocessing code to be built around assumed ECMWF metadata and then patched later.

However, while waiting for the first GRIB, you MAY prepare only metadata-independent scaffolding, including:

* directory/module structure;
* configuration schemas;
* abstract data interfaces;
* logging;
* unit-test framework;
* generic valid-time helper functions;
* test cases using synthetic arrays;
* device-agnostic CPU/CUDA utilities;
* placeholder preprocessing pipeline interfaces;
* target-building code that depends only on the observational Tx data;
* P90/EHD/EHE/HW episode unit tests;
* generic temporal-alignment tests using synthetic initialization and lead dates.

In particular, it is fine to prepare functions conceptually like:

```text
inspect_grib()
extract_valid_times()
standardize_coordinates()
standardize_units()
align_forecast_to_observation()
```

but do NOT hard-code actual GRIB keys, dimension names, level conventions, or ECMWF-specific mappings until the test file has been inspected.

So the sequence should be:

```text
1. Minimal ECDS retrieval
2. Inspect real GRIB metadata
3. Document actual archive structure
4. Freeze the internal canonical data schema
5. Implement Phase 1 preprocessing against that schema
6. Run preprocessing on the minimal sample
7. Verify temporal alignment against observations
8. Only then expand to the prototype dataset
```

Please continue all observational-target work in parallel, because it does not depend on the ECMWF GRIB structure.
To here/
#####



D1 — ECMWF/ECDS credential

Use the current ECDS/CDS-API authentication, not the legacy ECMWF Web API credential.

The configuration should be:

$HOME/.cdsapirc

with the ECDS endpoint:

url: https://ecds.ecmwf.int/api
key: <my personal access token>

Do not use ~/.ecmwfapirc for the S2S public-data workflow unless there is a separate demonstrated requirement for a legacy/private ECMWF service.

I will refresh/configure the ECDS token locally. Do not ask me to paste the token into the chat or repository.

After the credential is available, test authentication with the smallest possible ECDS request before doing anything larger.

D2 — ECMWF access path

Use:

ECDS -> cdsapi -> s2s-reforecasts

as the primary access path.

Do not use the legacy ECMWF Web API or direct MARS access for this project.

Use the current ECDS-generated CDS-API request syntax. Either the native/“beautified” CDS request or MARS-like request syntax accepted through cdsapi is acceptable, but prefer the native ECDS/CDS syntax when practical.

Before bulk retrieval, reproduce a request directly from the current ECDS “Show API request code” output and verify it.

Primary retrieval requirement:

dataset: s2s-reforecasts
origin: ECMWF
forecast member: control forecast only (cf)
predictor grid: coarse, preferably 1.5° × 1.5°

Do NOT retrieve the predictors at 0.25° for the primary workflow.

The intended modelling architecture is:

ECMWF S2S coarse (~1.5°)
        ->
foundation/spatiotemporal encoder
        ->
latent representation Z
        ->
0.25° Vietnam decoder
        ->
daily/weekly extreme-heat products at 0.25°

D3 — Vietnam target grid and regional mask

Confirmed.

Use:

VnGAD

as the authoritative 0.25° target grid for:

model outputs;
Tx targets;
EHD/EHE/HW products;
verification maps.

Use:

mask7reg_VnGAD

as the authoritative mask for regional R1–R7 analyses.

Before use, verify programmatically that:

latitude coordinates align exactly with VnGAD;
longitude coordinates align exactly;
orientation/order of latitude is handled correctly;
grid dimensions match;
missing-value conventions are understood;
region codes are correctly identified.

Do not regenerate or redefine R1–R7 if mask7reg_VnGAD already provides them.

D4 — GPU vs CPU prototype

Proceed now with a CPU-sized prototype.

GPU availability is NOT a blocker for Phase 0, Phase 1, Phase 2, or the small end-to-end prototype.

The CPU prototype should be deliberately small, for example:

limited years;
reduced predictor set;
reduced domain if necessary;
very small encoder;
very few epochs;
small batch size.

Its purpose is to validate:

download
-> preprocessing
-> temporal alignment
-> target construction
-> foundation-model data pipeline
-> forward/backward pass
-> decoding to 0.25°
-> event detection
-> verification

not to establish final forecast skill.

Design all model/training code to be device-agnostic:

CPU / CUDA

and record approximate memory requirements.

A GPU will be required/strongly preferred for the full foundation-model pretraining and cross-validation stages, but do not wait for a GPU to complete the prototype pipeline.

D5 — Station Tx

No station Tx file is required at this stage.

Proceed with:

VnGAD gridded Tx

as the primary observational target and verification dataset.

Keep station verification implemented as an OPTIONAL later module.

Do not let missing station data block:

target construction;
model training;
cross-validation;
gridded verification;
Phase 0–3 development.

When station data become available later, we will add:

station P90
station EHD/EHE/HW
station-level verification

and document whether the stations are independent of the VnGAD gridding procedure.

D6 — GRIB handling

For now, standardise the raw-data workflow on:

ecCodes command-line tools + CDO

for robust GRIB inspection and preprocessing.

Use tools such as:

grib_ls
grib_get
cdo

to inspect metadata and, where appropriate, convert/reorganize raw GRIB into analysis-ready NetCDF.

For the machine-learning/data-analysis pipeline, prefer:

xarray + NetCDF/Zarr

after preprocessing.

cfgrib / Python ecCodes bindings may be installed later and may be used if they improve the workflow, but they are not a blocker and should not be made a mandatory dependency for Phase 0.

Therefore the preferred architecture is:

ECDS GRIB
   ->
ecCodes/CDO validation and preprocessing
   ->
NetCDF/Zarr
   ->
xarray
   ->
AI/FM pipeline

Always retain the original downloaded GRIB files, at least until the processed products have been independently verified.

D7 — 45 valid VnGAD cells with no regional code

Do NOT assign these 45 cells arbitrarily to R1–R7.

Handle them as follows:

If they are valid mainland-Vietnam VnGAD cells with valid Tx, retain them in:
model training;
national 0.25° forecast products;
Vietnam-wide maps;
Vietnam-wide verification;
national boxplot/violin distributions.
Exclude them only from:
R1–R7 regional aggregation;
region-specific statistics;
region-specific lead curves.
Give them a dedicated region code such as:
region = 0

or:

region = UNASSIGNED

internally rather than setting them to missing data.

Produce a diagnostic map and table containing:
latitude;
longitude;
VnGAD validity;
existing region code;
land/coast/border status if identifiable.
Save something like:
metadata/unassigned_region_cells.csv
figures/unassigned_region_cells.png

so that these 45 cells can be reviewed later.

Do not remove them from the national domain unless subsequent inspection shows that they are genuinely outside the intended mainland-Vietnam target domain.

Thus:

National analysis: INCLUDE
R1–R7 analysis: EXCLUDE until region assignment is resolved

This treatment must be documented in README and verification metadata.

With these decisions, D3–D7 should no longer block the current work.

After the ECDS credential is configured, proceed with the minimal ECDS retrieval test and continue the audit/prototype workflow. Do not begin the full 2003–2022 bulk download yet.