# Data Inventory & Reconciliation — S2S Extreme-Heat Forecast for Vietnam Date compiled: 2026-09-12. Compiled by reading actual file headers and samples, not assuming filenames/units/grids. All paths are relative to the working directory `/home/tanpv/Progs/claude/QG_1`; every data item is a symlink into `/work/projects/G09/QG.25.196/` (read-only; nothing here is modified). --- ## TL;DR — four findings that change the design 1. **No ensemble.** Every S2S reforecast field is a single **control forecast** (`_cf_`) member. There are 2080 Tx init files = 104 inits/yr × 20 yr (2003–2022), one member each. No perturbed (`pf`) members exist anywhere in the archive. The "2,080 cases" in the prompt = 104 × 20 single-member inits. **Consequence:** ensemble-native methods (EMOS/NGR from spread, CRPS-trained ensemble calibration, rank histograms, spread–error, ensemble exceedance fraction for P(EHD)) cannot be driven by an ensemble. The calibration is re-framed as **single-member MOS / distributional regression**: a model that maps the deterministic forecast (+ covariates) to a *predictive distribution* and hence to P(EHD). This is a method adaptation forced by the data, not a scope change — see CLAUDE.md §Calibration. 2. **No downscaling step is needed as delivered.** The S2S Tx reforecast is already on the **exact same 0.25° grid as the gridded obs** (lon 101–111°E, 41 pts; lat 8–24°N, 65 pts; identical coordinate values). The archive was interpolated to 0.25° upstream. **Consequence:** the prompt's "coarse S2S grid → downscaling" requirement is moot for this dataset; the "regrid-before-calibrate" known-failure-mode is *already baked in* upstream and becomes a documented limitation rather than a design choice. 3. **Forecast Tx is 6-hourly `mx2t6`, not daily.** Daily Tx must be derived as the daily max of the four 6-hourly `mx2t6` values. Forecast length = 184 steps × 6 h = 46 days (days 0–45), matching the six lead blocks exactly. 4. **Region labels are available two ways and station files are region-tagged.** The mask file carries a gridded `reg` field (integer 1–7) on the 0.25° grid, and each Tx station file's header line encodes its region code (1–7) as the 4th field. The shapefile is retained for the publication map only. --- ## 1. S2S reforecast (ECMWF), `data/EC/` Control forecasts only, per-init NetCDF, organised `EC///_..._.nc`. | Group dir | Variable(s) | Cadence | Files | |---|---|---|---| | `Tx_tm/` | `mx2t6`, `mn2t6` (max/min 2 m T in last 6 h, **K**, packed short) | 6-hourly | 2080 | | `Srf_daily_ave/` | `t2m, d2m, skt, cape, tcw, tcc, sm20, sm100, st20, st100` | daily mean | 2080 | | `Srf_24hour/` | `sshf, slhf, msl, u10, v10, ssrd, strd, ssr, str, ttr, cp, tp` | 24-h | 2080 | | `T850/` | T at 850 hPa | — | 2080 | | `H500/` | geopotential height 500 hPa | — | 2080 | | `T_plev/ H_plev/ W_plev/ QUV_plev/(pquv)` | multi-level T, Z, W, moisture/wind | — | 2080 each | | `UV200/` | 200 hPa wind | — | **707 (incomplete)** | - **Grid:** longitude 101.0–111.0°E (41 pts, +0.25°); latitude 8.0–24.0°N (65 pts, −0.25°, i.e. N→S). Identical to obs & mask grids. - **Time:** `int hours since 1900-01-01`, gregorian. First stamp = init + 06:00; 184 steps at 6 h → last = init + 45.75 d. So **day 0 = first 24 h after init**, through day 45. - **Inits:** 104/yr on **Thursday and Sunday** (52 + 52). Years **2003–2022** (20 yr). - **Packing:** `mx2t6` stored as `short` with `scale_factor`/`add_offset`; xarray unpacks to K automatically. Convert to °C for comparison with obs. ## 2. Gridded observed Tx — `data/obs/Tx_EC_0.25_2002-2023.nc` - Variable `tasmax`, **units °C** (already Celsius), `_FillValue = -9999`. - Grid identical to reforecast (41×65, 0.25°). - Time: `float days since 1961-1-1`, **calendar `365_day`** — **29 Feb removed** (`-del29feb` in provenance). 8030 days = 22 yr × 365, covering **2002-01-01 … 2023-12-31**. - Provenance (global `history`): derived from `Tx_EC_0.25_1961-2023.nc` via CDO. ## 3. Station observed Tx — `data/obs/obs_stn_daily/Tx_*.txt` (200 stations) The folder mixes four variables by prefix: `Tx_` (200, **daily max T** — our target), `Tm_` (200, daily min), `T2m_` (194, daily mean), `R_` (650, **rainfall mm** — not used). **File format (fixed-width text):** ``` line 1: STATION_NAME line 2: lon lat elevation_m region_code(1-7) then per year: ... # 31 rows × 12 monthly columns, °C ``` - Missing value = **`-99.0`** (also used to pad non-existent days, e.g. 31 Feb). - Record length varies by station (e.g. `Tx_ALUOI` 1974–2025; period & completeness vary). - Region code examples: SAPA=2, HANOI=3, ALUOI=4, DANANG=5, PLEIKU=6, CAMAU=7 — consistent with the gridded `reg` codes. ## 4. Mask — `data/Mask_VN_S2S_EC_0.25.nc` - Grid 41×65 (0.25°, matches all above). `vn` = land mask (value 1 over land; 541 land cells). `reg` = integer **region id 1–7** per land cell. Used for land-masking and zonal aggregation; the gridded route avoids shapefile rasterisation for the numeric products. ## 5. Shapefile — `data/VN_SubRegs1/` Multiple shapefiles (`7,NB_region.shp`, `B1.shp`, `B2.shp`, `6_TN_region.*` — note the latter `.shp` is 0 bytes). DBFs carry only an `ID` field (no region names). Region **names** will be derived from the gridded `reg` extents (centroid lat/lon per code) and cross-checked against station region codes, not read from the DBF. Shapefile used for the map outline. ## 6. Environment Intended env: **conda `ehd`** (`/home/tanpv/.conda/envs/ehd`, Python 3.11). Present: numpy, xarray, netCDF4, pandas, scipy, scikit-learn, xgboost, lightgbm, torch (CPU), cartopy, shapely, pyproj, matplotlib. **Missing (to add or work around):** geopandas, rasterio, properscoring/xskillscore (CRPS computed directly), statsmodels, dask, regionmask. CLI: `ncdump`, `ncks`, `cdo` available. --- ## Open reconciliation items carried forward (tracked in MEMORY.md) - **R-01 Single member** → calibration = distributional single-member MOS. P(EHD) comes from a fitted predictive distribution, never a raw ensemble fraction. *Surfaced to user.* - **R-02 Grid already 0.25°** → downscaling stage omitted; upstream interpolation documented as a limitation. *Surfaced to user.* - **R-03 Obs calendar 365_day vs forecast gregorian** → align on (month, day); drop 29 Feb from forecast-derived daily Tx before matching to obs climatology. - **R-04 Gridded vs station** disagreement to be quantified per region before choosing the calibration target (next stage).