# Running the Seasonal Drought Forecast Jobs

All commands below run from `/home/tanpv/Progs/CLI`.

## 1. Build Stage 1 climatology

Interactive run:

```bash
cd /home/tanpv/Progs/CLI
./scripts/build_stage1_climatology.sh
```

Slurm run:

```bash
cd /home/tanpv/Progs/CLI
sbatch scripts/submit_stage1_climatology.sbatch
```

Outputs:

- `Dr_SFC_GD1/outputs/Clim/Obs_Monthly_Clim.nc`
- `Dr_SFC_GD1/outputs/Clim/Mod_Monthly_Clim.nc`

## 2. Generate Stage 1 parallel task manifest

```bash
cd /home/tanpv/Progs/CLI
./scripts/generate_manifests.sh
```

This writes `manifests/stage1_bc_tasks.jsonl` with 2,880 task records. The split is:

- 2 variables: `pr`, `t2`
- 6 lead times
- 2 RCMs: `RegCM`, `clWRF`
- 5 AI models: `ridge`, `elasticnet`, `rf`, `gbm`, `xgboost`
- 3 feature sets per variable
- 8 spatial units: `national`, `R1`..`R7`

The Slurm array uses `%120`, so at most 120 tasks run concurrently, matching 6 nodes x 20 cores/node.

## 3. Submit Stage 1 parallel BC array

```bash
cd /home/tanpv/Progs/CLI
./scripts/submit_stage1_bc_chunks.sh
```

This cluster has `MaxArraySize=1001`, so the wrapper submits three chunks of 960 tasks. Each chunk uses `%120`, keeping the maximum concurrency at 120 tasks.

Logs are written to:

- `logs/stage1_bc_<jobid>_<arrayid>.out`
- `logs/stage1_bc_<jobid>_<arrayid>.err`

Each array task now fits one AI member and writes:

- `<variable>_bc.nc`
- `model.joblib`
- `metrics.json`

## 4. Submit Stage 1 ensemble array

Run this after the Stage 1 BC array has completed successfully:

```bash
cd /home/tanpv/Progs/CLI
sbatch scripts/submit_stage1_ensemble_array.sbatch
```

This builds 12 ensemble products: 2 variables x 6 lead times. For each variable and lead it combines 60 members: national outputs plus regional mosaics across 2 RCMs, 5 AI models, and 3 feature sets. Outputs are written under:

- `Dr_SFC_GD1/outputs/BC_Ensemble/pr/lead<LT>/pr_bc_ens.nc`
- `Dr_SFC_GD1/outputs/BC_Ensemble/t2/lead<LT>/t2_bc_ens.nc`

## 4b. Build Stage 1 SPEI verification maps

After Stage 1 SPEI files exist, generate gridded drought/wet event verification metrics for `spei_ens` against `spei_obs`:

```bash
cd /home/tanpv/Progs/CLI
sbatch scripts/submit_stage1_spei_verification.sbatch
```

Outputs are written per lead:

- `Dr_SFC_GD1/outputs/SPEI/lead<LT>/spei_event_verification_maps.nc`

The file contains gridded `POD`, `POFD`, `FAR`, `FBI`, `CSI`, and `PC` maps plus contingency counts for `D3`, `D2`, `D1`, `D0`, `W1`, `W2`, and `W3` across train, validation, and all periods.

## 5. Submit Stage 2 operational lead array

Example for initial date 2026-05-01:

```bash
cd /home/tanpv/Progs/CLI
YEAR=2026 MONTH=05 DAY=01 sbatch scripts/submit_stage2_operational_array.sbatch
```

The Stage 2 array has 6 tasks, one per lead time. Each lead task applies the saved Stage 1 `model.joblib` artifacts to the operational `*_current.nc` and `*_current_pre.nc` inputs, then writes `pr` and `t2` ensemble products under:

- `Dr_SFC_GD2/outputs/Operational/YYYYMMDD/lead<LT>/pr_bc_operational.nc`
- `Dr_SFC_GD2/outputs/Operational/YYYYMMDD/lead<LT>/t2_bc_operational.nc`

After all 6 lead tasks finish, build the operational SPEI drought/wet probability maps:

```bash
cd /home/tanpv/Progs/CLI
YEAR=2026 MONTH=05 DAY=01 sbatch scripts/submit_stage2_spei_probability.sbatch
```

This writes:

- `Dr_SFC_GD2/outputs/Operational/YYYYMMDD/spei_probability_operational.nc`

The probability categories use the DOCX thresholds exactly: `W1: SPEI >= 0.5`, `W2: SPEI >= 1.0`, `W3: SPEI >= 1.5`, `D0: -0.5 < SPEI < 0.5`, `D1: SPEI <= -0.5`, `D2: SPEI <= -1.0`, `D3: SPEI <= -1.5`.