# Dr_SFC operational automation

The scheduler checks at 15 minutes past every hour. A cycle becomes eligible
only three full days after its initialization date. Both raw RegCM files must
exist before remapping or forecast submission. Missing/late input is logged and
retried on the following hourly check.

Workflow:

1. `run_operational_auto.sh` scans eligible cycles in the current and previous
   month, skips outputs containing `SUCCESS`, and avoids duplicate queued jobs.
2. `remap_one_cycle.sh YYYY MM DD` remaps both RCM files atomically to the
   0.25-degree grid with CDO.
3. `submit_operational_cycle.sbatch` runs `operational_forecast.py`, producing
   `Ope_Pr_T2_and_Ano.nc`, `Dr_Prob.nc`, figures, `index.html`, detailed logs,
   and `SUCCESS` under `outputs/Ope/YYYY/MM/DD`.

The installed crontab entry is bounded by `# BEGIN Dr_SFC operational` and
`# END Dr_SFC operational`. Cron output is appended to
`logs/operational_cron.log`; SLURM stdout/stderr use
`logs/operational_auto_JOBID.{out,err}`. A failed SLURM job is resubmitted on a
later hourly check because it has no `SUCCESS` marker and is no longer queued.

Manual commands:

```bash
./run_operational_auto.sh
./remap_one_cycle.sh 2026 08 01
sbatch submit_operational_cycle.sbatch 2026 08 1
tail -f logs/operational_cron.log
```

Backfill audit (2021-01 through 2026-07): 395 cycles completed successfully.
The seven unavailable cycles are 2024-03-21, 2024-06-16, 2024-06-21,
2025-10-16, 2026-05-26, 2026-06-11, and 2026-06-21. The first five have
incomplete clWRF inputs; the last two have no source cycle directory.