CFSv2 STORAGE AND CRONTAB WORKFLOW REPORT ========================================== Report date: 2026-09-04 (Asia/Ho_Chi_Minh) Host scope: the current HPC only Method: read-only inspection. No existing file or directory was changed or deleted. Explicit exclusions: /mnt/data and /home/tanpv/Data/QG.25.196 are S2S data and are not part of this CFSv2 hindcast inventory. 1. EXECUTIVE SUMMARY -------------------- The two ope_download.sh jobs download CFSv2 GRIB2 files named flxf*.grb2 (surface/flux fields) and pgbf*.grb2 (pressure-level fields). The first job stores them in /home/nampq/data/CFS_Ope and starts both the CLWRF and RegCM forecast chains. The third cron job stores a shorter forecast subset inside its own project tree and does not start a downstream model in the inspected script. The second cron job is not a CFS downloader. It converts existing CLWRF and RegCM outputs into standardized monthly and daily NetCDF products for NCCB_562. The largest confirmed CFSv2 hindcast/reforecast collection is: /work/projects/G09/QG.25.196/CFS_Rfcs/4RCMs/ It contains separate cfs_flux and cfs_pgb trees. The full /work scan found 2,907,731 matching flxf/pgbf GRIB2 files in the parent project group /work/projects/G09/QG.25.196, with filename timestamps spanning 1999-01-01 00 through 2011-03-31 00. This is the main historical CFSv2 collection found. Other important historical copies/download trees are listed in Section 3. 2. CRONTAB JOBS AND COMPLETE DEPENDENCY CHAINS ---------------------------------------------- All three schedules use the server's local time unless cron has a separate TZ setting. The shell scripts themselves use UTC only where explicitly noted. 2.1 Operational Seasonal_FC downloader (daily at 10:10) Cron command: cd /work/users/nampq/working/Proj_T.Tan/Operational/Seasonal_FC/CLWRF/CFS ./ope_download.sh >& cfs_log Entry script: /work/users/nampq/working/Proj_T.Tan/Operational/Seasonal_FC/CLWRF/CFS/ope_download.sh Behavior: * Removes *.log files in its working directory. * Uses the NCEP NOMADS operational CFS endpoint. * With no argument, sets the reference time to current UTC date at 00 UTC. * Examines the preceding three days, but downloads only cycles whose day of month is 01, 06, 11, 16, 21, or 26. * Reads the remote 6hrly_grib_01 directory listing and selects only flxf*.grb2 and pgbf*.grb2. * Stores member 01 data under: /home/nampq/data/CFS_Ope/00/YYYYMMDD/ * Uses wget -c, so an existing partial file can be resumed/overwritten in place. * Calls ./autorun.sh after checking the three candidate dates. Verified current storage (2026-09-04 inspection): /home/nampq/data/CFS_Ope 64,496 matching files: 32,248 flxf and 32,248 pgbf. The directory contains 42 date folders: one empty-looking 20060501 folder and 41 folders in 2026 (oldest 20260206, newest 20260901). CLWRF dependency chain: ope_download.sh -> autorun.sh -> source /work/users/nampq/working/Proj_T.Tan/Operational/configure_wrf -> run_wps.sh START END -> sbatch run_wrf.sh START END -> real.exe -> wrf.exe -> run_post-process.sh -> clean -> convert_nc/0_run_ARWpost.sh -> ARWpost.exe and CDO import_binary -> convert_nc/1_selvar_packed.sh -> CDO/NCO variable extraction, reordering, and NetCDF4 compression -> convert_nc/2_selvar_xtrm.sh (when xtrm.nc is absent) -> daily-extreme extraction and time-axis construction Details: * configure_wrf sets DATA_DIR=/home/nampq/data, WPS_DIR, WRF_DIR, TOPO_DIR, compiler/MPI libraries, and executable search paths. * autorun.sh examines /home/nampq/data/CFS_Ope/00/YYYYMMDD. It derives START and END from the first and last flxf files and skips dates already present in wdir, wdir2, done, or output. * run_wps.sh uses only member 01, requires a matching flxf and pgbf file every six hours, ungribs flux and pressure files separately with Vtable.CFSR, and runs metgrid.exe. It modifies only its generated work directory and namelist copies. * run_wrf.sh is a two-node Slurm job (20 tasks/node, 48-hour limit). It generates a date-specific namelist, runs real.exe with 10 ranks and wrf.exe with all allocated ranks, moves wrfout/wrfxtrm files, and starts post-processing. * clean moves successfully completed WRF runs from wdir to wdir2 after detecting "SUCCESS COMPLETE WRF". It removes intermediate input/work files. This is a destructive part of the production script, but it was only read, never run, during this investigation. * Post-processing creates compressed per-variable NetCDF files and moves final files to CLWRF/CFS/done/START_DATE. RegCM dependency chain started by the same autorun.sh: autorun.sh -> cd .../Operational/Seasonal_FC/RegCM/CFS -> grb2nc_CFS.sh -> wgrib2 extraction to Dataset/CFS/START/01/{SST,PLEV} -> autorun.sh -> run_icbc.sh -> sstCLM45 and icbcCLM45 -> sbatch run_regcm.sh -> run_icbc.sh (called a second time) -> srun regcmMPICLM45 -> run_post-process.sh -> clean -> 1_extract_ATM.sh -> 2_extract_SRF.sh -> 3_extract_STS.sh -> 4_extract_RAD.sh -> final 3hourly/6hourly/daily products in done/START_DATE Details: * configure_regcm sets DATA_DIR=/home/nampq/data and the RegCM executable path. * grb2nc_CFS.sh extracts surface temperature from flxf and pressure-level TMP, RH, UGRD, VGRD, and HGT from pgbf into appendable NetCDF files. * RegCM autorun requires the converted dataset size to exceed 4,083,948 KiB and skips dates already represented by Dataset/CFS, wdir, wdir2, or done states. * run_icbc.sh builds date-specific input and invokes RegCM SST and IC/BC tools. * run_regcm.sh calls run_icbc.sh again before the model. This duplicates the call already made by autorun.sh and may be intentional regeneration or redundant work; the scripts contain no comment explaining it. * RegCM post-processing extracts ATM, SRF, STS, and RAD streams and then deletes the processed wdir2 run tree. Again, this script was inspected but not run. 2.2 NCCB_562 monthly/daily product job (daily at 10:10) Cron command: cd /work/users/nampq/working/Proj_T.Tan/NCCB_562/forecast ./get_data_monthly.sh >& log Entry script: /work/users/nampq/working/Proj_T.Tan/NCCB_562/forecast/get_data_monthly.sh Inputs: /work/users/nampq/working/Proj_T.Tan/Operational/Seasonal_FC/data/current/CLWRF /work/users/nampq/working/Proj_T.Tan/Operational/Seasonal_FC/data/current/RegCM Behavior: * Processes start dates from 2021 onward. * Maps day 01/06/11/16/21/26 to ensemble labels 01/02/03/04/05/06. These labels represent six initialization dates, not six native CFS members. * Produces monthly/YYYY/MM/clWRF_NN.nc and RegCM_NN.nc. * For CLWRF it derives precipitation from cumulative rainc+rainnc, calculates monthly totals/means, converts temperatures from K to degC, remaps selected fields to hgt_domain_20km.nc, and extracts u850/u200, h850/h500, and w700. * For RegCM it processes equivalent daily/3-hourly/6-hourly sources, converts precipitation to daily units and temperature to degC, and derives q2 from rh2, tas, and slp. * Merges variables, sets a 1900-01-01 monthly reference time, and removes temporary NetCDF files in the current working directory. * Calls check_mem.sh and then get_data_daily.sh. Dependency chain: get_data_monthly.sh -> check_mem.sh -> warns when a daily model/month has 1-5 files instead of expected 6 -> get_data_daily.sh -> creates daily/YYYY/MM/{clWRF,RegCM}_NN.nc -> outputs pr, tas, tasmin, tasmax, uas, and vas -> sets a 1900-01-01 daily reference time Important ordering detail: check_mem.sh runs before get_data_daily.sh, so its warnings describe the daily state before the current invocation creates missing daily products. 2.3 QG.25.196 S2S CFS downloader (daily at 09:00) Cron command: cd /work/users/nampq/working/Proj_T.Tan/QG.25.196/S2S_FC/data/CFS ./ope_download.sh >& cfs_log Entry script: /work/users/nampq/working/Proj_T.Tan/QG.25.196/S2S_FC/data/CFS/ope_download.sh Behavior: * Removes *.log files in its working directory. * Uses current UTC date at 00 UTC unless an argument is supplied. * Examines each of the preceding three days (no 01/06/11/16/21/26 filter). * Downloads flxf*.grb2 and pgbf*.grb2 from NOMADS member 01. * Rejects files later than the first day of the month three months after the initialization, so this is a shorter S2S subset than the Seasonal_FC download. * Uses wget -nc, so existing destination files are not overwritten. * Stores data relative to the script directory: /work/users/nampq/working/Proj_T.Tan/QG.25.196/S2S_FC/data/CFS/00/YYYYMMDD/ * No downstream script is called by this ope_download.sh. Verified current storage (2026-09-04 inspection): 154,760 matching files: 77,547 flxf and 77,213 pgbf. Filename initialization dates begin in December 2025 and extend through the inspection date; the unequal flxf/pgbf count indicates missing/incomplete pairs somewhere in this tree. 3. CFSv2 HINDCAST/REFORECAST STORAGE FOUND ON THE HPC ----------------------------------------------------- Search definition: regular files whose basenames match flxf*.grb2 or pgbf*.grb2. These are the same two file families used by the cron downloaders. Symbolic links were excluded by the regular-file test, reducing duplicate WPS input links. 3.1 Complete /work scan (Lustre) The full Lustre scan found 3,655,299 matching regular files, grouped below at a high directory level. "Timestamp range" is the first ten-digit timestamp after the flxf/pgbf prefix; because two filename conventions coexist, it should be used as a discovery range, not always interpreted as initialization time. Group Total flxf pgbf Timestamp range /work/projects/G09/QG.25.196 2,907,731 1,865,710 1,042,021 1999010100-2011033100 /work/users/hiepn/nampq 286,276 143,450 142,826 2021111600-2023050100 /work/users/nampq/working 219,181 109,719 109,462 2025120900-2027040100 /work/users/thanhnx/202605-SubSeason 190,248 95,671 94,577 2000010100-2001040100 /work/users/thuctd/Operational 51,730 26,484 25,246 1999010500-2027020100 /work/users/thuctd/check_var_clwrf 132 66 66 1999010500-1999011506 /work/users/longtt/Programe 1 1 0 2011021100 The primary historical roots identified within those groups are: * /work/projects/G09/QG.25.196/CFS_Rfcs/4RCMs/cfs_flux/YYYY/YYYYMM/YYYYMMDD Surface/flux reforecasts. A typical 1999 daily initialization directory has about 185 six-hourly files (roughly 46 days in the inspected examples). * /work/projects/G09/QG.25.196/CFS_Rfcs/4RCMs/cfs_pgb/YYYY/YYYYMM/YYYYMMDD Pressure-level counterparts to the cfs_flux tree. The high-level count is smaller than the flux count, so completeness should not be assumed. * /work/projects/G09/Exchange/4Tuan/data-ori /work/projects/G09/Exchange/4Tuan/data-2-wrf Working/original copies used for four-week experiments; inspected examples use 1999 flxf filenames. These may duplicate subsets of CFS_Rfcs. * /work/users/thanhnx/202605-SubSeason/CFSdownloader/downloads A newer downloader tree containing both flxf and pgbf hindcasts. The scan found 190,248 files in its high-level group, with examples organized as downloads/YYYY/YYYYMM/YYYYMMDD. The current visible range is around 2000-2001. * /work/users/hiepn/nampq/working/Proj_T.Duc/TC-KC09/Operational/CLWRF_v3.9.1/CFS Historical CFS files copied/linked into old CLWRF run input trees. These are downstream working copies rather than the canonical hindcast archive. * /work/users/thuctd/Operational/Seasonal_FC/data/DATA_TEST/{original,cropped} /work/users/thuctd/check_var_clwrf/{4RCMs_cropped,_test_*} Small 1999 test subsets, including original and cropped variants. 3.2 Backup/archive tree /work/users/hiepn/nampq/working/data/CFS_bk This tree is important even though most data are no longer loose *.grb2 files: * ref_download.sh and ref2_download.sh explicitly download the NCEI CFS "reforecast" 9-month-run endpoints for flxf and pgbf files. * arc_download.sh downloads the NCEI "operational-9-month-forecast" archive. * The current tree contains six .tar.xz archives under done/compression/com, totaling 2,830,513,352 compressed bytes: 2021020100.tar.xz 2021082600.tar.xz 2022011600.tar.xz 2022120100.tar.xz 2022121600.tar.xz 2022122100.tar.xz * These compressed archives do not appear in the loose-file counts above. 3.3 Operational/current stores (not hindcast) * /home/nampq/data/CFS_Ope Canonical input of the first cron chain; 64,496 matching files. * /work/users/nampq/working/Proj_T.Tan/QG.25.196/S2S_FC/data/CFS Canonical input of the third cron chain; 154,760 matching files. * Other /work groups contain multiple Operational/Seasonal_FC/CFS_Ope copies, including user thuctd. These were counted by the scan but are not hindcast. 4. SEARCH COVERAGE, EXCLUSIONS, AND LIMITATIONS ----------------------------------------------- * /work (123-TB Lustre mount): completely traversed using lfs find for both flxf*.grb2 and pgbf*.grb2. Counts in Section 3.1 come from this completed scan. * /home/nampq/data/CFS_Ope: completely counted because it is the direct output of the first cron downloader. Other relevant /home locations may exist, but no completed unrestricted /home count is claimed here. * /home/tanpv/Data/QG.25.196: explicitly excluded because it contains S2S data. Nothing under this path contributes to any count in this report. * /mnt/data: explicitly excluded because it contains S2S data. Nothing under this mount contributes to any count in this report. * A script under /work/users/nampq/sv_k66 references /mnt/WEB_DAT/tmp/data/CFS_Ope, but that path is absent on this host at inspection time. It is distinct from the excluded /mnt/data mount. * The system locate database did not index the mounted datasets, so it could not be used to discover additional relevant storage outside the completed /work scan and directly inspected paths. * Permission-denied diagnostics were suppressed during global discovery. Thus, inaccessible directories could contain additional data. * Compressed tar/xz archives are not discoverable by basename scans unless their archive filenames themselves contain flxf/pgbf. The known CFS_bk archives were identified separately through directory and script inspection. * The report inventories storage names and workflows. It does not validate GRIB contents, file integrity, paired flxf/pgbf coverage, or scientific correctness. 5. NOTABLE RISKS / ITEMS TO REVIEW ---------------------------------- 1. Both cron downloaders begin with rm -fv *.log. Historical remote listings are discarded on every run, and both cron commands overwrite their main log file. 2. The first downloader checks only the preceding three days while accepted cycle days are five days apart. A network outage longer than that can permanently miss a cycle unless a manual downloader is used. 3. Neither downloader enables strict shell error handling. They end with exit 0, so download or downstream failures may still produce a successful exit status. 4. curl uses -k and wget disables certificate verification in the first workflow. 5. File completeness is inferred from presence, not checked against size/checksum. Interrupted files resumed by wget -c may remain unnoticed if later logic only tests -f. 6. The QG store currently has 334 more flxf than pgbf files, indicating incomplete pairing or an in-progress download. 7. RegCM IC/BC generation is invoked once in autorun.sh and again in run_regcm.sh. 8. NCCB check_mem.sh runs before get_data_daily.sh, so warnings can be stale by the end of the same cron invocation. 9. NCCB scripts create and broadly remove *.nc and *tmp* in their current working directory. Concurrent runs would be unsafe and could interfere with each other. 10. The CLWRF and RegCM post-processing clean scripts remove substantial temporary trees after success; operational execution should not overlap another run for the same initialization. 6. CONCISE DATA FLOW -------------------- NCEP NOMADS -> flxf/pgbf GRIB2 -> /home/nampq/data/CFS_Ope -> WPS -> real.exe -> CLWRF -> compressed variable NetCDF -> wgrib2 -> RegCM SST/PLEV NetCDF -> IC/BC -> RegCM -> stream NetCDF -> Operational/Seasonal_FC/data/current/{CLWRF,RegCM} -> NCCB monthly and daily ensemble-labelled NetCDF products NCEP NOMADS -> flxf/pgbf GRIB2 -> QG.25.196/S2S_FC/data/CFS (three-month subset; no downstream call in the inspected downloader) NCEI reforecast archive -> historical flxf/pgbf GRIB2 -> primarily /work/projects/G09/QG.25.196/CFS_Rfcs/4RCMs -> experimental/cropped/WRF-input copies in other /work project trees 7. RCM TWO-STAGE OUTPUT RETENTION CHECK --------------------------------------- Follow-up inspection confirmed that absence of raw integration output does not mean that a model was never run: * CLWRF GD1 completion is recorded by "SUCCESS COMPLETE WRF" and by wrfout files under wdir2/S_DATE/output after clean moves the successful run. The active post-processing script extracts selected NetCDF variables into done/S_DATE. Its command to remove wdir2 is currently commented, so recent raw CLWRF output is retained. An older _old/run_rclone.sh uploads NetCDF output and removes the transferred local files, but no active caller was found in the inspected tree. * RegCM GD1 completion is recorded by "RegCM V4 simulation successfully reached end". Its clean script moves wdir/S_DATE to wdir2, then deletes the converted Dataset/CFS/S_DATE and other S_DATE intermediates. After extracting 3hourly, 6hourly, and daily NetCDF products, the active run_post-process.sh explicitly removes wdir2/S_DATE. Thus raw RegCM integration output is intentionally absent after successful GD2. * The legacy run_check.sh scripts accept final products from either done/S_DATE or Seasonal_FC/data/current/{CLWRF,RegCM}/S_DATE. At follow-up inspection there were 1,095 CLWRF and 1,093 RegCM date directories in data/current, versus 34 CLWRF and zero RegCM date directories in wdir2. This strongly matches the cleanup behavior above. * Old rclone scripts provide an additional historical deletion path: the RegCM version synchronizes done/S_DATE to nam_gd and then removes both done/S_DATE and wdir2/S_DATE; the CLWRF version removes uploaded NetCDF files. These scripts are under _old and no matching active process was present. Reading nampq's crontab was denied for lack of privilege, so an external cron caller cannot be ruled out solely from this inspection. For status checks, GD2 NetCDF output in done or data/current is authoritative and also implies that GD1 completed, even when large raw model output was deleted. 8. COMPARISON OF THE S2S AND SEASONAL CFSv2 STORES -------------------------------------------------- Snapshot date: 2026-09-04. These are active operational stores, so counts can change after this inspection. Compared paths: S2S: /work/users/nampq/working/Proj_T.Tan/QG.25.196/S2S_FC/data/CFS/00 Seasonal/RCM input: /home/nampq/data/CFS_Ope/00 Both stores contain member-01 CFSv2 GRIB2 data at six-hour intervals and use the same filename convention. The two file families are flxf (surface/flux fields) and pgbf (pressure-level fields). There were 24,694 filenames common to both stores. A deterministic byte-for-byte sample of 13 common, non-empty, same-size files, distributed across dates and both file families, found all 13 identical. This supports a common data source, but is not a checksum of every common file. Inventory summary: Store S2S Seasonal CFS_Ope Disk usage approximately 1.9 TB approximately 786 GB Initialization directories 262 42 Total flxf/pgbf files 155,238 64,496 flxf files 77,668 32,248 pgbf files 77,570 32,248 Initialization range 2025-12-09--2026-09-03 2026-02-06--2026-09-01 Common initialization dirs 41 41 CFS_Ope also has a separate 20060501 directory, but it contributed no matching file to the observed initialization range. The S2S store had 221 initialization directories not present in CFS_Ope; the only CFS_Ope directory absent from S2S was 20060501. The principal design difference is temporal coverage. S2S downloads nearly every initialization day and truncates forecasts at approximately three months. CFS_Ope downloads only initialization days 01, 06, 11, 16, 21, and 26, but keeps a much longer seasonal forecast. Examples: Initialization S2S last valid time CFS_Ope last valid time 2026-02-06 2026-05-01 00 2026-09-01 00 2026-06-01 2026-09-01 00 2027-01-01 00 2026-08-21 2026-11-01 00 2027-03-01 00 2026-09-01 2026-12-01 00 2027-04-01 00 Completeness differs materially: * CFS_Ope had 32,248 paired flxf/pgbf keys, no unpaired files, and no zero-byte flxf or pgbf files. * S2S had 77,570 paired keys, 98 flxf files with no matching pgbf, no pgbf files without flxf, 16 zero-byte flxf files, and 200 zero-byte pgbf files. In total, 216 paired keys contained at least one zero-byte component. * At least 17 comparisons among common filenames produced a size mismatch. The observed cases included zero-byte S2S files and S2S pgbf files exactly 20 MiB in size whose CFS_Ope counterparts were larger, consistent with interrupted or incomplete downloads. Therefore, the stores are not complete mirrors. Their common portion generally represents the same CFSv2/member-01 source, while their selection policies differ: S2S has dense daily initializations with a short forecast horizon; CFS_Ope has sparser five-day initialization cycles with a longer seasonal horizon. At this snapshot, CFS_Ope is internally paired and free of empty files, whereas the S2S store requires repair or re-download of missing, empty, and apparently truncated files before completeness-sensitive processing.