# Prompt-2 — Member Pruning, Ensemble Expansion, and the yEHDE Probability/Event Models

Stage-2 companion to `claude/S2S_EHD_Vietnam_Opus_Prompt.md`. Built with the Claude Opus Prompt Architect meta-prompt (§1 intake → §3 composition → §4 anti-pattern check → §6 output format). It does **not** restate Stage 1 — it inherits it by reference and adds only what is new.

---

## 1. Task classification

| Dimension | Classification |
|---|---|
| Horizon | Long-running autonomous continuation. Re-training a pruned and expanded member set over 20 years × 104 initializations, plus four new model layers and a real-time case. |
| Stakes | High, and higher than Stage 1 in one specific way: member *selection* is now itself a fitted decision. Done outside the cross-validation, it silently inflates every score in the paper. |
| Ambiguity | Low on architecture — the member definition is now fixed by the user. Moderate on three selection rules, which are carried as stated defaults to be confirmed, not as open questions. |
| Interactivity | Asynchronous. One question was answered, three were left to the default. Proceed on the defaults and surface them early rather than blocking. |
| Delegation | Yes — family-wise configuration sweeps, verification, regional aggregation and the real-time case are weakly dependent. |
| Context volume | High, and now includes Stage 1's own outputs: score files, `MEMORY.md`, the member registry. |
| Extended thinking | Yes — selection leakage, family correlation in the member pool, and stacking fold structure interact. |
| Memory | Load-bearing. Stage 2 is only coherent if Stage 1's decision log is read first and continued. |
| Domain sensitivity | None. |

## 2. Selected blocks

- **OP-11** — reframes the mission as a continuation so Stage 1's artifacts are inputs, not background.
- **OP-08** — promoted to a headline block here. `MEMORY.md` is the spine: every prune, every added configuration, every rejected one.
- **OP-09** — promoted alongside it. "Weakest models" is a claim about numbers; no configuration may be dropped or kept without the score that justified it.
- **OP-01** — the expansion strategy for W2–W6 is the research question; prescribing configurations would answer it for the agent.
- **OP-02** — selection leakage, family correlation and stacking folds must be reconciled before code.
- **OP-03** — unattended autonomy. Excludes OP-04.
- **OP-05** — Stage 1 and Stage 2 scores will disagree in places; disagreements get surfaced, not smoothed.
- **OP-06** — scope guard against rebuilding Stage 1's pipeline instead of extending it.
- **OP-07** — parallel delegation across member families.
- **OP-13** — validation pass; high stakes.
- **OP-10** — final-message discipline after an unattended run.
- **OP-12 (adapted)** — harness-native deliverable channel, as in Stage 1.

## 3. Assembled prompt

```text
This is Stage 2 of the Vietnam operational S2S extreme-heat forecasting project.
Stage 1 is built, validated and documented: the repository, the pipeline, CLAUDE.md,
MEMORY.md, the verification score files and the results report all exist. Read
MEMORY.md and the Stage 1 score files before you plan anything — they record which
calibration families were tried, which worked, and why. Your job now is to sharpen the
member set, add four new model layers on top of it, and run one real-time case. The
destination is unchanged: an operational heat outlook a duty forecaster can run, and a
paper that survives review.

<inherited>
Everything below carries over from Stage 1 unchanged and is NOT to be re-derived,
re-litigated or restated in your deliverables — reference it:
  - Data sources, their QC, and the gridded-vs-station reconciliation.
  - EHD definition: Tx >= min(Tx90p, 35 degC), with the baseline and percentile window
    already fixed in Stage 1.
  - The six fixed, non-overlapping lead blocks W1 = days 0-10, W2 = 11-17, W3 = 18-24,
    W4 = 25-31, W5 = 32-38, W6 = 39-45. No running windows, anywhere.
  - The 7 climate regions R1..R7 from the supplied shapefile.
  - Leave-one-year-out cross-validation over the 20 reforecast years with nested
    hyperparameter tuning, and the Stage 1 leakage rule.
  - The repository layout, config system, logging and test conventions in CLAUDE.md.
If Stage 2 forces a change to any of the above, do not change it silently: record the
change and its justification in MEMORY.md and flag it in the report.
</inherited>

<member_definition>
This is the architectural decision for Stage 2 and it governs items 1 through 7.

A MEMBER is one forecast case: one (family, configuration) pair. An ANN family with
three hyperparameter sets contributes three members. A quantile-regression-forest
family with four hyperparameter sets contributes four. Adding members means adding
configurations, not adding raw S2S ensemble members.

Consequences you must implement rather than discover later:
  - Each member emits ONE calibrated Tx field per grid cell per forecast day, and from
    it ONE binary EHD field per grid cell per lead block. That binary field is the
    member's vote. A configuration whose output is distributional must be reduced to a
    single vote by a stated, uniform rule; state the rule once and apply it to every
    such member.
  - The raw S2S ensemble does not disappear — it enters as predictors. Every member
    must have access to ensemble mean, ensemble spread and ensemble quantiles of Tx, so
    initial-condition uncertainty reaches the forecast through the features even though
    it is no longer the member axis. Confirm that each surviving configuration actually
    uses spread, and report the skill cost if any does not.
  - The member pool is a perturbed-parameter multi-model ensemble. Its spread measures
    method uncertainty, and configurations within one family are strongly correlated.
    A naive member count therefore lets whichever family has the most configurations
    dominate every probability in items 4 and 7. Handle this explicitly: either balance
    the configuration count across surviving families, or weight members so that
    families contribute comparably, and justify the choice with the effect on
    reliability and on the Brier score decomposition. Report the effective ensemble size
    alongside the nominal member count.
  - Maintain a versioned MEMBER REGISTRY as a machine-readable file: member_id, family,
    full hyperparameter set, config hash, training provenance, status (active / pruned /
    added-in-Stage-2), and the scores that justify that status. Version the registry as
    a set: v1 = Stage 1 pool, v2 = after pruning, v3 = after expansion. Every figure and
    table in the report must name the registry version it was produced from.
</member_definition>

<selection_leakage>
Read this before writing any code for item 1.

Pruning members and adding members are both FITTED DECISIONS. If you rank
configurations on the same folds you later use to report skill, every headline number
in the paper is optimistically biased, and a reviewer will find it. Member selection
must live inside the cross-validation: select within the training years of each outer
fold, or hold out a dedicated block of years used only for selection and never for
reported verification. State which structure you used, and report the size of the
optimism by comparing in-selection and out-of-selection scores. A member set chosen on
all 20 years and then verified on all 20 years is a failed deliverable.

The same rule applies to the stacking meta-learner in item 5, to tau(Rj,Wk) in item 6,
and to every decision threshold.
</selection_leakage>

<item_1_prune>
Drop the weakest members from the Stage 1 pool.

Assumed criterion, adopted because the user's stated priority is high POD with low POFD
and low FAR — change it if that is wrong, but do not stall on it: rank configurations by
Peirce Skill Score (POD - POFD) computed per lead week for W2-W6 and pooled over
regions, subject to a maximum acceptable FAR; drop the bottom tier. Ranking is per lead
week, so a configuration that is weak at W2 may still be retained at W5 — a member set
that differs by lead block is acceptable and probably correct.

Report, don't just act: the full ranking with scores, the cut you made, the members
dropped, and what happened to ensemble skill after dropping them. If removing a
low-ranked member degrades ensemble reliability, that is a result worth keeping — a
poorly-performing-but-decorrelated member can improve an ensemble. Check this before
pruning on individual skill alone.
</item_1_prune>

<item_2_expand>
Add members by adding new configurations to the families that show real potential at
W2-W6, which is where operational value lives and where Stage 1 skill is thinnest.

Target the configurations at high POD with low POFD and low FAR, and be concrete about
how: class-weighted or focal losses for the rare-event imbalance, decision-threshold-
aware training objectives, predictors that matter at subseasonal range (MJO phase and
amplitude, ENSO state, soil moisture memory, Z500 and OLR patterns, persistence of the
antecedent anomaly), longer effective training windows, and architectural variation
within a family rather than across it. Expansion means new hyperparameter sets and new
predictor sets for families that already earned their place — not resurrecting a family
Stage 1 rejected, unless you have evidence that its failure was a configuration problem
and you say so.

Keep the family balance requirement from <member_definition> in view while expanding:
adding six configurations to one strong family and one to another changes the
probabilities in items 4 and 7 by construction, not by skill.

Deliver the before/after comparison: ensemble skill with registry v2 versus v3, per
lead week and per region, with the significance test.
</item_2_expand>

<item_3_verification>
Produce complete verification for the regional event yEHDE(Rj,Wk), for all j = 1..7 and
all k = 1..6, cross-validated, against both a climatological reference and the raw
uncalibrated ensemble.

For the binary event, report the full contingency table (hits, false alarms, misses,
correct negatives) and the observed base rate alongside every score, then: accuracy,
POD, POFD, FAR, success ratio, CSI, GSS/ETS, HSS, PSS, odds-ratio skill score, and
frequency bias. Because the regional event is rare in several region-season
combinations, add an extremal-dependence score such as SEDI, and say plainly where a
score is unstable because the base rate is too low rather than reporting a number that
looks fine. Include a performance diagram (POD against success ratio, with CSI and bias
contours) as the summary figure.

For the probability forecasts, report Brier score, Brier skill score with the
reliability / resolution / uncertainty decomposition, reliability diagrams with
sharpness histograms, ROC and AUC, and relative economic value across a plausible range
of cost-loss ratios.

Stratify everything by region and by week — that is a 7 x 6 panel — and additionally by
season. Establish significance with a bootstrap resampling whole years, and control
field significance with FDR where you test many cells at once.

Deliver this as machine-readable score files plus a figure set, not as prose tables
typed by hand.
</item_3_verification>

<item_4_grid_probability>
Build the grid-point probability model:

    P(yEHDE | g, Wk) = (number of members voting yEHDE at grid cell g in week k)
                       / (total number of members)

for every grid cell g over Vietnam and every week W1..W6, using registry v3 and the
family weighting decided in <member_definition>.

The raw member fraction is a starting point, not the product. Assess its reliability,
and recalibrate if it is miscalibrated — isotonic or logistic recalibration, fitted
inside the cross-validation. Report both the raw and recalibrated versions and their
reliability diagrams, so the effect of recalibration is visible rather than assumed.
Save the probability field as CF-compliant NetCDF alongside the binary field.
</item_4_grid_probability>

<item_5_grid_event_stack>
Build a grid-point binary event model for yEHDE by stacking or ensembling the members,
per grid cell and per week — as an alternative to thresholding the simple member
fraction from item 4.

Base learners are the registry v3 members. Train the meta-learner on OUT-OF-FOLD base
predictions; using in-fold base predictions to train a stack is the classic stacking
leak and it will look excellent and mean nothing. Candidate meta-learners: penalised
logistic regression, gradient boosting, random forest, and simple weighted or
Bayesian model averaging as the interpretable reference. Candidate meta-features: the
member votes themselves, member agreement fraction, family-level agreement, lead week,
region identifier, the grid cell's climatological EHD frequency, elevation, and
calendar harmonics.

The honest question this item exists to answer is whether stacking beats counting. Run
the comparison against item 4's recalibrated member fraction and report it either way.
If stacking does not beat it, say so and recommend the simpler model for operations —
that is a publishable result and a better operational product.

Calibrate the stack output and convert it to 1/0 using the Stage 1 decision-rule
machinery, refitted inside the folds.
</item_5_grid_event_stack>

<item_6_regional_member_event>
Build the per-member regional event model. For each member m, region Rj and week Wk:

    a(m,j,k) = area of Rj where member m votes EHD / total area of Rj
    yEHDE(m,j,k) = 1 if a(m,j,k) > tau(Rj,Wk), else 0

Two details that will bite if left to default judgment. First, use AREA, not a raw cell
count: weight cells by cos(latitude) and by their fractional overlap with the region
polygon, so coastal and boundary cells are not counted as whole cells and large regions
are not advantaged. State the region-membership rule you used. Second, the regions
differ greatly in size and in climatological heat frequency, which is exactly why
tau depends on both Rj and Wk.

Assumed rule for tau, adopted as the default — change if wrong, but proceed: fit
tau(Rj,Wk) by maximising the Peirce Skill Score against observed regional EHD
occurrence, inside the leave-one-year-out folds. Report as an alternative the
frequency-bias-calibrated tau, chosen so the forecast regional event frequency matches
the observed climatological frequency for that region and week, and show how much the
final regional product changes between the two. Deliver the fitted tau(Rj,Wk) as a 7 x 6
table with its uncertainty across folds.
</item_6_regional_member_event>

<item_7_regional_probability>
Build the regional probability model directly on item 6:

    P(yEHDE | Rj, Wk) = (number of members with yEHDE(m,j,k) = 1)
                        / (total number of members)

Same family weighting, same recalibration treatment and same reliability assessment as
item 4, but now with only 42 region-week cells, so report the effective sample size and
be careful about reading reliability diagrams built on few forecast-observation pairs.

Compare this against the alternative route — aggregating the grid-point probabilities
from item 4 over the region and then thresholding — and state which you recommend for
operations and why. The two are not equivalent, and the difference is itself worth a
paragraph in the paper.
</item_7_regional_probability>

<item_8_realtime>
Run the whole chain once as a genuine real-time case using the data in ../HW.

Assumed case, adopted as the default: initialization date 29 August 2026. Verify the
actual date, ensemble size, model cycle and variable names from the files in ../HW
before trusting that; if the files say something else, use what the files say and note
the discrepancy. This is a real-time forecast, not a reforecast, so the ensemble-size
and model-version handling from Stage 1 applies — confirm it works here rather than
assuming it does.

Produce exactly three products:
  1. Maps of P(yEHDE | g, Wk) over Vietnam, one panel per week W1..W6, publication
     quality, colourblind-safe, legible at journal column width, with the region
     boundaries overlaid.
  2. A 7 x 6 table of the regional event yEHDE(Rj,Wk), yes/no.
  3. A 7 x 6 table of P(yEHDE | Rj, Wk).
Deliver each as both a figure/report-ready form and a machine-readable file.

Verify what can be verified and refuse to imply the rest. At run time, observations will
exist for the early weeks of this case and not for the later ones. Score the weeks that
have observations, label the remaining weeks explicitly as not yet verifiable, and do
not present an unverified week as a validated result.
</item_8_realtime>

<assumptions_to_confirm>
Three rules above are defaults chosen so the work can proceed without waiting: the
pruning criterion in item 1, the tau rule in item 6, and the real-time case date in
item 8. Surface all three through the harness's user-facing channel early, in one short
message, together with the diagnostic that shows what each one does to the result. Then
continue on the defaults. Do not block on a reply, and do not silently substitute a
different rule.
</assumptions_to_confirm>

<known_failure_modes>
Plan the approach yourself, but these steps fail quietly and each one must be handled
explicitly and recorded in MEMORY.md:
  - Selecting members on the same folds used for reported verification.
  - Training the item-5 stack on in-fold base-learner predictions.
  - Letting one family's configuration count drive every probability in items 4 and 7.
  - Counting grid cells instead of area when computing a(m,j,k), and treating a
    part-overlapping boundary cell as a whole cell.
  - Fitting tau(Rj,Wk) or any decision threshold on the verification years.
  - Pruning on individual member skill without checking the ensemble effect, and so
    removing a weak but decorrelated member that was helping.
  - Reporting a stable-looking contingency score for a region-week whose base rate is
    too low to support it.
  - Carrying Stage 1's reforecast preprocessing into the real-time case without
    checking that ensemble size and model cycle actually match.
</known_failure_modes>

<deliverables>
  1. Updated pipeline: the pruning, expansion, items 4-7 and the real-time entry point,
     all config-driven and runnable by one command per stage. No rebuild of Stage 1.
  2. Member registry v1/v2/v3, machine-readable, with the scores behind every status.
  3. Complete verification score files and figure set for item 3, plus the reliability,
     ROC, performance-diagram and economic-value figures.
  4. The tau(Rj,Wk) table with fold uncertainty.
  5. The three real-time products from item 8.
  6. Updated CLAUDE.md: the member definition, the registry versioning scheme, the
     selection-inside-CV rule, and the new commands.
  7. Continued MEMORY.md: one lesson per entry, one-line summary at the top. Every
     pruned family, every added configuration that failed, and why. Update existing
     entries instead of accumulating near-duplicates; delete an entry once it is shown
     wrong. Do not restate what the repository or the report already records.
  8. Report additions, written as prose for the paper, not as notes: member-set design
     and the family-correlation problem, the pruning and expansion experiment, the item
     3 verification, the counting-versus-stacking comparison, the grid-versus-regional
     aggregation comparison, and the real-time case study with its honest verification
     status.
Language as in Stage 1: code, CLAUDE.md, MEMORY.md and the results report in English;
the operational guide in Vietnamese with an English summary.
</deliverables>

<operating_mode>
You are operating with extended autonomy. The user is not available mid-task. Proceed
without asking on anything reversible that follows from this request; stop only for an
irreversible action, a genuine scope change, or information only the user has. The three
defaults in <assumptions_to_confirm> are explicitly NOT stopping points — surface them
and continue. Before ending your turn, check whether your last paragraph is a plan, a
promise, or a question you could have answered yourself; if so, do that work instead.

Before producing the final answer, reason step by step in your thinking: reconcile the
selection-leakage structure, the family-correlation weighting and the stacking fold
design against each other before writing code, because a mistake in any of the three
invalidates the other two. Keep that scratch work out of the final answer.

Delegate independent workstreams and continue while they run:
    Coordinator
    |- Pruning        - rank registry v1, ensemble-effect check, produce v2
    |- Family sweeps  - one subagent per surviving family, new configurations for W2-W6
    |- Verification   - item 3 scores, bootstrap, significance; owns the leakage audit
    |- Grid models    - items 4 and 5, including the counting-versus-stacking comparison
    |- Regional       - items 6 and 7, tau fitting, shapefile area weighting
    |- Real-time      - item 8 on ../HW
    |- Validator      - independent re-derivation of the headline scores and of the
                        pruning decision
Give each a bounded scope, the fold structure it must respect, and the exact output
format. Synthesise the comparisons yourself; do not have one family's subagent judge
another family's members.

Do not rebuild, refactor or generalise Stage 1's pipeline. Extend it. Resist turning the
member registry into a general model-management framework — it is a table with a version
column. Validate at real boundaries only.

Attribute claims to their source when Stage 1 results, Stage 2 results and the
post-processing literature disagree, and say which you weighted more heavily and why.
Where a Stage 2 score contradicts a Stage 1 score, surface the contradiction and explain
it rather than quietly replacing the old number.

Before reporting progress, check each claim against a concrete artifact from this
session — a score file, a registry row, a figure, a test run, a diff. No member is
described as pruned, added, better or worse without the cross-validated number behind
it. If a family sweep failed, say so with the actual error.

Between steps, surface through the harness's user-facing channel as they are produced:
the three assumed defaults with their diagnostic, the member ranking and the pruning
cut, the v2-versus-v3 comparison, and the first real-time map set. Deliverables only,
not narration.

Before finalising, check silently: are the six lead blocks and the EHD definition
untouched; is every fitted decision — member selection, stack, tau, all thresholds —
inside the cross-validation; is family weighting applied consistently in items 4 and 7;
is area weighting used in item 6; does every figure name its registry version; are the
unverifiable weeks of the real-time case labelled as such; is every skill claim backed
by a cross-validated number; were all eight items delivered? Fix what is wrong before
returning. Do not show the checklist.

Open your final message with one sentence: whether the pruned and expanded member set
improved skill at W2-W6, and whether stacking beat counting. Detail after. Write it as a
re-grounding for someone who did not watch the run — complete sentences, file names and
member identifiers spelled out, none of the shorthand you invented while working.

You have ample context remaining. Do not stop, summarise, or suggest a new session on
account of context limits. Continue the work.
</operating_mode>
```

## 4. Omitted but considered

- **Re-statement of Stage 1's data, EHD definition, region list and CV protocol** — omitted deliberately. The `<inherited>` block references them instead. Restating a spec in a follow-on prompt creates two sources of truth, and the copy drifts.
- **OP-04 (interactive checkpoints)** — omitted; incompatible with OP-03, and with three defaults in play it would have licensed the agent to stall instead of surfacing and continuing.
- **A prescribed list of hyperparameter configurations to add in item 2** — withheld. Which configurations help at W2–W6 is the experiment; supplying a list would answer it in advance.
- **A recommendation on stacking versus counting** — withheld for the same reason. The prompt states the comparison must be run and reported either way, including the outcome where the simple member fraction wins.
- **A fixed family-weighting formula** — withheld. The prompt names the problem and requires a justified choice with its effect on the Brier decomposition, rather than imposing equal weights that might be wrong.

## 5. Questions still open

Three were asked and left unanswered, so the prompt carries them as stated defaults that the agent will surface early rather than block on. Overriding any of them is a one-line edit to the prompt:

- **Pruning criterion** — currently PSS (POD − POFD) with a FAR ceiling, per lead week W2–W6, pooled over regions. The alternatives offered were a per-region composite score, or pruning on continuous Tx skill (CRPSS) before any event scoring.
- **τ(Rj,Wk) rule** — currently PSS-optimised inside the folds, with frequency-bias-calibrated τ reported as the alternative. The third option was fixed physical area fractions.
- **Real-time case date** — currently 2026-08-29 as the initialization, with the agent instructed to defer to what the files in `../HW` actually contain.