Estimating GB history

This chapter is the GDA-side pipeline that turns settlement telemetry into a lake of per-period WLS-estimated GB grid states. Everything runs from the GDA repo root:

cd D:\Work\GDA\v1

The pipeline has two independent reconstruction routes per day - a demand-scaled solve (the presolved lake) and a measurement-fitted estimate (the estimated lake) - plus two comparison products built from them (divergence and closure). A scheduled task keeps it all following reality and walking backward through history; you only run the steps by hand for a specific day or a supervised deep backfill.

Every step is horizon-guarded: a settlement day newer than the acquisition horizon is rejected. Estimable history reaches back to 2016-03-01 (the Physical Notification floor).

Step 1 - per-GSP measurements for a day

python NetworkModel/build_gsp_measurements.py --date 2024-03-01
python NetworkModel/build_gsp_measurements.py --date 2017-06-15 --source pn

Flag	Default	Effect
`--date YYYY-MM-DD`	required	the settlement day (must be historical)
`--source auto\\|b1610\\|pn`	`auto`	`auto` uses B1610 settlement metering when the day has a full period set (>=40 periods), else falls back to PN
`--min-confidence F`	`0.6`	crosswalk confidence floor for allocating a unit to its GSP

One read of the day's B1610 (or PN) parquet plus the BMU->GSP crosswalk produces three outputs under DataSources/Derived/, each in year=YYYY/week=WW/ partitions:

gsp_generation_measurements/.../gsp_generation_measurements_YYYYMMDD.parquet
- metered generation per (settlement period x GSP), with an explicit sigma_mw per aggregate. Units the crosswalk could not bind to a GSP are reported on a per-period NULL-GSP row, never smeared across the network.
gsp_group_demand/... - per-GSP-group net demand from the supplier settlement legs (B1610 days only).
gsp_ic_flows/... - interconnector flows at their landing GSPs (import-positive MW).

The PN fallback extends history three years past B1610: when a day has no (or a partial) B1610 file, the same aggregation runs on Physical Notification levels - submitted intent, not metering - so its rows carry provenance="pn" and looser sigmas (10 MW / 10% vs B1610's 2 MW / 2%), and no group-demand sibling is emitted. PN depth is 2016-03 onward.

Step 2 - WLS estimates for every period

python NetworkModel/build_estimated_state.py --date 2024-03-01
python NetworkModel/build_estimated_state.py --date 2024-03-01 --case-dir NetworkModel/out/gb-full

Flag	Default	Effect
`--date YYYY-MM-DD`	required	the settlement day (step 1 must have run)
`--case-dir`	`NetworkModel/out/gb-spine`	the target network
`--periods 1,2,...`	every period in the day	restrict to specific settlement periods

For each settlement period (46/48/50 depending on DST), the builder composes the real measurement set (build_estimator_measurements.py: B1610 per-GSP generation + interconnector flows + demand/voltage pseudos with honest sigmas), then invokes the GridSim CLI:

gridsim estimate <case-dir> --measurements <sp>.json --prior solve --warm-start
        --export-solve --max-iter 100 --lean --backfill-removals

The CLI path defaults to the Release build of GridSim.Cli.exe (Debug fallback); override with the GRIDSIM_CLI environment variable.

Output: DataSources/Derived/estimated_state/<case>/year=YYYY/week=WW/estimated_state.parquet (weekly merge-on-write, so re-runs are idempotent) plus a once-per-case topology.json. Each row is one settlement period carrying the fit's honesty layer - status, converged, objective_j, chi_threshold, chi_passed, dof, removed_n/removed_json - the slim result_json state block the bridge streams, and - when the case carries a boundaries.json - a boundary_json block per converged row (the ETYS boundary transfers evaluated at the estimated state). Non-converged periods are stored honestly (converged=False, no result block), never skipped; the script exits 3 when any period failed to converge, which the wrappers treat as "stored honestly", not as an error.

Step 2a - the solve-route sibling

The presolved lake is the deterministic counterpart the divergence product compares against. Build the same day on the solve route:

python NetworkModel/presolve_lake.py --case gb-spine --start 2024-03-01T00:00:00Z --days 1

This re-solves each 30-minute period at a demand scale tracking the reconstructed national demand shape and writes Derived/presolved_state/<case>/year=/week=/presolved_state.parquet (plus its own topology.json). Do not run a bulk pre-solve alongside the live ingest pipeline - see the GDA RUNBOOK.md deferred-procedures section.

Steps 3 and 4 - divergence and closure

python NetworkModel/build_state_divergence.py --case gb-spine --date 2024-03-01
python NetworkModel/report_estimation_closure.py --case gb-spine --date 2024-03-01

Divergence joins the two routes per period and writes max/mean |deltaVm|, max |deltaVa|, max |deltaflow| with the worst bus/branch named, to Derived/state_divergence/<case>/year=/week=/state_divergence.parquet. Where the routes persistently disagree, model or dispatch error is localised - treat it as evidence, not noise.

Closure reconciles each estimated day against independent national records: estimated generation/load totals vs the NESO demand series, chi-square pass rate, removal totals, and the systematic-outlier league (which measurements get removed period after period - the signal that has repeatedly fed crosswalk and model fixes). One row per day to Derived/estimation_closure/<case>/year=/week=/estimation_closure.parquet, with a JSON summary on stdout.

The scheduled task

GDA_EstimatedState runs daily at 08:40 (after GDA_UnifiedRefresh at 08:00, so the demand series is fresh), executing Scripts/refresh_estimated_state.py --backfill 2 via the standard headless wrapper. Each run:

Daily fill - picks the most recent settlement day that is at least 8 days back (safely behind both B1610 publication lag and the 7-day + 1-hour inference horizon), has a B1610 file, and is not yet in the estimated lake, then runs the full dual route for it (steps 1, 2a, 2, 3, 4). One day per run; missed days catch up one per subsequent run.
Backfill (--backfill 2) - additionally estimates up to two historical days, walking a persistent cursor backward from the lag boundary toward the 2016-03-01 PN floor. Already-estimated days and days with neither B1610 nor PN data are skipped cheaply; a day that fails to build is recorded and the walk moves on. The cursor lives in Scripts/state/estimated_backfill.json (next_day, failed_days), so every run - scheduled or manual - continues exactly where the last stopped.

Supervised deep runs

To backfill faster than two days per morning, run the same script by hand with a larger budget:

python Scripts/refresh_estimated_state.py --backfill 20

It shares the cursor with the scheduled task, so manual and scheduled progress never collide or repeat work. Budget on runtime: each day is a full dual route (every settlement period through the CLI), so size N to the time you have.

Sigma calibration

The measurement ladder's sigmas are explicit constants. To check they match reality, run the calibration harness:

python NetworkModel/calibrate_sigmas.py
python NetworkModel/calibrate_sigmas.py --days 2019-08-07,2026-05-01 --periods 12,24,36

It runs the estimator with full residual output over a spread of periods across the backfilled eras and aggregates standardised residuals per measurement source class, writing NetworkModel/resolved/sigma_calibration.json. Read rms_lev (the leverage-corrected residual RMS): ~1 means the class is calibrated, well below 1 means the declared sigma overstates the noise (information wasted), well above 1 means overconfidence (chi-square pressure and removal churn). It is a report, not an auto-tuner - ladder constants stay in reviewed code.

Where everything lands

All under D:\Work\GDA\v1\DataSources\Derived\, all weekly-partitioned (year=YYYY/week=WW/) with merge-on-write:

Product	Path	Grain
Estimated states	`estimated_state/<case>/.../estimated_state.parquet` + `<case>/topology.json`	one row per settlement period
Presolved states	`presolved_state/<case>/.../presolved_state.parquet` + `<case>/topology.json`	one row per settlement period
Divergence	`state_divergence/<case>/.../state_divergence.parquet`	one row per period both lakes cover
Closure	`estimation_closure/<case>/.../estimation_closure.parquet`	one row per day