State estimation

The estimate verb fits a network state to redundant, noisy telemetry. Unlike a power flow - which needs a complete, exactly-determined P/Q specification at every bus - the estimator takes whatever measurements exist, wherever they are, and lets redundancy fight noise. It reports how well the fit succeeded (chi-square), identifies and removes gross errors, and tells you when the measurements cannot determine the state at all (unobservable).

Results agree with pandapower.estimation to ~1e-13 pu on the IEEE cases (tools/crossval/crossval_estimation.py); an independent NumPy check (tools/crossval/se_residual_check.py) verifies any export's residuals and optimality without GridSim's math.

Running an estimate

gridsim estimate <case> (--measurements <m.json> | --synth [--seed N] [--noise S]
                        [--coverage-vm F] [--coverage-inj F] [--coverage-flow F])
                        [--method wls|constrained|huber|lav] [--prior solve|flat]
                        [--warm-start] [--backfill-removals] [--no-bad-data]
                        [--lean] [--trace] [--export-solve]
                        [--tol 1e-8] [--max-iter N] [--out <dir>]

<case> is a built-in name (case9, case14, ...), a MATPOWER .m file, or a JSON case directory containing system.json - so the GDA materialisations (gb-spine, gb-full) can be estimated on directly:

gridsim estimate case14 --synth
gridsim estimate D:\Work\GDA\v1\NetworkModel\out\gb-spine --measurements sp20.json --prior solve --warm-start

You must supply measurements one of two ways:

--measurements <m.json> - a gridsim-measurements/1 file (format below).
--synth - solve the case as ground truth, sample a deterministic noisy measurement set from the solution, estimate on it, and score the estimate against the truth. The full pipeline exercised end to end with no external data; the sampled set is saved to <out>/measurements.json so a re-run or the crossval sidecar consumes the identical input.

Flags

Flag	Default	Effect
`--measurements <m.json>`	-	load a `gridsim-measurements/1` set (mutually exclusive with `--synth`)
`--synth`	-	sample measurements from the case's own solved truth
`--seed N`	`20260713`	synthetic sampler RNG seed (deterministic per seed)
`--noise S`	`1.0`	scales the injected noise; `0` gives exact values (declared sigmas unchanged)
`--coverage-vm F`	`1.0`	fraction of buses with a voltage meter (deterministic stride over model order)
`--coverage-inj F`	`1.0`	fraction of buses with P/Q injection telemetry
`--coverage-flow F`	`1.0`	fraction of branches with P/Q flow telemetry
`--method M`	`wls`	estimation method: `wls`, `constrained`, `huber`, `lav`
`--prior P`	`flat`	operating-point prior: `flat` or `solve` (see below)
`--warm-start`	off	iterate from the case's own solved operating point instead of a flat start
`--backfill-removals`	off	replace a removed real-power row with a prior-anchored pseudo (see below)
`--no-bad-data`	off	skip the chi-square test and the removal loop
`--lean`	off	skip the residual-covariance solves when chi-square already passed (backfill throughput switch; normalised residuals become NaN diagnostics)
`--trace`	off	record the per-iteration convergence path into the export
`--export-solve`	off	additionally project the estimate into a `gridsim-solve-export/1` document
`--tol T`	`1e-8`	convergence threshold on the largest state update
`--max-iter N`	`30`	Gauss-Newton iteration cap
`--out <dir>`	`results/estimate-<case>`	output directory

There is no CLI flag for the dense/sparse switchover: above 3,000 buses the estimator selects the sparse step backend automatically (the same threshold convention as the power-flow solvers). The dense backend additionally proves observability by gain-matrix rank; the sparse backend relies on the island sweep plus singularity detection in the factorisation.

Choosing a method

wls (default) is classical weighted least squares: Gauss-Newton on the normal equations, chi-square consistency, largest-normalised-residual bad-data removal. Use it for everything unless one of the situations below applies - it is the method every other one is validated against.

constrained promotes measurement rows tagged virtual-zero-injection to hard equalities via a KKT augmented system: a switching node's zero injection is a network fact and is honoured exactly rather than approximately through a small sigma. Use it when your set carries zero-injection virtuals and the plain-WLS approximation is not tight enough. Dense backend only: above the 3,000-bus sparse threshold it fails loud rather than silently running unconstrained.

huber is a robust M-estimator (IRLS, k = 1.345): residuals within k standard deviations get quadratic loss, beyond it linear - outliers are down-weighted smoothly instead of removed. Use it when you suspect several simultaneous gross errors, where the one-at-a-time removal loop is slow or can misidentify; on clean Gaussian data it matches WLS closely.

lav is least-absolute-values (IRLS with weight proportional to 1/|residual|): the most aggressive outlier rejection - the fit follows the trustworthy majority. Use it when the measurement set may be badly contaminated and you want the estimate to ignore the liars entirely.

Robust runs (huber/lav) report chi-square against the original sigmas and skip the removal loop - the reweighting is their bad-data treatment.

Priors and warm starts

--prior selects where the operating-point prior comes from:

flat (default) - the set's own voltage-schedule rows are used as-is.
solve - the estimator builds a power flow from the set's own measured injections, solves it with the validated Newton engine, and uses the solved state twice: the set's voltage-schedule rows are replaced with the solved profile (tighter sigma, source="prior-solved"), and the solution becomes the warm start. If the prior solve does not converge, the run falls back to the flat schedule with a warning. This is the production setting for real GB data.

--warm-start (when --prior solve has not already supplied a start point) seeds the iteration from the case's own solved operating point - standard EMS practice. Falls back to a flat start with a warning if the base case will not solve. Real-data sets generally need one of these two: an undamped flat start on stressed telemetry is the classic way to walk voltage magnitudes through zero (the estimator trust-caps each step, but a good start is still worth having).

Bad data, removal and backfill

After convergence with redundancy, the objective J is tested against the 95% chi-square quantile for the degrees of freedom. On failure, the measurement with the largest normalised residual above 3.0 is removed and the estimate re-runs warm from the current state, until the test passes or nothing removable remains (cap: 20 removals). Critical measurements - sole observers of a state direction - are protected from removal, and a gross error on one is structurally undetectable; the residual table flags them.

--backfill-removals changes what "removed" means for real-power rows: when the loop removes a PInjMw or PFlowMw measurement, it is replaced by a pseudo-measurement evaluated at the start state (the solved prior), with a deliberately loose sigma (3x the removed one, floored at 5% of the value), tagged source="removal-backfill-prior" and itself never removable. Without this, removing the only informative constraint in a weakly-redundant region (a radial GSP's sole injection) leaves the state free to run away from every remaining measurement; with it, the fitted answer stays anchored to the physically-consistent prior. Reactive kinds are deliberately not backfilled. The GB backfill pipeline always sets this flag.

--no-bad-data skips the test and the loop entirely - useful when you want the raw fit to a set you already trust, or when diagnosing the removal loop itself.

The measurement file format

A measurement file is camelCase JSON with schema gridsim-measurements/1:

{
  "schema": "gridsim-measurements/1",
  "case": "case14",
  "baseMva": 100.0,
  "takenUtc": "2024-03-01T12:00:00Z",
  "note": "optional provenance",
  "measurements": [
    { "kind": "VmPu",    "value": 1.045,  "sigma": 0.004, "bus": 2,  "source": "scada" },
    { "kind": "PInjMw",  "value": 18.3,   "sigma": 1.0,   "bus": 3,  "source": "scada" },
    { "kind": "PFlowMw", "value": 156.9,  "sigma": 1.6,
      "from": 1, "to": 2, "end": "From", "circuit": "#0", "source": "scada" }
  ]
}

Kinds: VmPu (bus voltage magnitude, pu), PInjMw / QInjMvar (net bus injection, generation minus demand, MW/MVAr), PFlowMw / QFlowMvar (branch flow into the measured end, MW/MVAr), VaDeg (bus voltage angle, degrees - a PMU kind). A VaDeg value must be referenced against the model's slack: a real feed carries angles against its own arbitrary reference, so subtract the feed's angle at a chosen datum bus before authoring rows - the estimator does not solve for a per-feed offset.
Addressing: bus measurements carry the bus id (never a solver index); branch measurements carry the from/to endpoint ids, plus end (From or To) for which terminal is metered. Parallel circuits between the same endpoints need circuit to disambiguate: either the branch's name, or the ordinal form "#k" (the model branch index) when parallels are unnamed or share a name - IEEE imports and the gb-full SGT banks both need this form, and the synthetic sampler emits it.
sigma is the measurement's standard deviation in the same units as the value. It must be finite and positive - the file is validated against the model on load and every offender is named in one error.
baseMva must match the model's base; a mismatch is rejected.
takenUtc is optional, but when present it must be historical: a stamp newer than the 169-hour (7-day + 1-hour) horizon is rejected with a forward-inference error. The estimator reconstructs the past only.
source is a free provenance tag carried through to the residual table (scada, synthetic, pseudo-demand, prior-solved, virtual-zero-injection, removal-backfill-prior, ...). Rows tagged virtual-zero-injection become hard constraints under --method constrained.

Reading the output

The console summary:

=== WLS state estimate: case14 ===
  status: Converged   iterations: 3   max |dx|: 4.51E-11
  measurements: 116   states: 27   redundancy (dof): 89
  observability: observable (rank 27/27)
  consistency: J = 82.31 vs chi2(95%) = 112.02 -> PASS

status / iterations / max |dx| - the solve outcome and how far the last step moved the state. NonFinite and MaxIterations are reported honestly, never smuggled through as a converged answer.
redundancy (dof) - measurements minus states. With no redundancy the chi-square line reads n/a (no redundancy): the fit is an interpolation, not a corroborated estimate.
observability - whether the set can determine the state at all. An UNOBSERVABLE report lists the unmeasured island buses and the exact free state directions (Va@bus 12); add measurements there and re-run.
consistency - J against the 95% chi-square threshold (see below).
removed bad data - one line per removal, with the kind, location and normalised residual.
critical measurements - a count of sole-observer rows. Treat their values as unchecked telemetry, not corroborated estimates.
--synth runs add vs synthetic truth: max |dVm| ... max |dVa| ... - the estimate-against-truth score.

Exit code 0 means converged and observable; 3 otherwise; 2 for usage errors; 1 for load failures.

Files written to --out:

File	Contents
`estimate-export.json`	schema `gridsim-estimate-export/1`: the network, the measurement set, the estimated state, the residual table, observability, removals, run meta, and (with `--trace`) the per-iteration trace
`residuals.csv`	one row per measurement: `kind,location,value,sigma,predicted,residual,normalized_residual,critical`
`run-metadata.json`	backend, git commit, host, timestamp
`measurements.json`	(`--synth` only) the sampled input set
`estimated-solve-export.json`	(`--export-solve`, converged only) the estimate projected into a `gridsim-solve-export/1` document - injections, losses and flows at the estimated state - so replay, the bridge and the dashboards consume estimated states unchanged. This is the feed the GDA `Derived/estimated_state` lake is built from

What chi-square PASS/FAIL means

The test asks: are the residuals consistent with the sigmas the measurements declared? PASS means the weighted sum of squared residuals J is within the 95% quantile for the redundancy - noise looks like the noise you promised, and the estimate is statistically credible. FAIL means something is lying: at least one measurement value is wrong, at least one sigma is overconfident, or the model itself does not match the measured network.

On FAIL:

Check the removed list first - the loop may already have identified the offenders but hit the removal cap, or the worst residual may sit on a protected critical measurement.
Sort residuals.csv by normalized_residual. A single value far above 3 is a bad measurement; a broad shelf slightly above 1 across one source tag is a miscalibrated sigma class; residuals clustered on one region of the network point at a model error there (a missing shunt, a wrong impedance, a mis-sited unit).
If several simultaneous gross errors are plausible, re-run with --method huber and compare which rows it down-weights.
Confirm baseMva and the case directory match the authoring side.

A FAIL with a converged state is still a state - the lake pipeline stores such periods honestly with chi_passed = false rather than skipping them - but do not treat the numbers as corroborated until you know why the test failed.