Technical manual — chapter list

Validation

This chapter says what is validated against what, to which tolerance, and how to re-run each check. Treat every tolerance here as a maintainer expectation: if your change makes any of them worse, the change is wrong until proven otherwise. The full narrative with captured result tables is docs/benchmarking.md; this chapter is the map of the instruments.

The validation stance is two-layered by design. The primary instruments are independent NumPy re-derivations that ask "does the answer satisfy the equations?" - a stronger question than "do two iterative solvers agree?", and one that stays valid where a second solver cannot converge from a flat start. The corroboration instruments diff GridSim against named industry tools (pandapower, scipy) where those tools converge.

case9 against MATPOWER

The WSCC 9-bus case (IO/IeeeCases.Case9()) is the physics ground truth. Its solve is validated against the published MATPOWER solution: converged in 4 iterations to a max mismatch of 1.84e-14 pu, total real loss ~ 4.641 MW (tests/GridSim.Tests/PowerFlowTests.cs; docs/benchmarking.md sec. 2a canonical figures). Its dynamic data is the published Anderson & Fouad WSCC set - H = 23.64 / 6.40 / 3.01 s on MBase 247.5 / 192 / 128 MVA, system total ~ 7.465 GVA.s - pinned by tests/GridSim.Tests/Case9InertiaTests.cs (three facts: per-generator H/MBase, per-generator stored energy, the system total). Inertia does not enter the load flow; the pin protects the frequency work.

The independent residual check (power flow)

tools/crossval/residual_check.py rebuilds the bus admittance matrix from the exported branch/shunt data with the standard MATPOWER pi-model - in NumPy, a different language and codebase - evaluates S = V ⊙ conj(Y*V) * baseMVA at every bus from GridSim's solved voltages, and compares it to the specified injections, additionally checking every PV bus's implied reactive output against its aggregate Q-limits. Expectation: every case solves to a machine-epsilon residual (worst mismatch anywhere across the 14-case desktop ladder, including the 3,539-bus gb-full, is ~ 2e-7 MW; a tolerance of 1e-6 MVA on the 100 MVA base is the pass line) with zero Q-limit violations (docs/benchmarking.md sec. 2a).

gridsim crossval <case|file.m|case-dir> --tol 1e-6      # solve + export + both checks
python tools/crossval/residual_check.py <solve-export.json> --tol 1e-6

Machine-epsilon pandapower cross-checks

Power flow

tools/crossval/crossval.py (driven by the crossval verb) rebuilds the network in pandapower 3.4.0, solves with enforce_q_lims=True on both sides, aligns voltages to the slack, and diffs. Expectations on the reference cases: max |deltaVm| between 4.4e-16 and 5.5e-12 pu, max |deltaVa| between 1.6e-14 and 2.6e-10 deg (case9/14/30/57/118/145 - docs/benchmarking.md sec. 2d). Where pandapower disagrees or fails (case24's multi-generator Q-limit switching, case300 flat-start non-convergence, PEGASE singular Jacobians), the residual check arbitrates - in each captured instance GridSim's solution satisfies the equations. tests/GridSim.Tests/CrossvalContractTests.cs locks the export-JSON field contract in CI so the sidecars cannot silently break; the Python side is env-gated (absent pandapower degrades gracefully).

State estimation

tools/crossval/crossval_estimation.py feeds pandapower.estimation (WLS, flat start, tol 1e-8) the identical measurement set and diffs the estimated states slack-aligned. Expectations on the seeded synthetic sets (--synth --seed 20260713, full coverage, nominal noise): max |deltaVm| <= ~2.3e-13 pu, max |deltaVa| <= ~2e-11 deg on case14/30/118. --bad-data corroborates identification: a +20sigma gross error injected into a case14 flow makes both tools' chi-square reject and both remove exactly that measurement (docs/benchmarking.md sec. 2e). tests/GridSim.Tests/EstimateExportContractTests.cs locks gridsim-estimate-export/1 in CI.

The scipy RK45 swing agreement

tools/crossval/crossval_freq.py re-integrates the identical swing physics from a gridsim-frequency-export/1 document with scipy.solve_ivp (RK45) and compares trajectories. Expectation: the 20 ms explicit-Euler trace matches the high-order integrator to < 2e-4 Hz - captured: initial RoCoF -0.165 vs -0.16484 Hz/s (delta 1.6e-4), nadir 49.2420 vs 49.2418 Hz (delta 2.2e-4) (docs/benchmarking.md sec. 2d). The export carries the discrete recurrence in plain text precisely so this re-derivation needs no GridSim code.

The independent estimation instrument - se_residual_check.py

tools/crossval/se_residual_check.py is the estimator's primary instrument: it validates a gridsim-estimate-export/1 document with no GridSim math involved (it borrows only residual_check.py's independent Ybus/pi-model). Four checks (se_residual_check.py docstring):

  1. Honesty - the reported residuals equal z - h(x^) with h evaluated from the independently built Ybus (expectation ~1e-12).
  2. Objective - the reported J recomputes exactly as sum (r_i/sigma_i)^2.
  3. Optimality - x^ is the true WLS minimum: one independent Gauss-Newton step deltax = (H^TWH)^-1H^TW(z - h(x^)), with a finite-difference Jacobian (no shared analytic algebra), moves ||delta-x||inf ~ 1e-11.
  4. Leverage (--leverage) - re-derives the residual-covariance diagonal (K_ii, Omega_ii = sigma_i^2(1 - K_ii)) and arbitrates the export's criticality flags. This mode caught a real pivot-order bug in the estimator's factor-once LU on its first day of use - keep it in the loop when touching the bad-data path.
gridsim estimate case118 --synth --out results/est
python tools/crossval/se_residual_check.py results/est --tol 1e-6 --leverage

Exit codes: 0 all checks pass, 1 mismatch, 3 unreadable.

The Monte-Carlo sweep

tools/crossval/mc_estimate_sweep.py sweeps seeds x noise scale x measurement coverage through estimate <case> --synth (default case118, 10 seeds per configuration; the sampler scores each estimate against its own sampled truth) and aggregates error statistics. Three properties are the expectations (docs/benchmarking.md sec. 2g; raw table tools/crossval/mc_estimate_sweep_case118.json):

  • error scales linearly with noise (x2 noise -> ~x2.3 error);
  • accuracy degrades gracefully with redundancy (dof 863 -> 30 costs only ~x2 in error);
  • the chi-square gate is honest - measurements corrupted to twice their declared sigma fail consistency 10/10 while every honestly-declared configuration passes 10/10.
python tools/crossval/mc_estimate_sweep.py                # case118, 10 seeds
python tools/crossval/mc_estimate_sweep.py --case case30 --seeds 5

Agreement tests - dense<->sparse, method<->method, and the bitwise gate

These run inside the ordinary test suite and pin the claim that alternative numerical paths are the same physics:

  • Dense <-> sparse power flow. Where both engines solve a case they agree on iterations and mismatch (bench reports both; the desktop matrix is the captured evidence), and the sparse path's solutions pass the same residual check. GPU solve exports are bit-for-bit the CPU's.
  • Dense <-> sparse estimation. tests/GridSim.Tests/SparseEstimatorTests.cs forces the sparse backend (SparseBusThreshold = 0) on the standard cases and asserts: exact-truth recovery to < 1e-9 pu / 1e-8 deg, dense-vs-sparse agreement on a noisy set to < 1e-8 pu / 1e-7 deg with matching J and chi-square verdicts, and identification of the same gross error. The sparse observability report is honest about its weaker proof (rank = -1, island sweep instead of a rank certificate).
  • Method <-> method. Also in SparseEstimatorTests.cs: Huber matches WLS on clean data and shrugs off multiple gross errors; LAV estimates through corruption; --method constrained honours virtual-zero-injection rows exactly (hard KKT equalities) while agreeing with WLS elsewhere.
  • The bitwise injection-agreement gate. SparseEstimatorTests.Neighbour_injection_evaluation_is_bit_identical_to_the_solver_path pins that the measurement model's neighbourhood-only injection evaluation equals the full-scan solver injections bit for bit (Assert.Equal on doubles, no tolerance) at an off-solution state. The skipped terms are exact structural zeros, so any drift between the estimator's h(x) and the solver's power equations is a copy-divergence bug - this test makes such a divergence a hard failure, not a slow numerical erosion. Preserve it as an exact equality; loosening it to a tolerance defeats its purpose.
  • Export determinism. SolveExportTests assert the recorder is observational (bit-identical voltages with and without a trace) and the JSON byte-for-byte reproducible for a deterministic solve.

The test suite

dotnet test tests/GridSim.Tests        # expect 552 passing

552 tests, 0 failed/skipped (verified on this tree; ~20 s). The suite is 70 fixture files under tests/GridSim.Tests/; the larger fixtures and what they guard:

Fixture Guards
BridgeProtocolTests (17) Wire-message parsing, model rebuild, slim-state reconstruction, correlation ids
GridModelValidateTests (15) Model validation diagnostics
SolverPropertyTests (13) Physical invariants any converged solution must obey: power balance, non-negative losses, slack closure, dense<->sparse agreement, bit-for-bit determinism
WlsEstimatorTests (11) The estimation loop: convergence, chi-square, determinism, priors
SparseEstimatorTests (10) Dense<->sparse/method<->method agreement + the bitwise injection gate
MatpowerParserTests (10) .m parsing and unit normalisation
InertiaAndLimitsTests / FrequencyIntegratorTests (10 each) Inertia estimation, limit ladder, integrator behaviour
GdaPathsTests / ForwardInferenceGuardTests (9 each) GDA path resolution; the 169-hour horizon (reject/accept boundaries)
AnalysisCoverageTests (9) Contingency/boundary analysis coverage
QLimitRevertTests (8) + QLimitTests PV->PQ switching and reversion
StabilityMetricsTests (7) Purchased time / MIF tables reproduced from the published corpus
EstimateExportContractTests, CrossvalContractTests The sidecar JSON contracts, locked without Python
EventCatalogTests, ReplayValidatorTests, EvidenceReportTests, ScenarioEngineTests The evidence chain and its reproduced-baseline gate
Distributed*Tests (6 fixtures) The distributed inertia model: exact reduction to lumped, invariants, qualitative behaviour, scale divergence

The pedantic build is part of validation: dotnet build GridSim.slnx -c Release must produce 0 warnings / 0 errors (Directory.Build.props turns on TreatWarningsAsErrors + .NET analysers).

bench and the expected-non-convergence list

gridsim bench sweeps every case x every available engine (dense, sparse, BFS, --gpu adds CUDA/ROCm) with 10 timing runs each, writing Markdown + CSV

  • run-metadata.json; nminus1-sweep, stress and chain exercise the solver under load (docs/benchmarking.md sec. 1, sec. 3). When reading a bench report, three NO classes are properties of the input, not solver bugs (docs/benchmarking.md sec. 5): PGLib-OPF operating points (optimisation solutions, not flat-start power flows - their non-OPF twins solve), radial distribution feeders (BFS territory, a scope boundary), and the 135/sqrt(3) parser gap in two case533mt files. OOM is a dense-path memory ceiling, recorded only when the allocation actually throws.
gridsim bench                          # solver x backend matrix
gridsim bench --gpu                    # add GPU engines when a device answers
gridsim nminus1-sweep --all --sample 3000 --parallel 28
gridsim stress --all --from 0.5 --to 3.0 --step 0.05
gridsim chain case118 --steps 100 --mode fixed

GDA-side validation of the estimated history

The estimated GB history has its own closure instruments, run per backfilled day (details in 09-gda-integration):

  • Derived/state_divergence - the demand-scaled solve route vs the measurement-fitted estimate route per period; divergence localises model or dispatch error.
  • Derived/estimation_closure - estimated generation/load totals closed against the NESO national demand series, chi-square pass rate, and the systematic-outlier league (repeat-removed measurements feed crosswalk/fold fixes).
  • Sigma calibration - GDA NetworkModel/calibrate_sigmas.py aggregates standardised residuals per measurement source class across eras; the expectation is no overconfident class in the raw r/sigma view (rms <= ~1.4), with the leverage-corrected floor attributed to network fidelity, not sigma tuning (docs/benchmarking.md sec. 2f).

Tolerance summary - what a maintainer must preserve

Check Instrument Expectation
case9 solve MATPOWER published solution 4 iterations, 1.84e-14 pu, loss ~ 4.641 MW
Any converged solve residual_check.py Machine-epsilon residual (<= 1e-6 MVA gate; worst observed ~ 2e-7 MW), 0 Q-limit violations
Power flow vs pandapower crossval.py <= ~1e-12 pu / ~1e-10 deg where pandapower converges
Estimation vs pandapower crossval_estimation.py <= ~1e-13 pu / ~1e-11 deg on the seeded synthetic sets
Swing vs scipy RK45 crossval_freq.py < 2e-4 Hz on RoCoF and nadir
Estimate honesty/optimality se_residual_check.py Residuals ~1e-12; independent GN step ~1e-11
Dense <-> sparse estimation SparseEstimatorTests < 1e-8 pu / 1e-7 deg, same chi-square verdict, same removals
Injection evaluation bitwise gate Exact (bit-for-bit), no tolerance
MC sweep mc_estimate_sweep.py Linear-in-noise error; chi-square 10/10 honest each way
Suite dotnet test 552 / 552 passing; Release build 0 warnings

See also

File and wire formats Glossary