Validation tools

How to run each of GridSim's validation instruments and read what they tell you. The pattern is always the same: GridSim exports its inputs and its answer, and an independent tool (pandapower, NumPy/SciPy) re-derives the answer from the recorded inputs. Agreement is the trust signal; disagreement means either your build or your input is broken.

Requirements

The C# side needs nothing extra. The Python sidecars in tools/crossval/ need:

pip install pandapower scipy   # numpy comes with both

Without Python or pandapower the crossval verb degrades gracefully: it still writes the export, reports the sidecar as env-gated, and exits 77 (skipped) rather than 0 (validated) - a CI gate can tell "agrees" apart from "never ran".

crossval - power flow vs pandapower

gridsim crossval case118
gridsim crossval <path-to-gb-full-case-dir>          # any JSON case dir with system.json
gridsim crossval results/run/solve-export.json       # or an existing export

Solves the case, writes solve-export.json, then runs tools/crossval/crossval.py, which rebuilds the identical network as a MATPOWER-style ppc, re-solves it with pandapower's Newton power flow (Q-limits enforced on both sides), and diffs bus voltages and slack-aligned angles.

Healthy result - the classic cases agree far below the default --tol 1e-3:

=== Cross-validation vs pandapower: case118 ===
  buses compared: 118
  max |delta Vm|: 6.7E-16 pu   max |delta Va|: 0.0000 deg
  tolerance: 1.0E-03   -> AGREES

case9/14/30/57/118/145 agree to ~1e-12 pu or better (machine epsilon). If crossval reports agreement orders of magnitude worse than that on an embedded case, your build is broken - do not tighten the tolerance to make it pass.

Bad results. MISMATCH (exit 3) means the two solvers landed on different states: check crossval-diff.json in the output directory for the worst buses. Exit 77 means the sidecar never ran (install Python/pandapower). Some pandapower failures are known and are pandapower's, not GridSim's - case300 will not flat-start there and case24's multi-generator Q-limit handling switches a bus it should not; the independent residual check (tools/crossval/residual_check.py) is the arbiter in those cases (see docs/benchmarking.md sec. 2a and sec. 2d).

crossval_estimation.py - state estimation vs pandapower

gridsim estimate case118 --synth --out results/est-118
python tools/crossval/crossval_estimation.py results/est-118            # reads estimate-export.json
python tools/crossval/crossval_estimation.py results/est-118 --bad-data

Reads a gridsim-estimate-export/1 document (a run directory works - it appends estimate-export.json), rebuilds the network, feeds pandapower's WLS estimator the identical measurement set, and diffs the estimated states slack-aligned.

Flag	Default	Meaning
`--tol-vm` / `--tol-va`	1e-6 / 1e-5	Agreement tolerances (pu / degrees).
`--bad-data`	off	Feed pandapower the ORIGINAL set (gross errors included), run its chi-square + largest-normalised-residual removal, and check both tools fingered the same measurement(s).
`--out`	-	Write the diff JSON.

Healthy result: "within_tolerance": true with max_dvm_pu around 1e-13 on the seeded synthetic sets (case14/30/118). In --bad-data mode, "counts_agree": true - a +20sigma gross error makes both tools' chi-square reject and both remove exactly that measurement. Exit codes: 0 within tolerance, 1 mismatch, 2 pandapower absent, 3 unreadable export, 4 pandapower estimation crashed.

Bad result: a state mismatch means the two WLS estimators disagree on the optimum for the same inputs - suspect the estimator before the export plumbing, and run se_residual_check.py next: it decides whose optimum is real.

se_residual_check.py - the independent optimality check

python tools/crossval/se_residual_check.py results/est-118 [--tol 1e-6] [--fd] [--leverage]

The more fundamental instrument, with no GridSim math and no pandapower involved. From the same estimate-export.json it rebuilds Ybus and h(x^) in NumPy and checks:

Honesty - the reported residuals equal z - h(x^) evaluated independently.
Objective - the reported J recomputes exactly as sum(r_i/sigma_i)^2.
Optimality - one independent Gauss-Newton step from x^ is negligible (x^ really is the WLS minimum, not just a point the estimator stopped at).

Flag	Default	Meaning
`--tol`	1e-6	Residual-honesty tolerance (engineering units) and optimality-step tolerance (pu/rad).
`--fd`	off	Cross-check the script's own analytic Jacobian against central differences - the instrument validating itself. Small cases only (it is O(states) full re-evaluations).
`--leverage`	off	Recompute the residual-covariance diagonal (leverage K_i_i) and arbitrate the export's criticality flags: CONFIRMED or `NOT critical (numerical artifact)`.

Healthy result:

[se-check] residual honesty: max |(z - h_indep) - reported| = 1.2e-12
[se-check] objective: recomputed J = 96.31  reported = 96.31  |dJ| = 1.4e-13
[se-check] optimality: independent gauss-newton step |dx|_inf = 3.1e-11
[se-check] PASS: residuals honest, objective consistent, x_hat is the independent WLS optimum

Expect honesty ~1e-12 and an optimality step ~1e-11. Exit 0 pass, 1 any check failed, 3 unreadable export.

Bad results. A honesty failure means the export lies about its own residuals (plumbing bug). An optimality failure means the estimator stopped somewhere that is not the WLS minimum (convergence or Jacobian bug). A --leverage failure means a measurement is flagged critical that independently is not (or vice versa) - a numerical-cancellation bug in the estimator's factorisation path, which this check has caught before.

mc_estimate_sweep.py - Monte-Carlo accuracy vs redundancy

dotnet build src/GridSim.Cli -c Release        # the sweep drives the Release binary
python tools/crossval/mc_estimate_sweep.py                     # case118, 10 seeds
python tools/crossval/mc_estimate_sweep.py --case case30 --seeds 5

Sweeps seeds x noise scale x measurement coverage through estimate <case> --synth (seven fixed configurations, from full coverage at half noise down to injections-only) and aggregates estimate-vs-truth error per configuration. It drives src/GridSim.Cli/bin/Release/net10.0/GridSim.Cli.exe - build Release first. Output: a console table plus tools/crossval/mc_estimate_sweep_<case>.json.

Healthy result (case118, 10 seeds): error scales linearly with noise (x2 noise -> ~x2.3 error); accuracy degrades gracefully as redundancy falls (dof 863 -> 30 costs only ~x2 in error); and the chi-square gate is honest - the noise x2 configuration (data corrupted to twice its declared sigma) fails consistency 10/10 while every honestly-declared configuration passes 10/10. Reference numbers are in docs/benchmarking.md sec. 2g.

Bad result: a chi-square pass rate that stays high under noise x2, or error that explodes rather than doubles as coverage falls, means the sigma model or the gain matrix is mis-built.

bench and bench --estimate - the timing instruments

gridsim bench                      # solver x backend matrix, whole case archive
gridsim bench --gpu                # add ROCm/OpenCL/CUDA engines when a device answers
gridsim bench --estimate           # WLS estimator, dense vs sparse step backend

bench is a performance instrument, but its convergence columns are a validation signal too: where dense and sparse both solve a case they must agree on iterations and mismatch. Healthy: every non-expected-failure case shows yes / <n> / ~1e-9..1e-12 / <ms>; sparse carries the large cases (9,241-bus PEGASE in ~6 iterations). Bad: a **NO** on a case outside the canonical expected-non-convergence list (see 17-troubleshooting.md), or dense and sparse disagreeing on a case both converge, is a solver regression. bench --estimate likewise: dense and sparse step backends agree to ~1e-8 on case118 with identical verdicts; a verdict difference between backends is a bug, not noise.

gpucheck - GPU preflight

gridsim gpucheck

Reports each backend's availability (CPU, ROCm, OpenCL, CUDA) and runs one tiny case9 solve on each, flushing every line. Healthy (on a CPU-only box the ROCm/OpenCL/CUDA sections just say not present - skipped; an OpenCL device without FP64 is reported rejected: no usable FP64 and skipped):

[CPU] cpu   (vendor=Cpu)
  CPU dense: build solver... solve case9... PASS  (4 it, 2.1 ms, |mis|=1.8E-14)
  CPU sparse: build solver... solve case9... PASS  (4 it, 5.0 ms, |mis|=1.8E-14)

Bad: a FAIL line names the failing call; if the process dies mid-line, that call segfaulted (a P/Invoke signature mismatch on the GPU binding) - the last printed line identifies it. A ran but NOT converged on case9 means the backend's arithmetic is wrong, since case9 always converges on a healthy build. Driver gotchas (e.g. CUDA on A100 needing driver >= R550) are in docs/gpu-rental-runbook.md.

Physics-law conformance

gridsim physics-check <case> audits a solved state against the foundational physical laws (energy conservation, Ohm, Joule, Kirchhoff, complex power, ...) and prints a per-law verdict plus a GATE line - the state is physically admissible when every hard law holds. gridsim conformance sweeps that audit across the whole case corpus and writes a case × law matrix to results/conformance/. To constrain the estimator so it can only ever produce a physically-possible state, run gridsim estimate <case> --method feasible. The independent NumPy cross-check is python tools/crossval/physics_audit.py <export.json>. Full details: technical ch. 16.