Technical manual — chapter list

GDA integration

GridSim and the GDA data lake meet at three seams: a WebSocket bridge that streams reconstructed historical grid states into the desktop and web clients, a case assembly chain on the GDA side that materialises the GB network as GridSim case directories, and a set of measurement and estimated-state lake products that feed (and are fed by) the estimate verb. This chapter documents each seam as it is implemented. The GDA repository is separate from GridSim (its scripts are cited here as GDA NetworkModel/... and GDA Applications/...); nothing in the lake is committed to this repository.

Division of labour: GDA computes the physics on the Python side (a sparse-NR + OLTC solve in GDA NetworkModel/validate/nr_check.py, WLS estimation by invoking the GridSim CLI) and serialises through the shared gridsim-solve-export/1 schema; GridSim renders, replays and analyses. Any solved state that crosses the seam is a document SolveImport.Read can reconstruct.

The bridge protocol

The server is GDA Applications/GridSim-Bridge/bridge_server.py (FastAPI, one WebSocket at /ws/bridge; GET / and GET /meta report the available cases and the current horizon cutoff). The C# wire schema is src/GridSim.Core/IO/BridgeProtocol.cs; the client is src/GridSim.Core/Net/GridBridgeClient.cs. Messages are JSON objects tagged by a top-level type discriminator, read case-insensitively with named floating-point literals allowed (BridgeProtocol.JsonOptions). BridgeProtocol.PeekType reads the discriminator with a forward-only Utf8JsonReader scan and returns null for a truncated or malformed frame, so one bad message is skipped rather than faulting the receive pump.

Client -> server commands:

Command Shape Effect
start {action:"start", case, start:"<iso>", fps, source:"live"\|"presolved"\|"estimated"} Begin a stream from start for the named case, replaying the chosen reconstruction
stop {action:"stop"} Stop the stream
set_speed {action:"set_speed", fps} Change replay cadence (clamped 1-144 fps)
seek {action:"seek", start} Restart the stream at a new instant
query {action:"query", op, at, params, id} Run an on-demand physics/data query (answered by a result)

Server -> client messages, in the order a session sees them:

topology - once per stream

{type:"topology", case, model:{...}}. The model block is exactly the model block of a gridsim-solve-export/1 document (gridsim_export.to_solve_export in GDA NetworkModel/gridsim_export.py produces it field-for-field, including the "Infinity"/"NaN" string tokens for non-finite generator limits). The client caches the reconstructed GridModel and reuses it for every subsequent tick (BridgeProtocol.TopologyMessage.ToModel -> BridgeProtocol.BuildModel). Bus records carry the geolocated x/y layout coordinates (0..1 on the GB bounding box); without them a streamed model renders at a single map point.

The model block optionally carries a boundaries array - the case's ETYS transfer boundaries, [{name, limitMw, branches:[{branch, direction}]}] (BridgeProtocol.ModelBlock.Boundaries, populated on the GDA side from the case's boundaries.json by gridsim_export.to_solve_export). BuildModel projects them to Boundary/BoundaryBranch, which is what makes the RP1 and OPERATOR boundary bars live in bridge mode. The field is additive: a server or reader that predates it interoperates unchanged.

state - one slim solved tick

{ "type": "state", "t": "<iso-utc>",
  "result":   { "converged", "iterations", "maxMismatch", "totalLossMw", "totalLossMvar",
                "buses": [ { "id", "vm", "vaDeg", "pInjMw", "qInjMvar" } ],
                "lineFlows": [ /* solve-export flow shape */ ] },
  "scalars":  { "freqHz", "rocofHzPerS", "genMw", "demandMw", "inertiaMwS",
                "conditionIndex", "rocofHeadroomHzS" },
  "overlays": { "stressByBranch": { "<name>": frac }, "conditionByBus": { "<id>": frac } },
  "operator": { /* OperatorBlock */ }, "market": { /* MarketBlock */ },
  "source":   "live" | "presolved" | "estimated",
  "estimate": { /* EstimateBlock, source=estimated only */ } }

The result block reuses the solve-export result shape but is slim: the network is not resent, and line-flow name/ratingMva omitted on the wire are backfilled from the cached topology's matching branch (StateMessage.ToPowerFlowResult, ToLineFlow). ToSimFrame assembles the SimFrame the views bind to; generation falls back to the sum of positive bus injections when genMw is absent, and absent demand is treated as unknown (0) rather than substituted from the cached topology's load-time Pd (BridgeProtocol.cs, fix #10 comment - the cached model's demand is frozen at topology time and is wrong for other ticks).

source names the reconstruction that produced the frame: live (a static converged solve with per-period scalars attached), presolved (the demand-scaled re-solve lake Derived/presolved_state), or estimated (the measurement-conditioned WLS lake Derived/estimated_state). It is null from a server that predates the field (StateMessage.Source).

estimate rides only source=estimated frames: the per-frame estimation quality the WLS fit carries that a demand-scaled re-solve cannot. Its keys are camelCase on the wire, each mirroring a column of the estimated_state lake row (bridge_server.py:_ESTIMATE_ROW_KEYS -> BridgeProtocol.EstimateBlock):

Wire key Lake column Meaning
objectiveJ objective_j The WLS objective J at the fitted state
chiThreshold chi_threshold chi-square(95 %) threshold for the fit's degrees of freedom
chiPassed chi_passed Whether J <= threshold
dof dof Degrees of freedom (redundancy)
measurementsN measurements_n Measurements in the set
removedN removed_n Bad-data removals
iterations iterations Gauss-Newton iterations

The WPF SRC picker surfaces chiPassed as the per-frame chi-square pill on the bridge status bar (src/GridSim.Wpf/MainWindow.xaml, bridge status pill).

operator and market are the lake side-channels - per-tick series joined from Derived/operator_market_frames (BM acceptances, constraints, warnings, inertia stratification; imbalance/market prices, carbon, interconnector flows). Every field is nullable: a tick omits whatever the lake has no record for (BridgeProtocol.OperatorBlock/MarketBlock; the field-by-field reference is 10-file-and-wire-formats).

result - the on-demand query response, correlated by id

{type:"result", op, id, at, payload:{...}}. The server accepts op in {inertia, condition, scenario, feasibility, post_event, asset, propagation} (bridge_server.py:_QUERY_OPS) and runs each query as an independent task, so same-op replies can complete out of order. Correlation is therefore by id: every reply - including horizon rejections, unknown-op errors and engine failures - echoes the request's id (and op, at) verbatim (bridge_server.py:handle_query; BridgeProtocol.ResultMessage.Id). A null id marks a legacy server.

payload.solveExport, when present, is a whole gridsim-solve-export/1 document captured raw and fed straight to SolveImport.Read (ResultMessage.ToQueryResult). The payload also carries verdict, bindingConstraint and violations[] ({kind, element, value, limit, message}); op="asset" adds the four per-asset row lists (envelope, bessSoc, tapWear, consumers, keys projected snake_case -> camelCase server side); op="propagation" adds corridors[] + provenance, validated at the protocol boundary - a negative, non-finite or implausibly large tau (> PropagationData.MaxPlausibleTauSeconds = 10 s) is dropped and counted in the provenance note rather than fed to the swing model.

A query whose at is newer than the horizon cutoff is answered with payload.verdict = "INFEASIBLE_HORIZON" and no engine run.

info, error, done

{type:"info"|"error", message} (BridgeProtocol.NoteMessage, read by ReadMessage). error faults the client stream (GridBridgeClient raises the fault so the failure is visible); info is advisory and not routed to handlers. {type:"done", framesSent[, source]} closes a finished stream.

The horizon guard and clock-skew tolerance

The no-forward-inference invariant is enforced on both ends of the wire:

  • Server - a start or seek past horizon.cutoff() (now - 7 days 1 h) is rejected outright with an error; every emitted tick is additionally masked t <= cutoff, even if a stored lake happens to contain a newer row; a query at past the cutoff returns INFEASIBLE_HORIZON (bridge_server.py:stream_from_start, stream_from_presolved, handle_query).
  • Client - every received state tick's Timestamp passes through ForwardInferenceGuard.RejectFuture (HorizonHours = 169.0, i.e. 7 days + 1 h; src/GridSim.Core/Diagnostics/ForwardInferenceGuard.cs). The client applies the guard with a small clock-skew allowance: GridBridgeClient.ClockSkewToleranceMinutes (default 15 minutes) is added to the client's notion of "now" before the check, so a modest client/server clock skew does not instant-fault an otherwise valid stream. A frame beyond the allowance is still a protocol violation and is refused (GridBridgeClient.cs:241-244).

Case assembly on the GDA side

Three stages produce the case directories GridSim loads (each a JSON case folder readable by JsonGridLoader.LoadDirectory).

Normalised LTDS tables -> assemble -> gb-full

Ingestion normalises each DNO licence's Long Term Development Statement into six per-licence parquet tables - nodes, circuits, tx2, tx3, demand, gen - under GDA NetworkModel/normalized/{licence}_{table}.parquet (GDA NetworkModel/assemble/run.py:TABLE_NAMES). The assembler (python -m assemble.run gb-full) then:

  1. loads the England & Wales licences (EMID, WMID, SWEST, SWALES, EPN, LPN, SPN, NPGN, NPGY, SPM, SEPD) plus the ENW synthetic subnet, repairing known data defects at load (imputed primary demand; implausible synthetic line charging recomputed from length);
  2. runs assemble.core.assemble with a stable bus-id registry (ids.BusIdRegistry, append-only - the same physical node always gets the same id across rebuilds), min_kv 132 by default, national slack mode;
  3. geolocates buses (assemble.geo) and, for gb-full, integrates the transmission spine (assemble.spine.integrate) - Scottish DNO meshes are not assembled directly; their demand/generation attach via equivalent BSPs at spine GSPs - then re-drapes the layout;
  4. writes the case directory (assemble.writer.write_case: system.json, buses.json, generation.json, demand.json, transmission.json, transformer.json, onloadtapchanger.json, boundaries.json, diagnostics.json, MODEL_CARD.md) plus the AssetGraph parquets, and
  5. bakes a warm start (assemble.warmstart.bake): the converged operating point (bus vm/vaDeg and settled OLTC taps) is written into the case so downstream solves start near the answer (skippable with --no-warm-start).

The result is GDA NetworkModel/out/gb-full - 3,539 buses / 5,918 branches - the network the solver cross-validation runs against (docs/benchmarking.md sec. 2b).

The spine build - synthetic + curated, with the opt-in ETYS overlay

GDA NetworkModel/spine/build_spine.py builds the transmission spine that gb-full's DNO fragments hang from: one spine bus per deduplicated NESO GSP site, with a synthetic-topology / real-nodes branch set - Delaunay triangulation filtered to the Gabriel graph, pruned above 120 km, re-unioned with the Euclidean MST so the graph stays one island (build_topology). A curated corridor overlay (spine/corridors.yaml, provenance curated-corridor) fixes the kV/circuit counts of known corridors, and electrify converts per-km typical impedances into r/x/b/rating for the circuit count. Boundary constraint sets (B4, B6, B7, B8, SEIMP) are classified by documented geographic/PES rules and their limits set to the median of the NESO 24-months-ahead constraint-limit columns; circuit counts are raised until each boundary's capacity meets its limit (classify_boundaries, scale_for_limits). Outputs: spine_out/spine_nodes.parquet, spine_out/spine_branches.parquet and a GridSim-shaped boundaries.json.

SPINE_ETYS=1 - the real-circuit overlay, opt-in. apply_etys_circuits can replace the synthetic per-km guesses with the real SHET/SPT/NGET circuit list from ETYS Appendix B (resolved/etys_circuits.parquet, built by build_etys_circuits.py): where both endpoints' site codes resolve to spine sites, the parallel combination of the real circuits (z_eq = 1/sum(1/z), susceptance and winter ratings summed) replaces the synthetic edge or adds a corridor the synthetic topology never drew; rows are tagged etys-appendix-b and electrify leaves their electrical values alone. The overlay is off by default because of a documented era mismatch: ETYS 2025 describes today's network, and its precision exposes reinforcements that did not exist on historical days - under the real 2025 circuits the 2019 event day regressed from 46/46 chi-square-clean settlement periods to 32/46 with removal churn. Until the overlay is era-aware (commissioning dates / per-era circuit sets), the validated synthetic+curated spine stays the default and the overlay serves investigations (build_spine.py:apply_etys_circuits, opt-in note).

gb-spine - the emit_spine_case fold

GDA NetworkModel/emit_spine_case.py emits gb-spine, a ~316-bus GSP-level reduction of gb-full whose granularity matches the per-GSP measurement layer one-to-one - the WLS estimation target network (gb-full's ~7,000-state size is reserved for the sparse backend). The fold rule is a documented approximation, not a boundary equivalent:

  • BFS owner map (_fold_map): a simultaneous breadth-first search from all spine buses at once over the gb-full branch+transformer graph assigns every non-spine bus to the spine bus that reaches it first (deterministic tie-break: seeds enqueued in ascending spine-bus-id order).
  • Every folded bus's demand and generation sum onto its owning spine bus; sub-GSP impedances are deliberately dropped - the reduced case supplies topology and demand proportions, the measurements supply the physics. Var-only "GSP volt ctrl" units and the national slack (type == "grid") are structural and not folded; interconnectors are kept. The spine case gets its own national slack at Melksham (MELK_1), matching gb-full.
  • Bus ids are the same registry ids gb-full uses (registry/bus_ids.json, key NGET:<gsp>), so the GSP->bus join for measurements is identity-stable across both cases.
  • fold_map.json - {owner: {gb-full bus -> owning spine bus}, pes: {spine bus -> PES group}} - is written to both case directories (gb-spine and gb-full) so downstream consumers (per-group demand scaling in the estimator measurement builder) can resolve any bus to its PES group.
  • boundaries.json is copied verbatim from gb-full: the ETYS boundary cuts live on spine branches, so the same cut definitions apply to both cases and the RP1 boundary bars stay live for gb-spine.

The measurement products

GDA NetworkModel/build_gsp_measurements.py reads one settlement day's B1610 parquet once and emits three sibling products from that single read, allocation driven by the BMU crosswalk (below). Discipline is MERGE-only: pure aggregation, no interpolation, no gap-filling; energy the crosswalk cannot place is reported, never smeared.

Derived/gsp_generation_measurements - per-GSP metered generation

One row per (settlement period x GSP): datetime_utc, settlement_date, settlement_period, gsp, generation_mw, sigma_mw, units_n, min_confidence, provenance. Scope is physical production units only (productionOrConsumptionFlag == 'P', excluding interconnector units, whose energy is a boundary flow and ships separately). generation_mw is 2 x the half-hourly MWh (mean MW). A per-period gsp = NULL row carries the unallocated bucket - units the crosswalk could not bind at the confidence floor - so closure always knows exactly what is missing.

Sigma model: B1610 is settlement-grade metering, so the aggregate sigma is max(2 MW, 2 % * |MW|), inflated by the weakest crosswalk confidence in the aggregate (sigma / min_confidence); NULL-GSP rows get 30 % * |MW| (a known-missing bucket, not a measurement).

PN fallback (B-3). When the day has no (or a partial, < 40-period) B1610 file, the same aggregation runs on Physical Notification levels - the submitted per-unit output profiles, time-weighted period means over the levelFrom/levelTo ramps. PN is intent, not metering: rows carry provenance = "pn" and looser sigmas (max(10 MW, 10 %)), and no group-demand sibling is emitted (a supplier's PN is not an offtake measurement - the estimator's demand ladder falls back to national scaling). PN depth is 2016-03 onward, three estimable years earlier than B1610.

Derived/gsp_group_demand - 14 per-PES-group net demand series

From the same B1610 read: supplier settlement legs (2__X... base BM units, where the character after 2__ is the PES group id - verified 851 units, 0 mismatch against the Elexon reference). Group net demand demand_mw = -2 x sum quantity per period; net of embedded export behind the meter, which the sigma reflects: max(10 MW, 3 % * |MW|). Columns: datetime_utc, settlement_date, settlement_period, gsp_group, demand_mw, sigma_mw, units_n, provenance. This is what lets the estimator scale each group's folded demand to its measured offtake instead of one national scalar (hardening plan B-2).

Derived/gsp_ic_flows - interconnector flows at their landing GSPs

Also from the same read: the I_* units' signed half-hourly MWh aggregated via the crosswalk's interconnector binding, ic_mw = 2 x sum MWh (import positive) per (period x landing GSP). Columns: datetime_utc, settlement_date, settlement_period, gsp, ic_mw, units_n. Settlement-grade and era-deep (2019+), unlike the recent-window operator_market_frames per-link columns - the estimator's interconnector ladder prefers it and falls back to the frames only where it is absent (build_estimator_measurements.py:_ic_flows_b1610). Leaving these flows unmodelled cracked the least-redundant regions' chi-square once the fleet reached ~6.5 GW of overnight import. Its sigma model is applied at measurement-composition time: max(5 MW, 2 % * |MW|) root-sum-squared into the landing bus's injection sigma.

The estimated_state lake

GDA NetworkModel/build_estimated_state.py is the daily builder: for each settlement period of a historical day it composes the real measurement set (build_estimator_measurements.build - B1610 per-GSP generation + interconnector flows + demand/voltage pseudos), invokes

gridsim estimate <case-dir> --measurements m.json --prior solve --warm-start
        --export-solve --max-iter 100 --lean --backfill-removals --out <dir>

and stores one row per period to DataSources/Derived/estimated_state/<case>/year=YYYY/week=WW/estimated_state.parquet (weekly partition, merge-on-write deduplicated on datetime_utc, atomic tmp+os.replace). Non-converged periods are stored honestly (converged = False, no result block) rather than skipped.

Row schema (build_estimated_state.py:ROW_COLUMNS, mirrored in the GDA DataSchema.json entry Derived/estimated_state):

Column Content
datetime_utc Settlement-period instant (UTC)
case Case directory name (gb-spine, gb-full)
status, converged, iterations, max_state_update Estimator outcome (estimate block of the export)
objective_j, chi_threshold, chi_passed, dof Fit quality (chi-square threshold NaN -> stored null)
measurements_n, removed_n Set size and bad-data removal count
removed_json JSON list of removals: [{kind, bus, rn}] (null when none)
vm_min, vm_max Estimated voltage envelope
boundary_json Per-period ETYS boundary transfers at the estimated state: [{name, flowMw, limitMw}] - the signed sum of each cut branch's from-end MW, the same rule as Core/Analysis/BoundaryAnalysis. Null for rows estimated before the column existed, or when the case has no boundaries.json
result_json The slim result block {buses:[{id,vm,vaDeg,pInjMw,qInjMvar}], lineFlows:[...]} extracted from the --export-solve projection; null when not converged

A once-per-case topology.json (the solve-export model block from the first converged period) sits beside the partitions; because GridSim's solve export does not carry boundaries, the builder merges the case's boundaries.json into it before writing, so bridge-mode RP1/OPERATOR bars populate for estimated replay too. The bridge's source:"estimated" path streams these rows through the same partition-pruned reader as presolved_state.

Two derived products audit the lake. Derived/state_divergence (build_state_divergence.py) joins the solve route and the estimate route per period and records max/mean |deltaVm|, max |deltaVa|, max |deltaflow| with the worst bus/branch - where the routes disagree, model or dispatch error is localised. Derived/estimation_closure (report_estimation_closure.py) closes each backfilled day against the NESO national demand series and keeps the systematic-outlier league (measurements removed period after period feed fixes back into the crosswalk/fold). Field tables for all three are in 10-file-and-wire-formats.

The BMU -> GSP crosswalk ladder

GDA NetworkModel/build_bmu_crosswalk.py binds every Elexon BM unit to the best GSP identity the lake supports - one row per BMU, the single best method wins, and match_method/match_detail/confidence record exactly how it was bound (resolved/bmu_crosswalk.parquet). The ladder, best first:

Confidence Method Rule
0.9 name-match bmUnitName -> Derived/demand_consumers.name -> the register asset's individual GSP and bus id. bmUnitName only - the former lead-party fallback bound whole multi-site fleets to one company-named row (every RWE unit in GB to Pembroke; Keadby 1/2 to Ferrybridge) and was removed
0.75 tec-register (strong) For T_* units, a strong TEC-register binding outranks the coordinate rung: a curated station-code entry (CURATED_STATION_PROJECT - Hornsea 1/2 platforms, Seagreen, Beatrice, Moray East, Torness, the Clyde units, Whitelee) or a >= 2-token project-name match. The register knows the connection; OSUKED knows the station centroid, and the two can be tens of km apart
0.7 osuked-nearest OSUKED Power Station Dictionary coordinate -> nearest NESO anchor GSP centroid (haversine, capped at 60 km) - the only location any bounded table gives most of the metered synchronous fleet
0.75 tec-register (weak) T_* units OSUKED does not know (the newer offshore fleet): token match of lead party / unit name against resolved TEC projects, with phase-number (ordinal) disambiguation; runs after the coordinate rung
0.6 interconnector / interconnector-landing I_* units: curated interconnectorId -> register-name tokens -> the resolved interconnector register row; falling back to the curated landing-substation table (INTERCONNECTOR_LANDING: Sellindge, Chilling, Grain, Auchencrosh, Deeside, Richborough, Blyth, Bicker Fen, Pembroke) resolved against the anchor by name, then coordinate
0.5 gsp-group-only The Elexon gspGroupId - one of the 14 PES GSP groups (region granularity, not a GSP). Also recorded as a side column on every row that has one, whatever the winning method
0.0 unmapped Nothing bound the unit; the top unmapped units by capacity are reported in the build metrics

build_gsp_measurements.py allocates a unit to its GSP only at or above the confidence floor (--min-confidence, default 0.6); everything below lands on the NULL-GSP bucket. The weakest confidence inside an aggregate inflates its sigma, so a name-matched aggregate (0.9) carries less inflation than an interconnector-landing one (0.6).

See also

Provenance and invariants File and wire formats