Technical manual — chapter list

File and wire formats

This is the field-by-field reference for every persistent and on-the-wire format GridSim reads or writes: the six versioned JSON document schemas, the bridge WebSocket messages, the CLI sidecar files, and the GDA parquet products the estimation pipeline produces and consumes. Every field is named against the serialiser that emits it or the DTO that binds it. The hand-editable JSON case folder (buses.json, generation.json, ...) is a loader input rather than an export and is documented with the data model in 02-data-model-and-per-unit and ../manual/02-loading-networks.md.

Schema strings and serialisation conventions

Each archived document carries a slash-versioned schema discriminator; the major suffix is a hard compatibility boundary (a /1 reader treats a /2 document as foreign).

Schema Owner Written by
gridsim-solve-export/1 IO/SolveExport.cs / SolveImport.cs --enableExport, --export-solve, GDA gridsim_export.py
gridsim-estimate-export/1 IO/EstimateExport.cs estimate verb
gridsim-measurements/1 Estimation/MeasurementSet.cs estimate --synth sampler, GDA build_estimator_measurements.py, hand authoring
gridsim-event-catalog/1 Events/EventCatalog.cs EventCatalog.Save; GDA build_event_catalog.py
gridsim-evidence/1 Evidence/EvidenceReport.cs evidence verb
gridsim-frequency-export/1 IO/FrequencyExport.cs freq/replay-event/transient frequency dumps

Bridge messages are transient wire frames, discriminated by a type field (topology | state | result | info | error | done) rather than a versioned schema string.

Shared reader tolerances (BridgeProtocol.JsonOptions, JsonGridLoader): property binding is case-insensitive (PascalCase and camelCase both bind), // comments and trailing commas are skipped, and NumberHandling = AllowNamedFloatingPointLiterals lets the string tokens "Infinity" / "-Infinity" / "NaN" deserialise into double fields - how a generator's qmin/qmax of +/-inf survives the wire. Writers are indented and omit null-valued fields (WhenWritingNull); enums serialise by name (JsonStringEnumConverter), so a ProvenanceTag appears as "Recorded", not an ordinal.

One casing quirk to know: SolveExport and EstimateExport build their documents from anonymous objects (already camelCase) but serialise the meta record without a naming policy, so meta keys are PascalCase (Case, Backend, GitCommit, TimestampUtc, ElapsedMs, Machine - SolveExportMeta, IO/SolveExport.cs:10) while everything else is camelCase. GDA's gridsim_export.py reproduces this exactly. FrequencyExport sets a camelCase policy globally, so its meta is camelCase (event, gitCommit, ...). Readers bind case-insensitively, so the asymmetry is harmless but visible in the files.

gridsim-solve-export/1

The audit record of one power-flow solve (docs/solve-export.md; IO/SolveExport.cs:ToJson). Byte-for-byte reproducible given a deterministic solve; the recorder is observational (attaching it does not change the solution).

Top level: {schema, meta, model, solve, result}.

meta - Case, Backend (dense-newton | sparse-newton | bfs, or a GDA engine string), GitCommit, TimestampUtc, ElapsedMs, Machine.

model - the network exactly as solved: name, baseMva, busCount, and three arrays:

Array Fields
buses[] id, type (Pq|Pv|Slack), baseKv, pd, qd, gs, bs, vm, name
generators[] bus, pg, qg, vg, qmin, qmax, pmax, mBase, h, type, name
branches[] from, to, r, x, b, tap, phaseShiftDeg, rateAMva, name, isTransformer

Impedances are p.u. on baseMva; pd/qd are MW/MVAr; qmin/qmax/pmax may be +/-Infinity; h is the inertia constant in seconds (dynamics only). A tap of 0 reads back as the 1.0 nominal (BridgeProtocol.BuildModel). GDA-produced exports additionally carry model.boundaries[] - {name, limitMw, branches:[{branch, direction}]} - the ETYS transfer boundaries (gridsim_export.to_solve_export; BridgeProtocol.BoundaryDto). The field is additive; readers that predate it ignore it.

solve - the solver's working: scalars solver, n, converged, iterations, maxMismatch, jacobianOmittedForSize, slackQNote, then

  • ybus[] - non-zero admittance entries {I, J, G, B}. Empty for BFS, which forms no Y-bus.
  • busSpec[] - {index, id, type, vmInit, vaInitDeg, pspec, qspec, qd, vg, qmin, qmax} per bus.
  • qLimitEvents[] - every PV->PQ switch: {busIndex, busId, pinnedAt, pass}.
  • passes[] - one per reactive-limit outer pass: {pass, effectiveTypes, pvpqIndices, pqIndices, iterations[]}, each iteration {iteration, maxMismatch, mismatch[], correction[], jacobian[][]}.

mismatch[k]/correction[k] share the pass ordering: the first pvpqIndices.Length entries are the P equations over non-slack buses, the remainder the Q equations over PQ buses. jacobian is present only for dense solves within the size gate (automatic for n <= 300, forced with --export-jacobian); it is captured before the factorisation. For BFS each iteration records per-bus |deltaV| as mismatch and correction is empty. GDA-produced exports carry only the scalar solve header (no trace arrays).

result - totalLossMw, totalLossMvar, buses[] {id, vm, vaDeg, pInjMw, qInjMvar}, and lineFlows[] {from, to, pFromMw, qFromMvar, pToMw, qToMvar, pLossMw, qLossMvar, name, ratingMva, loadingPercent}. A branch's pLossMw is pFromMw + pToMw; the totals are the sums over lineFlows.

gridsim-estimate-export/1

Everything one state estimation computed (IO/EstimateExport.cs:ToJson) - the whole input contract of the crossval_estimation.py / se_residual_check.py sidecars, locked in CI by tests/GridSim.Tests/EstimateExportContractTests.cs. Named float literals are allowed because normalised residuals are NaN for critical measurements and the chi-square threshold is NaN without redundancy.

Block Content
meta SolveExportMeta (PascalCase keys, as above)
model Exactly the solve-export model block, so the sidecars reuse one rebuild path
measurements The set the run was asked to fit: measurementSchema (gridsim-measurements/1), case, takenUtc, note, count, items[] (measurement fields below) - the original set, gross errors and all
estimate status (EstimationStatus name), converged, iterations, maxStateUpdate, objectiveJ, degreesOfFreedom, chiSquareThreshold (NaN without redundancy), chiSquarePassed, buses[] {id, vm, vaDeg}
residuals[] Per measurement that survived to the final fit: measurement (fields below), predicted, residual (engineering units), normalizedResidual (NaN when not computed - --lean on a passing fit - or critical), critical
observability observable, rank (-1 from the sparse backend, which runs the island sweep instead of a rank proof), stateSize, unobservableBuses[], unobservableStates[] (e.g. "Va@bus 12"), unmeasuredIslandBuses[]
removed[] Bad-data removals in order: measurement, normalizedResidual, pass
trace[] Optional, additive (--trace): the per-iteration convergence path - {pass, iteration, maxStateUpdate, stepScale} per Gauss-Newton iteration (Estimation/EstimationTrace.cs). Absent (null, omitted) when no trace was captured

A serialised measurement (EstimateExport.MeasurementDoc) is {kind, value, sigma, bus | from+to+end, circuit, source} - bus fields null for branch kinds and vice versa.

The difference between measurements.items and residuals is exactly removed.

gridsim-measurements/1

The estimator's input: one case's telemetry (Estimation/MeasurementSet.cs; camelCase, enums by name).

Top level: schema (must equal gridsim-measurements/1 - Load rejects anything else), case (informational), baseMva (checked against the model on Validate, default 100), takenUtc (optional; when present it must clear the 169-hour horizon or Load throws ForwardInferenceException), note (free provenance - the synthetic sampler records seed and coverage here), and measurements[]:

Field Meaning
kind VmPu, PInjMw, QInjMvar, PFlowMw, QFlowMvar, or VaDeg (PMU phase angle, degrees against the estimator's slack reference) - Estimation/Measurement.cs:MeasurementKind
value Engineering units (pu for voltage, MW/MVAr for power, degrees for angle); converted to per-unit once at ingestion
sigma Standard deviation in the value's own units; must be finite and > 0
bus Bus id (bus kinds)
from, to Endpoint bus ids (branch kinds), matched in the authored orientation
end From | To (default From) - which terminal the flow is measured into
circuit Disambiguates parallel branches: the branch name, or the ordinal form "#k" (model branch index) for unnamed parallels (MeasurementSet.TryResolveBranch)
source Free provenance tag (scada, pseudo-demand, virtual-zero-injection, pseudo-voltage-schedule, reference, synthetic, ...) - some tags are load-bearing: virtual-zero-injection rows become hard equalities under --method constrained, and pseudo-voltage-schedule rows are the ones --prior solve replaces

Validate(model) throws one InvalidDataException naming every offender: non-finite values, non-positive sigmas, unknown bus ids, unresolvable or ambiguous branch references, base-MVA mismatch.

gridsim-event-catalog/1

The store of recorded GB frequency events (Events/EventCatalog.cs; data/events/). The file is {schema:"gridsim-event-catalog/1", events:[...]}. Every event's whenUtc passes through ForwardInferenceGuard on load, so a catalogue cannot smuggle a future "event" into the pipeline.

GridEvent (Events/GridEvent.cs):

Field Meaning
id Stable id (e.g. gb-2019-08-09)
title Human title
whenUtc Onset timestamp (UTC), horizon-checked on load
preEventInertiaMwS System inertia before the event, MW.s
preEventDemandMw Pre-event demand, MW
losses[] PowerStep {atSeconds, deltaPMw, what, provenance} - negative deltaPMw is a loss, positive a restoration
recordedActions[] Same shape; protective/operator actions (LFDD enters as positive deltaP - load removed)
outcome RecordedOutcome below
provenance ProvenanceTag
notes Free text (carries [VERIFY] where unsourced)

RecordedOutcome: initialRocofHzPerS, nadirHz, settleHz?, lfddMwShed, nesoTraceCsvPath? (the data gate for the per-sample RMSE replay check - drop a real 1-second trace there to enable it), provenance, notes?.

Two projections matter to consumers: ToFrequencyEvents() merges losses and recorded actions time-ordered (the baseline replay input - the recorded LFDD is a fact to reproduce), while ExogenousLossEvents() returns losses only (the counterfactual input - a device is meant to replace the LFDD, so feeding it back in would be wrong).

Two built-ins ship (EventCatalog.Builtin): gb-2019-08-09 is sourced from the Ofgem investigation report (loss sequence, 892 MW LFDD, outcome; the steps carry Recorded/DerivedFromRecorded tags and Ofgem paragraph references), with the measured operating point additionally shipped as data/events/gb-2019-08-09-estimated.json (built by GDA build_event_catalog.py from the recorded outturn inertia and the WLS-estimated demand). gb-largest-loss-1320 is a nominal screening case whose figures remain CuratedPlaceholder + [VERIFY] until sourced - the tags are intentional and must be preserved.

gridsim-evidence/1

The evidence dossier: {schema, report} where report is the EvidenceReport record (Evidence/EvidenceReport.cs:61-72, 82), camelCase, enums by name:

Field Meaning
eventId, eventTitle, whenUtc, eventProvenance The replayed event
baselineVerdict Reproduced | Marginal | Failed
counterfactualUnlocked Structurally baselineVerdict == Reproduced
columns[] EvidenceColumn below (two always; a third when unlocked)
riskCurve? {deviceName, points[]}; each RiskPoint compares device vs no-device at one inertia level over SweepAxes.GbInertiaTrajectoryGvaS (260/155/120/102/50 GVA.s): inertiaGvaS, rocofNoDevice/nadirNoDevice/lfddNoDevice, rocofWithDevice/nadirWithDevice/lfddWithDevice, lfddAvoidedByDevice
fleet? FleetComparison {inertiaGvaS, units, staggerSeconds, aggregateNadirHz, fleetNadirHz, aggregateFfrAtNadirMw, fleetFfrAtNadirMw} (defaults: 100 units, 10 s stagger)
avoidedCost? AvoidedCostResult {lfddMwShedAvoided, vollPerMwh, durationHours, avoidedCostGbp, provenance, basis} - emitted only when the device keeps the nadir above LFDD and the record actually shed load. VoLL default £6,000/MWh and 0.5 h duration are CuratedPlaceholder policy numbers (Evidence/AvoidedCost.cs)
summary One-line human verdict

EvidenceColumn: label, provenance, rocofHzPerS, nadirHz, settleHz?, lfddAtRisk (nadir <= 48.8 Hz), deviceRotationalInertiaGvaS, deviceFfrGw - the two device quantities are separate labelled fields by design (intrinsic inertia is never blurred with the FFR injection), and are non-zero only on the counterfactual column. When the baseline does not reproduce, columns stops at two and riskCurve/fleet/avoidedCost are omitted by the null-ignoring writer.

gridsim-frequency-export/1

The swing-solve analogue of the solve export (IO/FrequencyExport.cs:ToJson) - everything a frequency solve used and produced, so RoCoF = f0*deltaP/(2E) is hand-checkable. All camelCase (including meta). Blocks:

Block Fields
meta event, gitCommit, timestampUtc, elapsedMs, machine, provenance (FrequencyExportMeta)
inertia systemInertiaMwS, deviceRotationalInertiaMwS (real (1/2)J*omega^2, never df/dt-synthetic), effectiveInertiaMwS (= the E in the swing equation, max(1, system + device)), note
fastResponse fastResponseMw, timeConstantS, deadbandHz, note - FFR is a separate power injection, deliberately not part of E
demandMw Scalar
config nominalHz, droopPu, dampingPctPerHz, responsiveFraction, governorTimeConstantS, stepSeconds, durationSeconds (the DynamicsConfig used)
events[] {atSeconds, deltaPMw}, time-ordered
recurrence[] The exact discrete recurrence as six plain-text lines, so the trajectory can be recomputed by hand
headline initialRocofHzPerS, nadirHz, nadirSeconds, settleHz
points[] {t, hz, rocofHzPerS} - the whole trajectory, t in seconds from the first sample

The bridge wire messages

The full protocol narrative is 09-gda-integration; this is the field reference. All shapes bind through IO/BridgeProtocol.cs; the server is GDA Applications/GridSim-Bridge/bridge_server.py.

Client -> server (plain JSON commands): {action:"start", case, start, fps, source}, {action:"stop"}, {action:"set_speed", fps}, {action:"seek", start}, {action:"query", op, at, params, id}.

topology - {type, case, model}. model = the solve-export model block (ModelBlock: name, baseMva (<= 0 -> 100), buses[], generators[], branches[], optional boundaries[]). Wire DTO defaults: BusDto.vm = 1.0, x/y = 0 when omitted; GenDto.vg = 1.0, qmin/qmax/pmax = -/+inf/+inf, mBase = 100; BranchDto.tap = 1.0; BoundaryBranchDto.direction = 1.

state - {type, t, result, scalars, overlays, operator, market, source, estimate}:

  • result (StateResultBlock): converged?, iterations?, maxMismatch?, totalLossMw?, totalLossMvar?, buses[] (ResBusDto {id, vm=1.0, vaDeg, pInjMw, qInjMvar}), lineFlows[] (FlowDto, the solve-export flow shape; omitted name/ratingMva are backfilled from the cached topology).
  • scalars (ScalarsBlock, all nullable): freqHz, rocofHzPerS, genMw, demandMw, inertiaMwS, conditionIndex, rocofHeadroomHzS.
  • overlays (OverlaysBlock): stressByBranch (branch name -> fraction), conditionByBus (bus id -> fraction).
  • operator (OperatorBlock, all nullable): largestLossMw, responseMw, reserveMw, deratedMarginMw, netInterconnectorMw, synchronousInertiaGvaS, outturnInertiaGvaS, marketInertiaGvaS, warning, warnings[], balancingActions[] ({bmUnit, kind:"BID"|"OFFER", levelMw, priceGbpMwh, fuelType, synchronous}), constraints[] ({name, flowMw, limitMw, costGbp, binding}).
  • market (MarketBlock, all nullable): imbalancePriceGbpMwh, systemSellPriceGbpMwh, marketIndexGbpMwh, netImbalanceVolumeMwh, dayAheadGbpMwh, bsuosGbpMwh, constraintCostGbp, carbonIntensityGco2PerKwh, demandForecastMw, interconnectors[] ({name, flowMw (import +ve), capacityMw, priceGbpMwh}).
  • source: "live" | "presolved" | "estimated" (null from older servers).
  • estimate (EstimateBlock, source=estimated only): objectiveJ, chiThreshold, chiPassed, dof, measurementsN, removedN, iterations - camelCase keys mirroring the estimated_state lake columns.

t parses via ParseUtc (invariant-culture ISO-8601, assume/adjust UTC; empty -> DateTime.MinValue) and must clear the forward-inference horizon (169 h) with the client's 15-minute clock-skew allowance.

result - {type, op, id, at, payload}. id echoes the request's correlation id on every reply, including rejections. payload (QueryPayload): solveExport (a whole gridsim-solve-export/1 document, raw), verdict (including INFEASIBLE_HORIZON for a future at), bindingConstraint, violations[] ({kind, element, value, limit, message}); for op="asset": bmUnit, busId, and the four raw row lists envelope, bessSoc, tapWear, consumers (each a list of column->value maps, rendered generically); for op="propagation": corridors[] ({from, to, tau, km, vKmS, class}) and provenance - tau values that are negative, non-finite or

10 s are dropped at the protocol boundary (PropagationData).

info / error - {type, message} (NoteMessage). error faults the client stream; info is advisory. done - {type, framesSent[, source]}.

run-metadata.json

The machine-readable sidecar every CLI report drops next to its output (src/GridSim.Cli/RunInfo.cs:WriteJson), making runs comparable over time:

Field Content
timestampUtc ISO-8601 run instant
backend cpu / gpu (...) - the active solver backend
gitCommit Short commit baked into the assembly at build time
buildConfig Debug / Release
coreVersion GridSim.Core assembly version
runtime, arch, serverGc .NET runtime description, process architecture, GC mode
machineName, os, cpu, logicalCores, ramGb Host identity
run The run description string (verb + key parameters)

The same information heads every Markdown/CSV report as # -prefixed header lines (RunInfo.HeaderLines).

residuals.csv

Written by the estimate verb into the run directory (src/GridSim.Cli/Program.cs, estimate handler). The file opens with the RunInfo header lines (each prefixed # ), then:

kind,location,value,sigma,predicted,residual,normalized_residual,critical

One row per measurement in the final fit: kind (measurement kind name), location (Measurement.Describe() - bus 4 or branch 1->2 (from)), value (G17), sigma (G6), predicted (G17, h(x^) in engineering units), residual (E6), normalized_residual (F3; NaN for critical rows or under --lean on a passing fit), critical (True/False).

The GDA parquet products

All are weekly hive partitions (year=YYYY/week=WW/), written atomically (tmp + os.replace) with merge-on-write deduplication. Authoritative column definitions live in the GDA DataSchema.json; builders are cited per table.

Derived/estimated_state/<case>/.../estimated_state.parquet

One row per settlement period (GDA NetworkModel/build_estimated_state.py):

Column Type Content
datetime_utc datetime Period instant (UTC)
case string gb-spine / gb-full
status string EstimationStatus name
converged bool Only Converged rows carry a result block
iterations int Gauss-Newton iterations
max_state_update double Final max |deltax|
objective_j double WLS objective J
chi_threshold double chi-square(95 %) threshold (null when NaN - no redundancy)
chi_passed bool J <= threshold
dof int Degrees of freedom
measurements_n int Measurements in the set
removed_n int Bad-data removals
removed_json string [{kind, bus, rn}], null when none
vm_min, vm_max double Estimated voltage envelope
boundary_json string [{name, flowMw, limitMw}] - ETYS boundary transfers at the estimated state (signed sum of cut branches' from-end MW, the BoundaryAnalysis rule); null pre-column or without boundaries.json
result_json string Slim result block {buses:[{id,vm,vaDeg,pInjMw,qInjMvar}], lineFlows:[...]}; null when not converged

A per-case topology.json (solve-export model block + merged boundaries) sits beside the partitions; the bridge source:"estimated" path streams both.

Derived/state_divergence/<case>/.../state_divergence.parquet

Solve route vs estimate route, one row per period both lakes cover (build_state_divergence.py): datetime_utc, case, buses_compared, flows_compared, dvm_max_pu, dvm_mean_pu, dva_max_deg, dflow_max_mw, worst_vm_bus, worst_va_bus, worst_flow_from, worst_flow_to, est_objective_j, est_chi_passed, est_removed_n. Flows join on (from, to, name); buses on id.

Derived/estimation_closure/<case>/.../estimation_closure.parquet

One row per backfilled day (report_estimation_closure.py): date, case, periods, converged_n, chi_passed_n, removed_total, est_gen_gwh, est_load_gwh (from the converged rows' injections, MWh = MW/2 per period), nd_gwh (NESO national demand over the same window), load_vs_nd_pct, and outlier_league - the top repeat-removed measurements as "kind@busN xCount; ...". Estimated load derives from supplier-leg-anchored regional demand (net of embedded generation), so load_vs_nd_pct below 100 reflects that definitional gap as well as coverage.

Derived/gsp_generation_measurements/.../gsp_generation_measurements_YYYYMMDD.parquet

One row per (settlement period x GSP) plus a per-period gsp = NULL unallocated bucket (build_gsp_measurements.py): datetime_utc, settlement_date (Europe/London settlement day - convert with settlement_to_utc, never anchor at 00:00 UTC), settlement_period, gsp (NESO anchor name), generation_mw (2 x half-hourly MWh over the bound production units), sigma_mw (max(2 MW, 2 %)/min_confidence; NULL-bucket 30 %), units_n, min_confidence, provenance (observed = B1610 metering; pn = Physical Notification fallback, sigma 10 MW/10 %).

Derived/gsp_group_demand/.../gsp_group_demand_YYYYMMDD.parquet

One row per (period x PES group), B1610 supplier legs: datetime_utc, settlement_date, settlement_period, gsp_group (_A..._P), demand_mw (= -2 x sum quantity; net of embedded export), sigma_mw (max(10 MW, 3 %)), units_n, provenance. Not emitted on PN days.

Derived/gsp_ic_flows/.../gsp_ic_flows_YYYYMMDD.parquet

One row per (period x landing GSP): datetime_utc, settlement_date, settlement_period, gsp (landing GSP via the crosswalk's interconnector binding), ic_mw (net import, +ve, = 2 x sum signed MWh of the I_* units), units_n. Settlement-grade, 2019 onward.

See also

GDA integration Validation