Technical manual — chapter list

Provenance and invariants

GridSim is an evidence apparatus as much as a simulator, and it rests on two structural guarantees: every datum the evidence engine emits carries a typed provenance tag saying where it came from, and every dated input passes a forward-inference guard that makes the engine structurally incapable of nowcasting. This chapter documents both, the [VERIFY] text convention that accompanies placeholders, and where each invariant is enforced in code.

The ProvenanceTag taxonomy

src/GridSim.Core/Diagnostics/Provenance.cs:ProvenanceTag has seven members. Every record type in the evidence path carries one - a PowerStep, a RecordedAction, a RecordedOutcome, a whole GridEvent, every EvidenceColumn, the avoided-cost inputs - so the tag cannot be forgotten (the constructor demands it) and it survives serialisation, so a reviewer reading a gridsim-evidence/1 file sees the same provenance the engine saw.

Tag Meaning
Recorded directly from an archived record (NESO/GDA/published event log)
DerivedFromRecorded computed from recorded data (e.g. inertia inferred from a recorded RoCoF + deltaP)
CuratedPlaceholder a placeholder standing in for real data not yet sourced - always accompanied by [VERIFY]
Synthetic model-generated, not from any record (e.g. a synthetic demand curve for engine testing)
Counterfactual a counterfactual result (what a device would have done) - never an input to another replay
Interpolated gap-filled / augmented replay input (e.g. ML/MoE) - usable to improve fidelity, never as an event source
UnvalidatedAtScale a distributed-inertia-model result with no GB-scale reference (spatial RoCoF spread, propagation delay) - self-consistent, passes the reduction/invariant checks, but its absolute value at scale is not validated (awaits PMU/DFR field data) and must never be presented as validated

One-way flow rules

The tags are permissions, not just labels, and the flow is one-directional: data may only ever move down the trust ladder, never up.

  • Only Recorded (and its computed consequence, DerivedFromRecorded) may seed the forward integrator as a real event. Replaying a record is replay; anything else fed in as if recorded would be prediction dressed as history.
  • Counterfactual is a conclusion, never a cause: it is never an input to another replay. Feeding a counterfactual back in is exactly the circular reasoning the apparatus exists to prevent.
  • Interpolated and Synthetic data may improve replay fidelity (smoothing, resolution, engine testing) but must never define that an event happened or masquerade as a recorded input.
  • CuratedPlaceholder figures let the pipeline run end-to-end before every primary source is in hand, but the results they touch are not quotable until the placeholder is replaced with a cited figure.
  • UnvalidatedAtScale results may be shown (they are self-consistent physics) but always carry the tag; nothing downstream may launder them into a validated claim.
  • Nothing ever promotes itself: a value acquires a stronger tag only by being re-sourced, not by passing through a computation. The dossier illustrates the direction - the recorded column carries the event's own tag, the no-device baseline is hard-tagged DerivedFromRecorded, and the device column is hard-tagged Counterfactual (Evidence/EvidenceReport.cs:EvidenceReport.Build).

The [VERIFY] convention

[VERIFY] is a text convention paired with the CuratedPlaceholder tag, and the pairing is a biconditional: a curated placeholder always carries a [VERIFY] marker, and a [VERIFY] marker always flags a curated placeholder. The enum types the datum; the greppable literal names the debt in plain text - a reviewer can run one search across code, JSON and generated reports and see every figure still awaiting a primary source. The markers survive serialisation verbatim.

The built-in event catalogue shows both sides (Events/EventCatalog.cs:EventCatalog.Builtin):

  • gb-largest-loss-1320 is scaffolding: every figure tagged CuratedPlaceholder, every text field flagged [VERIFY] ("single largest secured infeed [VERIFY]").
  • gb-2019-08-09 shows markers being retired the only way the convention allows - by sourcing. Its losses cite the Ofgem investigation report paragraph by paragraph and are tagged Recorded; the two figures the record only implies (the 350-430 MW DG RoCoF-trip midpoint, the ~550 MW DG loss at the LFDD instant) are tagged DerivedFromRecorded, not dressed up as Recorded.

Other standing [VERIFY] figures: the avoided-cost VoLL default (£6,000/MWh, Evidence/AvoidedCost.cs:AvoidedCostInputs - a policy number that must be replaced with a cited figure before the result is quoted) and the GB inertia trajectory constants (Scenarios/ScenarioSpec.cs:SweepAxes.GbInertiaTrajectoryGvaS). A gated dossier's summary even names the debt: "Reproduce the record first ([VERIFY] the event figures/tolerances)."

The markers are an integrity ledger, not a defect: deleting a [VERIFY] would not make its figure sourced, only make it look sourced. Do not strip them; retire them by sourcing.

The forward-inference guard

src/GridSim.Core/Diagnostics/ForwardInferenceGuard.cs enforces the hard invariant: GridSim is a historical state estimator and may only process data at least HorizonHours old. Anything newer - recent or future - is rejected at the ingestion boundary with a ForwardInferenceException, so the engine cannot nowcast or predict by construction, not by policy. It can only ever lag reality.

public const double HorizonHours = 169.0;   // 7 days + 1 hour
  • Horizon(utcNow) - the newest instant the engine may process (utcNow - 169 h).
  • RejectFuture(stampUtc, utcNow?, what?) - throws when the stamp is newer than the horizon. An Unspecified-kind stamp is assumed UTC; an offset-carrying one is converted.
  • AssertHistorical(stamps, ...) - a whole dated series, first violation throws.

Why 169 hours

The horizon was widened from 25 h to 7 days + 1 hour (2026-07-14, owner decision), and the rationale is documented on the guard itself in two parts:

  1. Forward-inference defence in depth. A state at least a week old cannot be mistaken for - or repurposed as - a nowcast under any publication-lag or clock-skew argument: every data source the project consumes has fully published well inside a week, so the guard margin is no longer comparable to the slowest feed's lag.
  2. Reduced operational constraint space. A week-lagged state carries no operationally actionable information - unit commitment, outage windows, market positions and constraint patterns have all rolled over - so the system's entire output space sits outside the envelope an operational or market-sensitive reading could attach to.

The extra hour keeps a full settlement week strictly behind the boundary across DST transitions.

Every dated ingestion point

The guard is called at every boundary where a dated datum enters the engine. If you add a new dated input, call it there too - that is the convention that makes the invariant structural.

Ingestion point Call What is checked
Event catalogue load Events/EventCatalog.cs:EventCatalog.Validate (via Load, Get, Builtin) every event's WhenUtc - a catalogue cannot smuggle a future "event"
Measurement set load Estimation/MeasurementSet.cs:MeasurementSet.Load the set's TakenUtc (when present) - the estimator only reconstructs the past
NESO frequency CSV Frequency/NesoFrequencyCsv.cs:NesoFrequencyCsv.Read every row's timestamp as it streams - the reader can only read the past
GDA time-series load Gda/GdaTimeSeries.cs:GdaTimeSeries.Load every step's TimeUtc
Historical replay loop Simulation/GdaReplayLoop.cs:GdaReplayLoop.Replay re-asserts the whole series at the ingestion boundary, even though Load already checked it
Bridge state ticks Net/GridBridgeClient.cs:GridBridgeClient.HandleState each streamed state frame's timestamp, before any frame is built - a future tick never reaches a view; a 15-minute ClockSkewToleranceMinutes allowance absorbs modest client/server skew, beyond which the tick is a protocol violation and refused
Replay RMSE path Replay/ReplayValidator.cs:ReplayValidator inherits the per-row guard by reading the attached trace through NesoFrequencyCsv

The GDA-side horizon split

The GDA data layer enforces the same boundary from its side, in D:\Work\GDA\v1\Scripts\horizon.py - the single module every replay-facing GDA component imports. It deliberately distinguishes two horizons:

Horizon Value Applies to
cutoff() - the inference horizon now - (7 d + 1 h), HORIZON_DAYS = 7 + 1/24 everything replay/reconstruction-facing: builders, the bridge server, replay tools
ingest_cutoff() - the acquisition currency now - 1.05 d, INGEST_HORIZON_DAYS = 1.05 the raw ingest layer only

The split encodes a principle stated in the module: raw acquisition is not inference - fetching and storing public source data is not looking at it. The lake may pull sources up to ~25 h behind real time to stay current, but nothing on the inference side (measurement builders, state reconstruction, the bridge) may read past cutoff(). The inference value matches GridSim's ForwardInferenceGuard.HorizonHours = 169.0 exactly, and the module carries the same two-part widening rationale.

GDA-side enforcement points include horizon.assert_historical in the measurement builders (NetworkModel/build_gsp_measurements.py, build_estimator_measurements.py) and the GridSim bridge server (Applications/GridSim-Bridge/bridge_server.py), which rejects any replay start past the cutoff outright and masks every emitted tick to t <= cutoff() even when stored frames extend further.

No forward inference as a code invariant

Putting the two sides together: the rule "GridSim never nowcasts" is not a policy statement, it is the composition of guard calls at every boundary a dated datum can cross -

  • catalogue load - a future event cannot enter the evidence pipeline (EventCatalog.Validate);
  • bridge state ticks - a too-recent streamed frame is refused before a SimFrame is even built (GridBridgeClient.HandleState), and the GDA server masks its output to the same cutoff, so the wire is guarded at both ends;
  • measurement load - telemetry newer than the horizon cannot be estimated (MeasurementSet.Load), and the GDA builders refuse to author such a set in the first place;
  • replay - the historical replay loop re-asserts the horizon on the whole series at ingestion (GdaReplayLoop.Replay), and the NESO trace reader guards every sample it yields.

Most paths are therefore guarded twice - once where the data is authored (GDA side) and once where it is consumed (GridSim side) - so a single missed check never opens the boundary. The provenance tags complete the picture: the guard keeps the future out, and the tags keep non-recorded data from masquerading as the recorded past. Together they are what lets an evidence dossier claim, structurally rather than rhetorically, that everything in it is a replay of settled history.

See also

The evidence engine GDA integration