Interposing live telemetry

The interpose verb lets GridSim ingest a live SCADA/RTU feed. It listens for protocol frames, corrects and decodes them, maps each point onto your network model, and writes them as estimator windows and — optionally — a GDA-shaped archive. This chapter covers authoring a crosswalk, watching a feed, writing windows, feeding the estimator, and the horizon rule you will meet immediately.

For the concepts behind it, see the overview; for the internals, the technical manual.

What you need

A protocol feed GridSim can reach over TCP. Run gridsim interpose --list-protocols to see the active stacks (IEC 60870-5-104 as iec104, Modbus TCP as modbus).
A network case (a built-in like case14, or a --dir model) whose bus and branch ids your telemetry points refer to.
A crosswalk file mapping each protocol point address to a measurement.

Authoring a crosswalk

Protocol addresses (an IEC-104 information-object address, a Modbus register) mean nothing to GridSim on their own — you tell it which address is which measurement in a gridsim-rtu-pointmap/1 file:

{
  "schema": "gridsim-rtu-pointmap/1",
  "case": "case14",
  "baseMva": 100.0,
  "bindings": [
    { "address": "IOA:1001",  "kind": "VmPu",    "bus": 4,           "sigma": 0.01 },
    { "address": "IOA:2001",  "kind": "PFlowMw", "from": 1, "to": 2, "sigma": 2.0  },
    { "address": "MB:U1:3:0", "kind": "VmPu",    "bus": 5, "scale": 0.001, "sigma": 0.01 }
  ]
}

address is the protocol-native key the decoder emits: IOA:<n> for IEC-104, MB:U<unit>:<fc>:<index> for Modbus. Run in --monitor first (below) to see exactly what addresses your feed produces.
kind is the measurement: VmPu, VaDeg, PInjMw/PFlowMw, QInjMvar/QFlowMvar, IMagPu, IAngDeg. Bus kinds take bus; branch kinds take from/to (and circuit to disambiguate parallels).
sigma is the measurement's standard deviation — its weight in the estimator. Trust a channel more with a smaller sigma.
scale recovers engineering units from raw registers (a Modbus register of 1030 × 0.001 → 1.030 pu). baseKv/baseAmps convert kV/A feeds to per-unit; datumAngleDeg re-references an angle feed onto GridSim's slack.

An address with no binding is decoded but skipped — that is normal.

Watching a feed (monitor mode)

The quickest way to confirm a feed and discover its addresses is --monitor, which decodes to the console and writes nothing:

gridsim interpose case14 --protocol iec104 --listen 0.0.0.0:2404 \
        --crosswalk map.json --monitor

Each decoded point prints its address, value, folded quality and device time:

interpose IEC 60870-5-104: listening 0.0.0.0:2404, case 'case14', 3 binding(s)
monitor mode: decoding to console only, no windows written.
  IOA:1001                   1.05  Good         -
  IOA:1002                   0.98  Good         2026-07-20T13:45:12.5000000Z

Press any key to stop; or add --ticks N to stop after N frames (for scripts).

Writing estimator windows

Drop --monitor and add --measurements-dir to buffer points into windows and write each as a gridsim-measurements/1 file:

gridsim interpose case14 --protocol iec104 --listen 0.0.0.0:2404 \
        --crosswalk map.json --measurements-dir ./windows --window-ms 1000

--window-ms sets the aggregation bucket (default 1000 ms). Each flushed window is one file named by its timestamp, ordinal-sortable so serve consumes them in order. The pending window is flushed on shutdown, so a short run keeps its last points.

Feeding the estimator

Point serve at the same directory to run the estimator over the windows:

gridsim serve case14 --measurements-dir ./windows --method wls

If your telemetry is present-time (a genuinely live feed), you will see this:

done: processed 0, converged 0, rejected-fresh 2, failed 0

That is correct — see the horizon rule next. To estimate over aged windows (device-timestamped history, or once your live windows are more than a week old), the same command processes them normally.

The forward-inference horizon

GridSim is a historical estimator: it will not process data newer than now minus 169 hours (7 days + 1 hour). Live telemetry is by definition newer than that, so the interposer never forces it into the estimator. Instead:

the interposer writes live windows now, with the horizon check deferred;
serve re-checks every window against the real clock and skips any that are still too recent — that is the rejected-fresh count.

A live window therefore becomes estimable only once it has aged past the horizon. If you want to prove the estimator end-to-end today, feed it a window whose points carry device timestamps at least eight days old, or advance serve's clock with --utc-now. Nothing you interpose can nowcast.

The quarantine archive and the GDA boundary

Add --out-store <dir> to also archive the telemetry as GDA-shaped Parquet:

gridsim interpose case14 --protocol iec104 --listen 0.0.0.0:2404 \
        --crosswalk map.json --measurements-dir ./windows --out-store ./store

This writes hive-partitioned Parquet (store/Parquet/rtu_live/year=YYYY/week=WW/*.parquet) with GDA's unified-table columns, so GDA tooling can read it. But it is deliberately not part of GDA's evidence lake: GDA is public-data-only and excludes live SCADA by rule. So the store tags every row source_class = "telemetry", drops a _QUARANTINE.json marker (evidential: false), and refuses to open inside the GDA lake (it errors if you point --out-store at a directory containing the lake's DataSchema.json). Never merge this store into GDA's unified tables.

Use --source-id to name the store (default rtu_live); --no-parquet skips this sink even when a store is configured.

Command summary

gridsim interpose [case] --protocol <name> (--listen host:port | --port N) --crosswalk <map.json>
                  [--measurements-dir <dir>] [--out-store <dir>] [--source-id <id>]
                  [--window-ms N] [--monitor] [--ticks N]
gridsim interpose --list-protocols

Flag	Effect
`--protocol`	select the stack (`--list-protocols` to enumerate)
`--listen` / `--port`	TCP endpoint to bind (host:port, or a port on all interfaces)
`--crosswalk`	the `gridsim-rtu-pointmap/1` address → measurement file (required)
`--monitor`	decode to console only; write no sinks
`--measurements-dir`	write `gridsim-measurements/1` windows for `serve`
`--out-store` / `--source-id`	write the quarantine GDA-shaped Parquet store
`--window-ms`	window aggregation bucket, ms (default 1000)
`--ticks N`	stop after N frames (headless); otherwise run until a keypress

Common faults

Symptom	Cause and fix
`error: unknown protocol 'x'. Try --list-protocols.`	typo, or the stack isn't wired yet — use a listed name
`error: --crosswalk <map.json> is required.`	every run needs a crosswalk; author one (above)
`error loading crosswalk ...`	the file isn't valid `gridsim-rtu-pointmap/1` — check `schema` and JSON
`error opening ... transport on ...`	the port is in use or the host isn't bindable — pick another `--listen`
decodes but `windows-written 0`	no address in your feed matched a binding, or every point failed quality — check `--monitor` output against your crosswalk
`serve` shows all `rejected-fresh`	the windows are present-time; this is the horizon rule working — age the data or use device timestamps
`refusing to write live telemetry into the GDA public lake`	`--out-store` points inside the GDA lake — choose a directory outside the evidence estate