Architecture: the data path, built and designed
Each part below carries its state. Built (local) runs in this repo today, with tests. Design is written down and not built. Nothing here runs in AWS, and nothing at SFB is connected.
The path
flowchart LR
subgraph Site["On site"]
S["Sensors and systems<br/>PLCs, BMS, maker clouds,<br/>retrofit sensors, cameras"]
R["Local relay unit<br/>read, buffer, sign, forward<br/>(design; simulated in tools/relay.mjs)"]
end
subgraph Collector["Collector (built, local)"]
C1["POST /collect<br/>store original by sha256"]
C2["verify, classify, check,<br/>dedupe, order"]
C3["enrich: bind the sensor,<br/>target snapshot"]
C4["trap: triggers name events"]
end
subgraph Interchange["Interchange (built, local)"]
Q["queue<br/>(local; Kinesis in the design)"]
P["rule path selection"]
L["log, history, action, alert,<br/>ticket, work order, analysis"]
T["twin: asset JSON"]
end
subgraph Out["Outside systems (stubs locally)"]
J["Jira Service Management"]
I["Simplify i3"]
A["email, SMS, chat"]
end
W["Warehouse and analytics<br/>(design)"]
S --> R --> C1 --> C2 --> C3 --> C4 --> Q --> P --> L --> T
L --> J
L --> I
L --> A
Q -.-> W -.->|"results back as events"| C1
MS["MissionSight catalogue<br/>event and sensor types"] -.-> C2
MS -.-> C4
The same path in words:
- Sensor to relay. The relay reads each sensor and system in its own protocol. It buffers when the link drops, signs each message, and forwards it.
- Relay to the Collector. One endpoint takes every form: JSON, webhooks, files, raw bodies. The original is kept untouched under its sha256 before anything else happens.
- Classify and check. The source's rules, kept in data, name a type in MissionSight's catalogue. The fields and the unit are checked against that type. Nothing is converted or guessed.
- Bind. The device is bound to its sensor asset. The event's asset is the sensor's target, taken from the sensor's snapshot, and the sensor is its source. No asset system is called per reading.
- Trap. Each arriving item is evaluated against the triggers. A match becomes a named event. An item that traps nothing still goes on to history.
- Rule path selection. The event's name and metadata choose its path. The path names the policy, the action, who is told, the systems and the timing. The choice and its reasons are written on the event.
- Act and show. The listeners act on the chosen paths: alert, ticket, work order (after a person confirms it), analysis. Then the twin's asset JSON is updated.
Built (local)
| Part | Where | Tests |
|---|
Collector stages, in the order data/pipeline.json names | src/collector.mjs | test/bhs.test.mjs |
| Sensor binding and the target snapshot | src/sensors.mjs, data/sensors/ | test/sensors.test.mjs |
| Trapping: value, trend, interval trend, area, image, AI, other, time | src/triggers.mjs, data/triggers/ | test/trapping.test.mjs |
| Rule path selection and its precedence | src/paths.mjs, data/paths/ | test/paths.test.mjs |
| The Interchange's listeners and the local queue | src/interchange.mjs | every test |
| Retention per point: full or summarised history | src/interchange.mjs (history) | test/trapping.test.mjs |
| The twin's asset JSON | src/twin.mjs | test/kone.test.mjs |
| The relay simulator: read, buffer, sign, forward | tools/relay.mjs | test/trapping.test.mjs |
| Thresholds from the Mission Studio's export, usable rows only | src/thresholds.mjs, tools/fetch-thresholds.mjs | test/interval.test.mjs |
| The snapshot refresh, run by hand | tools/refresh-snapshots.mjs | test/sensors.test.mjs |
Precedence in path selection
Several paths may match one event. Among the exclusive paths that match, one is chosen, in this order:
- the path the trigger that raised the event pins
- the more specific path: one point per criterion it names, one more for naming the event type exactly
- the higher priority
- the path id, alphabetically, so the choice never depends on file order
Every additive path that matches runs as well (the out-of-hours path tells the on-call manager beside the chosen path). The event carries path_selection: the paths chosen, why each was chosen, and why each other path was not.
Design: the local relay unit
Not built. tools/relay.mjs simulates its four steps so the tests and the demo run the real path.
- Read. One reader per protocol the site uses: BACnet/IP, Modbus TCP and RTU, OPC UA, EtherNet/IP tag reads, MQTT, and the makers' cloud APIs. Each reader maps a point to its device name, which the Collector binds to a sensor asset.
- Buffer. A disk queue, oldest first. When the link drops, the relay keeps reading and holds every message. It forwards in order when the link returns. When the disk nears full, alarms and states are kept ahead of routine readings, and the relay says so in its own health event. It never drops a message without recording it.
- Sign. Each message is signed with the relay's key (HMAC-SHA256 today). The key lives in the unit's secure store, never in a file. A device certificate for mutual TLS is the next step.
- Forward. HTTPS POST to
/collect, outbound only, with a message id per message so a repeat is a duplicate. - Health. The relay reports its own state as a sensor asset would: offline, buffer depth, clock drift.
- With the Authority's IT. Where the unit sits, its network segment, outbound-only firewall rules, who patches it, and the read-only rule for every connection. The sensor catalogue research names market examples of such units (HMS Ewon Flexy 205; Inductive Automation Ignition Edge, $945 to $1,950 per its price page). None is chosen.
Design: the AWS stream
Not built. It matches what the Fleet World shows running and planned:
- the fleet's own ingestion: Lambda
stellarview-fleet-lake-ingestion writing to the S3 bucket stellarview-fleet-space-lake (stellarview/walkthroughs/fleet-3d/data/fleet.json, read 2026-11-07) - MissionSight's event stream on Kinesis Data Streams (
missionsight/docs/backbone.md)
| Local today | In AWS |
|---|
POST /collect on this machine | API Gateway HTTP API, with a Lambda authorizer that checks the relay's signature |
| The Collector in one process | Lambda for JSON messages; a Fargate service for files, or a presigned S3 upload for anything over API Gateway's request size limit |
var/originals/<sha256> | An S3 originals bucket with Object Lock, keyed by sha256 |
| The local queue and outbox | Kinesis Data Streams, partitioned by the target asset id, so one asset's events stay in order |
| The listeners in one process | One consumer per listener on the stream, so a slow desk never holds the alert |
var/twin/<asset>.json | The twin's store, read by the Exploder |
| Stub Jira, stub i3, stub channels | The Jira and i3 adapters with credentials from the active galaxy, and the alert channels |
Airfieldpath's cost model chose AWS IoT Core into SQS for gateway messages, on cost (stellarview/walkthroughs/fleet-3d/data/parts/products-drone.json). Kinesis is proposed here because the path needs order per asset, several consumers on one stream, and replay after a fix. That choice is open, and the cost model should price both before a pilot.
Design: processing, cleansing and the warehouse
Not built.
- Processing on the stream. The trap and path selection run in the stream consumer, from the same code and the same data files as the local Collector. The catalogue is the vendored MissionSight copy, named by its commit.
- Cleansing. What the Collector does now, kept on the stream: a wrong unit is refused, a repeat is a duplicate, a late reading is flagged, and the source's quality flag is kept. Nothing is filled in.
- Loading the warehouse. Kinesis Data Firehose writes events to the data lake in S3 as Parquet, partitioned by date, system and asset. Glue catalogues the tables, and Athena queries them. This matches the fleet lake's S3 shape. Redshift is the step after, if query load needs it.
- Retention per point. Each sensor asset sets its retention. A trend point is kept in full from the first day; a trend that was never captured cannot be analysed later. Other points are summarised per interval (count, minimum, maximum, sum, last). Locally,
history holds the full events and state.json holds the summaries.
Design: analytics back to the twin
Not built, except the repeat-fault rule, which runs locally.
- Scheduled jobs over the warehouse compute trends, MTBF, MTTR, availability and the predictions.
- Each result comes back as an event: a derived type in MissionSight's catalogue, posted to the Collector like any other source. It is trapped, routed and shown on the twin the same way.
- A result that proposes work proposes it. A person confirms before anything reaches the CMMS.
Design: the snapshot refresh job
Built as a tool run by hand (tools/refresh-snapshots.mjs); the schedule is a design.
- On a schedule. EventBridge Scheduler runs it daily, and the run reports how many snapshots changed.
- On asset change. The CMMS adapter emits an asset-changed message when the asset system reports a change. The job then refreshes only the sensors that observe that asset.
- Age shown. Every event shows its snapshot's age (
snapshot_age_s), so a stale binding is visible on the event itself.