feat(e2e): Meshtasticator radio-level validation harness for the adapter #2

2026-09-08T03:56:47Z

eric commented

2026-09-08 03:56:47 +00:00

What

Radio-level counterpart to the contract suite: an end-to-end harness that drives hermes-meshtastic against live Meshtasticator simulated nodes (real MeshtasticD instances speaking the actual TCP/protobuf API on port 4404+), so the adapter is validated against the device-software/mesh layer without deploying or needing physical radios.

Why

The contract tests (plugin/tests/contract) guarantee adapter ↔ Hermes-gateway conformance but cannot exercise the radio path that previously failed at runtime. This fills that gap in the development loop:

Layer	Covered by
Adapter ↔ Hermes gateway API	contract suite
Adapter ↔ real MeshtasticD TCP session	this PR
Inbound text across the simulated LoRa mesh	this PR
Chunked outbound sends arriving at a peer node	this PR
True RF (silicon LoRa, propagation, duty cycle)	physical radio only (out of scope even for Meshtasticator)

What's in the PR

scripts/meshtasticator_e2e/run_e2e.sh — boots N Meshtasticator nodes (Docker mode or Linux-native MeshtasticD via --program), waits for their TCP API ports, runs the suite, tears down (incl. docker container cleanup).
scripts/meshtasticator_e2e/README.md — architecture, prerequisites, run instructions, status/open items.
plugin/tests/e2e/test_meshtasticator_e2e.py — skip-gated tests (MESHTASTICATOR_E2E=1): adapter connects to node 0 and receives a peer-node broadcast as a real gateway MessageEvent; adapter's >180-char send is chunked and the final chunk arrives at the peer.

Ordinary unit/CI runs are unaffected: the e2e module skips unless explicitly enabled, and no new CI job is added yet (needs a docker-enabled runner + a slow first-run firmware build — tracked as an open item).

Validation status

Structurally validated locally: module skips cleanly without the env flag (3 passed / 2 skipped in the plain unit run); with MESHTASTICATOR_E2E=1 + real gateway + meshtastic lib it progresses to a genuine adapter.connect() attempt (fails only because no simulator is running here — no Docker/Linux MeshtasticD on this machine).
A full green run still needs to happen on a Linux/Docker host with a MeshtasticD build. Expected iteration points (documented in README): deterministic in-range node placement (out/nodeConfig.yaml), channel alignment if mirroring production's channel-1 setup, and per-node lifecycle control to exercise the adapter's watchdog reconnect.

## What Radio-level counterpart to the contract suite: an end-to-end harness that drives `hermes-meshtastic` against live **Meshtasticator** simulated nodes (real MeshtasticD instances speaking the actual TCP/protobuf API on port 4404+), so the adapter is validated against the device-software/mesh layer without deploying or needing physical radios. ## Why The contract tests (`plugin/tests/contract`) guarantee adapter ↔ Hermes-gateway conformance but cannot exercise the radio path that previously failed at runtime. This fills that gap in the development loop: | Layer | Covered by | |---|---| | Adapter ↔ Hermes gateway API | contract suite | | Adapter ↔ real MeshtasticD TCP session | **this PR** | | Inbound text across the simulated LoRa mesh | **this PR** | | Chunked outbound sends arriving at a peer node | **this PR** | | True RF (silicon LoRa, propagation, duty cycle) | physical radio only (out of scope even for Meshtasticator) | ## What's in the PR - `scripts/meshtasticator_e2e/run_e2e.sh` — boots N Meshtasticator nodes (Docker mode or Linux-native MeshtasticD via `--program`), waits for their TCP API ports, runs the suite, tears down (incl. docker container cleanup). - `scripts/meshtasticator_e2e/README.md` — architecture, prerequisites, run instructions, status/open items. - `plugin/tests/e2e/test_meshtasticator_e2e.py` — skip-gated tests (`MESHTASTICATOR_E2E=1`): adapter connects to node 0 and receives a peer-node broadcast as a real gateway `MessageEvent`; adapter's >180-char send is chunked and the final chunk arrives at the peer. Ordinary unit/CI runs are unaffected: the e2e module skips unless explicitly enabled, and no new CI job is added yet (needs a docker-enabled runner + a slow first-run firmware build — tracked as an open item). ## Validation status - Structurally validated locally: module skips cleanly without the env flag (3 passed / 2 skipped in the plain unit run); with `MESHTASTICATOR_E2E=1` + real gateway + meshtastic lib it progresses to a genuine `adapter.connect()` attempt (fails only because no simulator is running here — no Docker/Linux MeshtasticD on this machine). - A full green run still needs to happen on a Linux/Docker host with a MeshtasticD build. Expected iteration points (documented in README): deterministic in-range node placement (`out/nodeConfig.yaml`), channel alignment if mirroring production's channel-1 setup, and per-node lifecycle control to exercise the adapter's watchdog reconnect.

eric added 2 commits 2026-09-08 04:06:07 +00:00

feat(e2e): Meshtasticator radio-level validation harness for the adapter a4abb5d15d

Adds a skip-gated end-to-end suite (plugin/tests/e2e) that drives the
adapter against live Meshtasticator simulated nodes - real MeshtasticD
instances speaking the actual TCP/protobuf API - covering the layer no
contract test can reach: TCP handshake, inbound text over the simulated
mesh, and chunked outbound sends arriving at a peer node.

scripts/meshtasticator_e2e/run_e2e.sh boots N nodes (Docker or Linux
native MeshtasticD), discovers their TCP API ports, and runs the suite.
Radio-level reconnect and CI wiring are documented as open items pending
a Linux/Docker validation run.

ci(e2e): on-demand Meshtasticator e2e job for the docker-enabled runner 4485ec3814

Adds .gitea/workflows/meshtasticator-e2e.yaml (workflow_dispatch until
validated green, pull_request trigger commented out): builds the pinned
Hermes contract env like release.yaml, installs the plugin with deps,
checks out Meshtasticator at a pinned commit, installs the simulator's
own (conflicting-version) requirements in an isolated venv, then runs
scripts/meshtasticator_e2e/run_e2e.sh --mode docker.

Supporting changes:
- run_e2e.sh: SIM_VENV support; automatic localhost->daemon TCP forwards
  (scripts/meshtasticator_e2e/tcp_forward.py) when DOCKER_HOST is a tcp://
  daemon, because Meshtasticator hardcodes localhost for its node control
  connections; node-boot settle wait; fixed docker cleanup (container
  'Meshtastic', volume 'Meshtasticator').
- e2e tests: connect via a module-long event loop in a background thread
  (adapter requires a running loop for watchdog + inbound dispatch, so
  per-call asyncio.run was wrong) with retry while the node TCP API boots.
- README: sim-venv guidance, remote-daemon forwarding, CI section.

Docker access follows the deployment examples in
eric/actions-docker (docker is available to runner jobs).

eric added 1 commit 2026-09-08 04:30:03 +00:00

fix(ci): make run_e2e.sh executable; invoke via bash in workflow 58b18d6260

First dispatch (run 10535) failed instantly with Permission denied on the
orchestrator: the file lost its executable bit in the branch move. Set mode
100755 and call through bash so a future mode regression cannot break the job.

eric added 1 commit 2026-09-08 04:33:50 +00:00

fix(e2e): headless TkAgg crash + readiness false-positive on forwards ba31d79368

Run 10536 reached pytest but every connect was reset: the simulator died at
import with 'Tkinter is needed' — lib/gui.py calls matplotlib.use("TkAgg")
unconditionally (an explicit use() overrides MPLBACKEND=Agg), and the port
probe passed instantly because it was accepting the localhost forwarders, not
the nodes.

- run_e2e.sh: patch meshtasticator lib/gui.py TkAgg->Agg before boot;
  probe readiness on the upstream host (docker daemon) when forwards are
  active; docker pull meshtastic/meshtasticd up front so the pull is off the
  simulator's node-boot path.
- e2e tests: cancel leftover loop tasks (watchdog) at loop teardown.

eric added 1 commit 2026-09-08 04:41:51 +00:00

fix(e2e): headless Graph construction + keep simulator cmdloop alive eff352b2ea

Run 10537 reached node boot but the simulator crashed again at Graph
construction: lib/gui.py move_figure() unconditionally touches the Tk window
manager (canvas.manager.window), which does not exist under Agg. Patch it to a
no-op guard alongside the TkAgg->Agg backend swap. Also feed the simulator's
interactive cmdloop from a FIFO held open until teardown — CI stdin is EOF, so
the sim would otherwise exit immediately after booting the nodes.

eric added 1 commit 2026-09-08 04:46:27 +00:00

fix(e2e): peer sendText moved to Interface; add RF failure diagnostics 6125130e6a

Run 10538 booted the full simulator and ran pytest — both failures were
test-side: the installed meshtastic client moved sendText from Node to the
Interface, and the outbound chunked message never arrived at the peer (cause
still unknown). Fix the sendText call and add in-run diagnostics so the next
failure is self-explaining:

- e2e: outbound-timeout failure now reports what arrived at the peer and the
  simulator log tail; sim now boots with -v (meshtastic debug logging).
- run_e2e.sh: on pytest failure in docker mode, dump per-node meshtasticd
  logs (meshtasticator-nodes.log) before cleanup removes the container.

eric added 1 commit 2026-09-08 04:50:28 +00:00

fix(e2e): instrument sim RF path; graceful shutdown; richer failure logs cbef4635cb

Both e2e directions time out: nothing is delivered between nodes although
the adapter connects fine and bridging code is equivalent to meshtastic's own
firmware_harness. Before concluding, instrument the pinned meshtasticator:
- on_receive now prints every observed packet (topic keys) to sim.log,
- a [sim-text] listener prints any text packet any sim interface sees,
- cleanup sends 'exit' through the cmdloop (graceful, no EOF-spam flood that
  was drowning the log tail),
- workflow failure step prints 400 sim-log lines + the node logs dump.

eric added 1 commit 2026-09-08 04:53:33 +00:00

ci(e2e): run simulator with current meshtastic client, not the stale 2.6.1 pin 2198741132

Meshtasticator's requirements.txt pins meshtastic~=2.6.1, which predates the
meshtastic/meshtasticd image it drives; with the old client the simulator
never observed node TX packets (both e2e directions timed out). The official
meshtastic firmware_harness uses the current client against the same
simulator-mode meshtasticd, so install the sim deps with meshtastic floating
instead of the pinned requirements file.

eric added 1 commit 2026-09-08 04:56:51 +00:00

fix(e2e): deterministic in-range node scenario + log receiver decisions b3063239f7

The sim now observes TX from both nodes (current client fixed observation)
but nothing is ever delivered to the other node. Suspect: random placement
leaving nodes out of range, or the receiver decision. Provide a --from-file
scenario with nodes spaced 10 m apart (guaranteed adjacency) and print the
transmitter -> receivers decision per TX in the sim log.

eric commented

2026-09-08 05:03:31 +00:00

Progress summary — Meshtasticator e2e validation attempt

Status: the harness, CI job, and full simulator boot are working; inter-node RF delivery through Meshtasticator's interactive sim is not, and the loop has been paused pending a runtime decision. Nothing here is merged or deployed; all work is on this branch (b306323, 9 commits over main).

What now works (proven on the CI runner, runs 10535–10542)

Full docker-enabled CI job: pinned Hermes contract env (mirrors release.yaml), plugin installed with deps, Meshtasticator pinned at 17ceb823, isolated sim venv, then scripts/meshtasticator_e2e/run_e2e.sh --mode docker.
Headless simulator boot. The interactive sim is Tk-bound in three ways; all are patched in the pinned checkout by run_e2e.sh:
1. lib/gui.py matplotlib.use("TkAgg") at import → "Agg" (crash: Tkinter is needed),
2. move_figure() unconditional canvas.manager.window access at Graph construction → guarded no-op (crash: FigureManagerBase has no attribute 'window'),
3. stdin EOF exits the sim's cmdloop immediately in CI → fed from a FIFO held open until teardown (exit is sent through it for a clean shutdown).
Remote-daemon support: this deployment's runner shares docker at DOCKER_HOST=tcp://docker.local:2375. Meshtasticator hardcodes localhost for its node connections, so run_e2e.sh starts one tcp_forward.py (localhost → daemon) per node port, and readiness is probed on the upstream daemon host — not on the forwarders (which would always pass).
The adapter itself: connects to the real MeshtasticD node in the simulator, and the sim observes the adapter's and the peer's TX on the wire.
Simulator node image warm-up (docker pull meshtastic/meshtasticd) so the pull is off the sim's boot path.

The blocker

Neither direction delivers a message between two nodes: the adapter (node 0) never receives the peer's broadcast, and chunked sends never arrive at the peer. Findings so far:

Run	Simulator state	Observation
10537	crashed at Graph construction (Tk)	fixed
10538	booted, pytest ran	both tests fail on client-API drift (`Node.sendText` → `Interface.sendText`) + delivery
10539	booted, verbose	inbound fixed; still no delivery either way; sim log drowned by teardown EOF spam
10540	booted, instrumented	sim log empty — sim blocked mid-init (reconnect race)
10541	booted, current client	sim observed TX from node 1 (`port=4405`) and node 0 (`port=4404`) — but no delivery
10542	booted, deterministic 10 m topology (`--from-file`)	zero TX observed, empty log — run-to-run non-determinism

Root-cause candidates, in order of suspicion:

Meshtasticator's interactive sim is maintained against old assumptions; its node reconnect dance (reconnect_nodes: close + reconnect every interface) is racy and sometimes strands the sim before subscriptions are live.
Its python client pin (meshtastic~=2.6.1 in requirements.txt) predates the meshtastic/meshtasticd image; run 10541 only started observing TX after the sim venv was switched to the current client.
Its forwarding matches the design of meshtastic's official firmware_harness (SimMesh, ships inside meshtastic/tests/) — same SIMULATOR_APP bridge — so the concept is sound; the maintenance/robustness is the gap.

Open decision (paused per request)

How to get a green e2e run:

A. Pivot the runtime to meshtastic's official firmware_harness.SimMesh (recommended): extract meshtasticd from the already-pulled image, run it natively on the Linux runner, deterministic topology, no docker port games, no GUI, maintained upstream. Keep the pytest module, adapter-under-test, and CI job shape.
B. Keep debugging Meshtasticator's interactive sim — possible, but upstream races/assumptions may cap progress and each cycle costs minutes.
C. Park the CI job and ship the scaffold + docs as local-only until validated on a Linux box.

Branch contents (all under `scripts/meshtasticator_e2e/`, `plugin/tests/e2e/`, `.gitea/workflows/meshtasticator-e2e.yaml`)

Skip-gated e2e pytest module (adapter ↔ live MeshtasticD TCP session, inbound broadcast, chunked outbound).
Orchestrator with headless patches, remote-daemon forwarding, image warm-up, deterministic topology file, per-node log capture on failure, graceful teardown.
On-demand CI job (workflow_dispatch; PR trigger commented out until green), concurrency guard.

Validation of the contract harness (separate concern, already merged to main in 310c024) is unaffected: 8 passed locally and in CI.

## Progress summary — Meshtasticator e2e validation attempt **Status: the harness, CI job, and full simulator boot are working; inter-node RF delivery through Meshtasticator's interactive sim is not, and the loop has been paused pending a runtime decision.** Nothing here is merged or deployed; all work is on this branch (`b306323`, 9 commits over `main`). ### What now works (proven on the CI runner, runs 10535–10542) - Full docker-enabled CI job: pinned Hermes contract env (mirrors `release.yaml`), plugin installed with deps, Meshtasticator pinned at `17ceb823`, isolated sim venv, then `scripts/meshtasticator_e2e/run_e2e.sh --mode docker`. - Headless simulator boot. The interactive sim is Tk-bound in three ways; all are patched in the pinned checkout by `run_e2e.sh`: 1. `lib/gui.py` `matplotlib.use("TkAgg")` at import → `"Agg"` (crash: *Tkinter is needed*), 2. `move_figure()` unconditional `canvas.manager.window` access at Graph construction → guarded no-op (crash: *FigureManagerBase has no attribute 'window'*), 3. stdin EOF exits the sim's `cmdloop` immediately in CI → fed from a FIFO held open until teardown (`exit` is sent through it for a clean shutdown). - Remote-daemon support: this deployment's runner shares docker at `DOCKER_HOST=tcp://docker.local:2375`. Meshtasticator hardcodes `localhost` for its node connections, so `run_e2e.sh` starts one `tcp_forward.py` (localhost → daemon) per node port, and readiness is probed on the *upstream* daemon host — not on the forwarders (which would always pass). - The adapter itself: **connects to the real MeshtasticD node** in the simulator, and the sim observes the adapter's and the peer's TX on the wire. - Simulator node image warm-up (`docker pull meshtastic/meshtasticd`) so the pull is off the sim's boot path. ### The blocker Neither direction delivers a message between two nodes: the adapter (node 0) never receives the peer's broadcast, and chunked sends never arrive at the peer. Findings so far: | Run | Simulator state | Observation | |---|---|---| | 10537 | crashed at Graph construction (Tk) | fixed | | 10538 | booted, pytest ran | both tests fail on client-API drift (`Node.sendText` → `Interface.sendText`) + delivery | | 10539 | booted, verbose | inbound fixed; still no delivery either way; sim log drowned by teardown EOF spam | | 10540 | booted, instrumented | sim log empty — sim blocked mid-init (reconnect race) | | 10541 | booted, current client | **sim observed TX from node 1 (`port=4405`) and node 0 (`port=4404`)** — but no delivery | | 10542 | booted, deterministic 10 m topology (`--from-file`) | zero TX observed, empty log — run-to-run non-determinism | Root-cause candidates, in order of suspicion: 1. Meshtasticator's interactive sim is maintained against old assumptions; its node **reconnect dance** (`reconnect_nodes`: close + reconnect every interface) is racy and sometimes strands the sim before subscriptions are live. 2. Its python client pin (`meshtastic~=2.6.1` in `requirements.txt`) predates the `meshtastic/meshtasticd` image; run 10541 only started observing TX after the sim venv was switched to the current client. 3. Its forwarding matches the design of meshtastic's **official** `firmware_harness` (`SimMesh`, ships inside `meshtastic/tests/`) — same SIMULATOR_APP bridge — so the concept is sound; the maintenance/robustness is the gap. ### Open decision (paused per request) How to get a green e2e run: - **A. Pivot the runtime to meshtastic's official `firmware_harness.SimMesh`** (recommended): extract `meshtasticd` from the already-pulled image, run it natively on the Linux runner, deterministic topology, no docker port games, no GUI, maintained upstream. Keep the pytest module, adapter-under-test, and CI job shape. - **B. Keep debugging Meshtasticator's interactive sim** — possible, but upstream races/assumptions may cap progress and each cycle costs minutes. - **C. Park the CI job** and ship the scaffold + docs as local-only until validated on a Linux box. ### Branch contents (all under `scripts/meshtasticator_e2e/`, `plugin/tests/e2e/`, `.gitea/workflows/meshtasticator-e2e.yaml`) - Skip-gated e2e pytest module (adapter ↔ live MeshtasticD TCP session, inbound broadcast, chunked outbound). - Orchestrator with headless patches, remote-daemon forwarding, image warm-up, deterministic topology file, per-node log capture on failure, graceful teardown. - On-demand CI job (`workflow_dispatch`; PR trigger commented out until green), `concurrency` guard. Validation of the **contract harness** (separate concern, already merged to `main` in `310c024`) is unaffected: `8 passed` locally and in CI.

eric closed this pull request

2026-09-08 06:12:10 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: eric/meshtastic#2