ci(e2e): on-demand Meshtasticator e2e job for the docker-enabled runner

Adds .gitea/workflows/meshtasticator-e2e.yaml (workflow_dispatch until validated green, pull_request trigger commented out): builds the pinned Hermes contract env like release.yaml, installs the plugin with deps, checks out Meshtasticator at a pinned commit, installs the simulator's own (conflicting-version) requirements in an isolated venv, then runs scripts/meshtasticator_e2e/run_e2e.sh --mode docker. Supporting changes: - run_e2e.sh: SIM_VENV support; automatic localhost->daemon TCP forwards (scripts/meshtasticator_e2e/tcp_forward.py) when DOCKER_HOST is a tcp:// daemon, because Meshtasticator hardcodes localhost for its node control connections; node-boot settle wait; fixed docker cleanup (container 'Meshtastic', volume 'Meshtasticator'). - e2e tests: connect via a module-long event loop in a background thread (adapter requires a running loop for watchdog + inbound dispatch, so per-call asyncio.run was wrong) with retry while the node TCP API boots. - README: sim-venv guidance, remote-daemon forwarding, CI section. Docker access follows the deployment examples in eric/actions-docker (docker is available to runner jobs).
2026-09-07 21:04:56 -07:00
parent a4abb5d15d
commit 4485ec3814
5 changed files with 306 additions and 23 deletions
@@ -0,0 +1,90 @@
+name: meshtasticator-e2e
+
+# Radio-level end-to-end validation against Meshtasticator's interactive
+# simulator (real MeshtasticD per node + simulated LoRa PHY). Runs on demand
+# until it is validated green on the runner; afterwards switch `on` to also
+# trigger on pull_request touching plugin/ and this workflow.
+#
+# Docker access follows the deployment examples in
+# https://git.ericxliu.me/eric/actions-docker (docker is available to jobs);
+# Meshtasticator's docker mode publishes node TCP API ports (4404+n) from the
+# meshtastic/meshtasticd container and the simulator connects to them on
+# localhost. scripts/meshtasticator_e2e/run_e2e.sh starts localhost->daemon
+# TCP forwards automatically when DOCKER_HOST is a tcp:// daemon, which is how
+# this job is configured.
+
+on:
+  workflow_dispatch:
+  # pull_request:
+  #   paths:
+  #     - 'plugin/**'
+  #     - 'scripts/meshtasticator_e2e/**'
+  #     - '.gitea/workflows/meshtasticator-e2e.yaml'
+
+concurrency:
+  # One e2e run at a time: Meshtasticator hardcodes the container/volume name
+  # ("Meshtastic"/"Meshtasticator") on the shared docker daemon.
+  group: meshtasticator-e2e
+  cancel-in-progress: true
+
+jobs:
+  e2e:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      DOCKER_HOST: tcp://docker.local:2375
+      MESHTASTICATOR_COMMIT: 17ceb8231079d87b070abc6132181e4c6b20202d
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.13 (matches Hermes image)
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.13'
+
+      - name: Install uv
+        run: pip install uv
+
+      # --- Hermes gateway contract env (mirrors .gitea/workflows/release.yaml) ---
+      - name: Clone pinned Hermes gateway source (deployed image tag)
+        run: |
+          git clone --depth 1 --branch v2026.8.31 \
+            https://github.com/NousResearch/hermes-agent.git plugin/.hermes-src
+          test "$(git -C plugin/.hermes-src rev-parse HEAD)" = \
+            "29112bef099274229cadff79cdff7bf7b99c4b77"
+
+      - name: Editable install hermes-agent + pytest + plugin (meshtastic client lib)
+        run: |
+          uv venv --python 3.13 plugin/.venv-contract
+          uv pip install --python plugin/.venv-contract/bin/python -e plugin/.hermes-src
+          uv pip install --python plugin/.venv-contract/bin/python pytest
+          uv pip install --python plugin/.venv-contract/bin/python -e plugin
+
+      # --- Simulator env (isolated: meshtasticator pins meshtastic~=2.6.1) ---
+      - name: Clone Meshtasticator at pinned commit
+        run: |
+          git clone https://github.com/meshtastic/meshtasticator.git "$HOME/meshtasticator"
+          git -C "$HOME/meshtasticator" checkout "$MESHTASTICATOR_COMMIT"
+          test "$(git -C "$HOME/meshtasticator" rev-parse HEAD)" = "$MESHTASTICATOR_COMMIT"
+
+      - name: Simulator venv (its own requirements + docker SDK)
+        run: |
+          uv venv --python 3.13 "$HOME/sim-venv"
+          uv pip install --python "$HOME/sim-venv/bin/python" \
+            -r "$HOME/meshtasticator/requirements.txt" docker
+
+      - name: Run Meshtasticator e2e suite
+        run: |
+          export MESHTASTICATOR_DIR="$HOME/meshtasticator"
+          export SIM_VENV="$HOME/sim-venv"
+          export E2E_VENV="plugin/.venv-contract"
+          scripts/meshtasticator_e2e/run_e2e.sh --mode docker
+
+      - name: Print simulator log (on failure)
+        if: failure()
+        run: |
+          echo "===== meshtasticator-sim.log ====="
+          tail -n 200 meshtasticator-sim.log 2>/dev/null || true
+          echo "===== docker ps ====="
+          docker ps -a 2>/dev/null || true
@@ -126,19 +126,60 @@ def adapter():


@pytest.fixture(scope="module")
-def node0(adapter):
-    """Connect the adapter to simulated node 0; disconnect afterwards."""
-    async def _connect():
-        return await adapter.connect()
+def loop():
+    """Long-lived event loop in a background thread.

-    assert asyncio.run(_connect()) is True, (
+    The adapter mirrors the Hermes gateway contract: connect()/watchdog/
+    handle_message() assume a *running* loop (the gateway's). A per-call
+    asyncio.run() would close the loop right after connect() and strand both
+    the watchdog and inbound dispatch, so the e2e module keeps one loop alive
+    and drives coroutines into it with run_coroutine_threadsafe.
+    """
+    import threading
+
+    _loop = asyncio.new_event_loop()
+    thread = threading.Thread(target=_loop.run_forever, daemon=True)
+    thread.start()
+    try:
+        yield _loop
+    finally:
+        _loop.call_soon_threadsafe(_loop.stop)
+        thread.join(timeout=5)
+
+
+def _run_in_loop(loop, coro, timeout_s=CONNECT_TIMEOUT_S):
+    import concurrent.futures
+
+    try:
+        return asyncio.run_coroutine_threadsafe(coro, loop).result(timeout=timeout_s)
+    except concurrent.futures.TimeoutError:
+        pytest.fail(f"coroutine did not finish within {timeout_s}s")
+
+
+@pytest.fixture(scope="module")
+def node0(adapter, loop):
+    """Connect the adapter to simulated node 0 (retrying while the node's TCP
+    API comes up); disconnect afterwards."""
+    import time
+
+    deadline = time.monotonic() + CONNECT_TIMEOUT_S
+    last_ok = False
+    while time.monotonic() < deadline:
+        last_ok = bool(_run_in_loop(loop, adapter.connect()))
+        if last_ok:
+            break
+        time.sleep(3)
+    assert last_ok, (
        f"adapter failed to connect to simulated node at {E2E_HOST}:{NODE0_PORT} "
        "(is the simulator running and the node reachable?)"
    )
    try:
        yield adapter
    finally:
-        asyncio.run(adapter.disconnect())
+        try:
+            _run_in_loop(loop, adapter.disconnect(), timeout_s=10)
+        except Exception:  # loop teardown must not mask test failures
+            pass


 def _wait_for(predicate, timeout_s=15.0, interval_s=0.5, what="condition"):
@@ -187,7 +228,7 @@ def test_adapter_receives_broadcast_from_peer(node0):
        peer.close()


-def test_adapter_chunked_send_reaches_peer(node0):
+def test_adapter_chunked_send_reaches_peer(node0, loop):
    """A long outbound message must be chunked and arrive at the peer node."""
    import time

@@ -200,9 +241,7 @@ def test_adapter_chunked_send_reaches_peer(node0):
    def _on_receive(packet, interface):
        decoded = packet.get("decoded", {}) or {}
        text = decoded.get("text", "")
-        if text and text.startswith("(1/"):  # our chunked framing
-            received.append(text)
-        elif text:
+        if text:
            received.append(text)

    peer = tcp_interface.TCPInterface(hostname=E2E_HOST, portNumber=NODE1_PORT)
@@ -214,7 +253,7 @@ def test_adapter_chunked_send_reaches_peer(node0):
            result = await node0.send("channel_e2e", content)
            assert result.success, f"adapter.send failed: {result.error}"

-        asyncio.run(_send())
+        _run_in_loop(loop, _send(), timeout_s=60)

        def _all_chunks_arrived():
            # Adapter frames multi-part sends as "(i/n) <chunk>"; the final
@@ -22,9 +22,12 @@ simulated node.
 ```
 scripts/meshtasticator_e2e/
  run_e2e.sh                  orchestrator: boots N nodes, runs pytest, tears down
+  tcp_forward.py              localhost->daemon TCP forward (remote docker daemons)
  README.md                   this file
 plugin/tests/e2e/
  test_meshtasticator_e2e.py  the e2e assertions (skip-gated)
+.gitea/workflows/
+  meshtasticator-e2e.yaml     on-demand CI job (docker-enabled runner)
 ```

 ## Prerequisites
@@ -38,10 +41,20 @@ plugin/tests/e2e/
  (`-e plugin` installs the meshtastic client library the tests also use.)
 * A clone of Meshtasticator: `git clone https://github.com/meshtastic/meshtasticator`
 * Runtime for the simulator — pick one:
-  * **Docker** (simplest, works on macOS/Windows): the simulator builds
-    MeshtasticD from a container image on first run (slow once).
+  * **Docker** (simplest, works on macOS/Windows): the simulator pulls the
+    `meshtastic/meshtasticd` image on first run (slow once). Node TCP API ports
+    are published on your docker host's localhost, which is what the simulator
+    itself expects.
  * **Linux native**: build MeshtasticD's `native` PlatformIO target and pass
    the build dir with `--program`.
+* Optional but recommended: an isolated simulator venv, because Meshtasticator
+  pins old client libs (`meshtastic~=2.6.1` in its requirements.txt):
+  ```bash
+  uv venv --python 3.13 /tmp/sim-venv
+  uv pip install --python /tmp/sim-venv/bin/python \
+    -r <meshtasticator>/requirements.txt docker
+  export SIM_VENV=/tmp/sim-venv     # else the script uses plain `python3`
+  ```

 ## Run

@@ -64,9 +77,26 @@ connections (Meshtasticator assigns node *n* → port `4404 + n`), then runs
 `pytest plugin/tests/e2e` with `MESHTASTICATOR_E2E=1`. Simulator output lands in
 `meshtasticator-sim.log` at the repo root.

+**Remote docker daemon** (`DOCKER_HOST=tcp://host:2375`): the simulator's own
+control connections assume nodes listen on localhost, so the script starts a
+`tcp_forward.py` per node port (localhost → daemon host) before booting the
+simulator. Nothing is forwarded when the daemon is local (unset/unix
+`DOCKER_HOST`).
+
 Without `MESHTASTICATOR_E2E=1` the e2e module skips, so ordinary unit/CI runs
 stay fast and gateway-free.

+## CI
+
+`.gitea/workflows/meshtasticator-e2e.yaml` runs the whole flow on a
+docker-enabled runner (on demand via `workflow_dispatch` until validated, then
+flip on the commented `pull_request` trigger): it builds the Hermes contract
+env (pinned like `release.yaml`), installs the plugin with deps, clones
+Meshtasticator at a pinned commit, installs the simulator's own requirements in
+an isolated venv, and calls `run_e2e.sh --mode docker`. Simulator logs print on
+failure. Only one e2e run at a time (the sim hardcodes the `Meshtastic`
+container/volume name on the shared daemon).
+
 ## What the tests do

 1. **Inbound**: peer node 1 broadcasts text; the adapter (connected to node 0)
@@ -95,5 +125,9 @@ items are expected to need iteration there:
  lifecycle control (native mode subprocesses; docker mode currently runs all
  nodes in one container), so the adapter's watchdog reconnect is **not** yet
  exercised here. See the watchdog unit coverage in the adapter for now.
-* **CI wiring**: an e2e job needs a docker-enabled runner and tolerates a long
-  first-run firmware build — deliberately left out of this PR.
+* **CI validation run**: `.gitea/workflows/meshtasticator-e2e.yaml` is drafted
+  and on-demand only; it still needs a first green run on the docker-enabled
+  runner (topology assumptions: docker reachable from the job, node ports on
+  the job's localhost via the automatic tcp forwards). Trigger it via
+  `workflow_dispatch`, watch the sim log step, and only then enable the
+  `pull_request` trigger.
@@ -15,6 +15,12 @@
 #     select 'native') and pass --program <dir containing binary 'program'>.
 #   * uv + python 3.13 for the contract venv (plugin/.venv-contract).
 #
+# Docker on a remote daemon: if DOCKER_HOST points at a tcp:// daemon, the
+# simulator's own control connections assume nodes listen on localhost, so
+# this script starts lightweight localhost->daemon TCP forwarders for the node
+# ports first (scripts/meshtasticator_e2e/tcp_forward.py). Nothing is
+# forwarded when the daemon is local (unset/unix DOCKER_HOST).
+#
 # Usage
 #   MESHTASTICATOR_DIR=/path/to/meshtasticator scripts/meshtasticator_e2e/run_e2e.sh
 #   # docker mode (default):     ... run_e2e.sh --mode docker
@@ -24,9 +30,11 @@
 # Environment
 #   MESHTASTICATOR_DIR  path to a clone of github.com/meshtastic/meshtasticator
 #   E2E_VENV            python env with hermes-agent + plugin deps installed
-#                       (default: plugin/.venv-contract — set it up with
-#                       `uv pip install -e plugin/.hermes-src` equivalents via
-#                       scripts/contract-test.sh, then `uv pip install -e plugin`)
+#                       (default: plugin/.venv-contract — see scripts/contract-test.sh,
+#                       then `uv pip install --python plugin/.venv-contract/bin/python -e plugin`)
+#   SIM_VENV            optional python env holding the simulator's deps
+#                       (meshtasticator/requirements.txt + `pip install docker`);
+#                       defaults to plain `python3`.
 set -euo pipefail

 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
@@ -37,6 +45,11 @@ PROGRAM=""
 CHANNEL_INDEX=0
 HOST=127.0.0.1
 VENV="${E2E_VENV:-$REPO_ROOT/plugin/.venv-contract}"
+if [ -n "${SIM_VENV:-}" ]; then
+  SIM_PYTHON="$SIM_VENV/bin/python"
+else
+  SIM_PYTHON="${SIM_PYTHON:-python3}"
+fi

 while [ $# -gt 0 ]; do
  case "$1" in
@@ -55,6 +68,30 @@ command -v uv >/dev/null 2>&1 || { echo "uv required" >&2; exit 1; }

 export MPLBACKEND=Agg   # headless matplotlib (no display needed)

+# ---------------------------------------------------------------------------
+# Remote-daemon port forwarding: Meshtasticator talks to nodes on localhost.
+# ---------------------------------------------------------------------------
+DOCKER_TARGET=""
+if [[ "${DOCKER_HOST:-}" == tcp://* ]]; then
+  DOCKER_TARGET="${DOCKER_HOST#tcp://}"
+  DOCKER_TARGET="${DOCKER_TARGET%%:*}"
+  if [ "$DOCKER_TARGET" = "127.0.0.1" ] || [ "$DOCKER_TARGET" = "localhost" ]; then
+    DOCKER_TARGET=""
+  fi
+fi
+FWD_PIDS=()
+PORT_BASE=4404  # matches meshtasticator lib/interactive.py TCP_PORT_OFFSET
+
+start_forwards() {
+  for ((i = 0; i < NODES; i++)); do
+    port=$((PORT_BASE + i))
+    "$VENV/bin/python" "$REPO_ROOT/scripts/meshtasticator_e2e/tcp_forward.py" \
+      --listen "127.0.0.1:$port" --target "$DOCKER_TARGET:$port" >/dev/null 2>&1 &
+    FWD_PIDS+=("$!")
+  done
+  sleep 1
+}
+
 SIM_LOG="$REPO_ROOT/meshtasticator-sim.log"
 SIM_ARGS=("$NODES")
 if [ "$MODE" = native ]; then
@@ -64,28 +101,36 @@ else
  SIM_ARGS+=(-d)
 fi

+if [ -n "$DOCKER_TARGET" ]; then
+  echo "== remote docker daemon detected ($DOCKER_TARGET): starting localhost port forwards =="
+  start_forwards
+fi
+
 echo "== booting Meshtasticator ($MODE) with $NODES node(s) from $SIM_DIR =="
 (
  cd "$SIM_DIR"
-  python3 interactiveSim.py "${SIM_ARGS[@]}" >"$SIM_LOG" 2>&1
+  "$SIM_PYTHON" interactiveSim.py "${SIM_ARGS[@]}" >"$SIM_LOG" 2>&1
 ) &
 SIM_PID=$!

-PORTS=""
 cleanup() {
  echo "== tearing down simulator (pid $SIM_PID) =="
  kill "$SIM_PID" 2>/dev/null || true
  wait "$SIM_PID" 2>/dev/null || true
+  for pid in "${FWD_PIDS[@]:-}"; do
+    kill "$pid" 2>/dev/null || true
+  done
  if [ "$MODE" = docker ]; then
-    # The sim container may outlive the process on abrupt kills.
-    docker ps -q --filter "ancestor=meshtastic-meshtasticd" 2>/dev/null | xargs -r docker rm -f >/dev/null 2>&1 || true
+    # The sim names its container/volume (auto_remove only fires on stop).
+    docker rm -f Meshtastic >/dev/null 2>&1 || true
+    docker volume rm Meshtasticator >/dev/null 2>&1 || true
  fi
 }
 trap cleanup EXIT

 echo "== waiting for $NODES node TCP API port(s) to accept connections =="
 READY=""
-for _ in $(seq 1 180); do   # up to ~5 min: docker builds firmware on first run
+for _ in $(seq 1 180); do   # up to ~5 min: docker pulls meshtastic/meshtasticd on first run
  READY="$("$VENV/bin/python" - "$HOST" "$NODES" <<'PY' || true
 import socket, sys
 host, count = sys.argv[1], int(sys.argv[2])
@@ -114,6 +159,11 @@ if [ -z "$READY" ]; then
 fi
 echo "   node ports: $READY"

+# Ports accept before the node firmware finished booting inside the container;
+# give meshtasticd a moment to open its TCP API for real.
+echo "== settling (node firmware boot) =="
+sleep 15
+
 echo "== running radio-level e2e tests =="
 (
  cd "$REPO_ROOT"
@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""Minimal TCP forwarder used by run_e2e.sh when the Docker daemon is remote.
+
+Meshtasticator's interactive simulator always connects to its simulated nodes
+at localhost:<port> (lib/interactive.py init_communication). On a Gitea
+Actions runner whose Docker daemon is remote (e.g. DOCKER_HOST=tcp://docker.local:2375)
+the container-published node ports live on the *daemon* host, not the job's
+localhost — so we bind 127.0.0.1:<port> in the job and forward to the daemon
+host, restoring the localhost view the simulator (and our e2e tests) expect.
+
+Usage: tcp_forward.py --listen 127.0.0.1:4404 --target docker.local:4404
+"""
+
+import argparse
+import socket
+import threading
+
+
+def _forward(src: socket.socket, dst: socket.socket):
+    try:
+        while True:
+            data = src.recv(65536)
+            if not data:
+                break
+            dst.sendall(data)
+    except OSError:
+        pass
+    finally:
+        try:
+            dst.shutdown(socket.SHUT_WR)
+        except OSError:
+            pass
+
+
+def _handle(client: socket.socket, target_addr):
+    try:
+        upstream = socket.create_connection(target_addr, timeout=10)
+    except OSError:
+        client.close()
+        return
+    t = threading.Thread(target=_forward, args=(client, upstream), daemon=True)
+    t.start()
+    _forward(upstream, client)
+    client.close()
+    upstream.close()
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--listen", required=True, help="local bind, e.g. 127.0.0.1:4404")
+    ap.add_argument("--target", required=True, help="forward target, e.g. docker.local:4404")
+    args = ap.parse_args()
+
+    lhost, lport = args.listen.rsplit(":", 1)
+    thost, tport = args.target.rsplit(":", 1)
+    target_addr = (thost, int(tport))
+
+    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+    listener.bind((lhost, int(lport)))
+    listener.listen(16)
+    print(f"forwarding {args.listen} -> {args.target}", flush=True)
+
+    while True:
+        client, _ = listener.accept()
+        threading.Thread(target=_handle, args=(client, target_addr), daemon=True).start()
+
+
+if __name__ == "__main__":
+    main()