fix(e2e): headless TkAgg crash + readiness false-positive on forwards

Run 10536 reached pytest but every connect was reset: the simulator died at
import with 'Tkinter is needed' — lib/gui.py calls matplotlib.use("TkAgg")
unconditionally (an explicit use() overrides MPLBACKEND=Agg), and the port
probe passed instantly because it was accepting the localhost forwarders, not
the nodes.

- run_e2e.sh: patch meshtasticator lib/gui.py TkAgg->Agg before boot;
  probe readiness on the upstream host (docker daemon) when forwards are
  active; docker pull meshtastic/meshtasticd up front so the pull is off the
  simulator's node-boot path.
- e2e tests: cancel leftover loop tasks (watchdog) at loop teardown.

plugin/tests/e2e/test_meshtasticator_e2e.py

@@ -143,6 +143,16 @@ def loop():
     try:
         yield _loop
     finally:
+        # Cancel leftover tasks (e.g. a watchdog started by a failed connect
+        # whose disconnect teardown never ran) before closing the loop.
+        async def _shutdown():
+            for task in asyncio.all_tasks(_loop):
+                task.cancel()
+
+        try:
+            asyncio.run_coroutine_threadsafe(_shutdown(), _loop).result(timeout=5)
+        except Exception:
+            pass
         _loop.call_soon_threadsafe(_loop.stop)
         thread.join(timeout=5)
 

scripts/meshtasticator_e2e/README.md

@@ -81,7 +81,15 @@ connections (Meshtasticator assigns node *n* → port `4404 + n`), then runs
 control connections assume nodes listen on localhost, so the script starts a
 `tcp_forward.py` per node port (localhost → daemon host) before booting the
 simulator. Nothing is forwarded when the daemon is local (unset/unix
-`DOCKER_HOST`).
+`DOCKER_HOST`). Readiness is probed on the *upstream* host (the daemon), never
+on the forwarders themselves.
+
+Other headless adaptations the script makes automatically:
+* patches `lib/gui.py` `matplotlib.use("TkAgg")` → `"Agg"` in the pinned
+  Meshtasticator checkout — the TkAgg call crashes headless runs even under
+  `MPLBACKEND=Agg` (only the interactive GUI mode is lost),
+* warms the `meshtastic/meshtasticd` image (`docker pull`) before booting the
+  simulator so the pull is not on the simulator's node-boot critical path.
 
 Without `MESHTASTICATOR_E2E=1` the e2e module skips, so ordinary unit/CI runs
 stay fast and gateway-free.

scripts/meshtasticator_e2e/run_e2e.sh

@@ -104,6 +104,26 @@ fi
 if [ -n "$DOCKER_TARGET" ]; then
   echo "== remote docker daemon detected ($DOCKER_TARGET): starting localhost port forwards =="
   start_forwards
+  PROBE_HOST="$DOCKER_TARGET"
+else
+  PROBE_HOST="$HOST"
+fi
+
+# The simulator forces TkAgg at import (lib/gui.py: matplotlib.use("TkAgg")),
+# which crashes headless runs ("Tkinter is needed") even under MPLBACKEND=Agg —
+# an explicit use() overrides the env var. Neutralize it in the pinned
+# checkout; only the interactive GUI mode (never used by this harness) is lost.
+if grep -q 'matplotlib.use("TkAgg")' "$SIM_DIR/lib/gui.py"; then
+  echo "== patching meshtasticator lib/gui.py: TkAgg -> Agg (headless) =="
+  sed -i.bak 's/matplotlib.use("TkAgg")/matplotlib.use("Agg")/' "$SIM_DIR/lib/gui.py"
+fi
+
+if [ "$MODE" = docker ]; then
+  # Warm the node image on the (possibly remote) daemon first: the simulator
+  # starts node processes ~4s after the container is created and control
+  # connections must succeed quickly, so the pull cannot sit in that path.
+  echo "== warming meshtastic/meshtasticd image =="
+  docker pull meshtastic/meshtasticd
 fi
 
 echo "== booting Meshtasticator ($MODE) with $NODES node(s) from $SIM_DIR =="
@@ -131,7 +151,9 @@ trap cleanup EXIT
 echo "== waiting for $NODES node TCP API port(s) to accept connections =="
 READY=""
 for _ in $(seq 1 180); do   # up to ~5 min: docker pulls meshtastic/meshtasticd on first run
-  READY="$("$VENV/bin/python" - "$HOST" "$NODES" <<'PY' || true
+  # Probe the upstream node host (the docker daemon when forwards are active,
+  # localhost otherwise) — probing the forwarders themselves would always pass.
+  READY="$("$VENV/bin/python" - "$PROBE_HOST" "$NODES" <<'PY' || true
 import socket, sys
 host, count = sys.argv[1], int(sys.argv[2])
 found = []
@@ -154,10 +176,10 @@ PY
   sleep 2
 done
 if [ -z "$READY" ]; then
-  echo "error: no Meshtasticator node ports became reachable on $HOST (see $SIM_LOG)" >&2
+  echo "error: no Meshtasticator node ports became reachable on $PROBE_HOST (see $SIM_LOG)" >&2
   exit 1
 fi
-echo "   node ports: $READY"
+echo "   node ports: $READY (probed on $PROBE_HOST)"
 
 # Ports accept before the node firmware finished booting inside the container;
 # give meshtasticd a moment to open its TCP API for real.