171 lines
45 KiB
HTML
171 lines
45 KiB
HTML
<!doctype html><html lang=en><head><title>Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive · Eric X. Liu's Personal Page</title><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=color-scheme content="light dark"><meta name=author content="Eric X. Liu"><meta name=description content="
|
|
Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive
|
|
|
|
|
|
Link to heading
|
|
|
|
|
|
|
|
Introduction
|
|
|
|
|
|
Link to heading
|
|
|
|
|
|
Flashing NVIDIA Jetson devices remotely presents unique challenges when the host machine is virtualized. This article documents the technical challenges, failures, and eventual success of flashing a Jetson Orin Nano Super developer kit using NVIDIA SDK Manager in various virtualized environments, specifically focusing on QEMU/KVM virtual machines and LXC containers on Proxmox VE."><meta name=keywords content="software engineer,performance engineering,Google engineer,tech blog,software development,performance optimization,Eric Liu,engineering blog,mountain biking,Jeep enthusiast,overlanding,camping,outdoor adventures"><meta name=twitter:card content="summary"><meta name=twitter:title content="Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive"><meta name=twitter:description content="Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive Link to heading Introduction Link to heading Flashing NVIDIA Jetson devices remotely presents unique challenges when the host machine is virtualized. This article documents the technical challenges, failures, and eventual success of flashing a Jetson Orin Nano Super developer kit using NVIDIA SDK Manager in various virtualized environments, specifically focusing on QEMU/KVM virtual machines and LXC containers on Proxmox VE."><meta property="og:url" content="/posts/flashing-jetson-orin-nano-in-virtualized-environments/"><meta property="og:site_name" content="Eric X. Liu's Personal Page"><meta property="og:title" content="Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive"><meta property="og:description" content="Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive Link to heading Introduction Link to heading Flashing NVIDIA Jetson devices remotely presents unique challenges when the host machine is virtualized. This article documents the technical challenges, failures, and eventual success of flashing a Jetson Orin Nano Super developer kit using NVIDIA SDK Manager in various virtualized environments, specifically focusing on QEMU/KVM virtual machines and LXC containers on Proxmox VE."><meta property="og:locale" content="en"><meta property="og:type" content="article"><meta property="article:section" content="posts"><meta property="article:published_time" content="2025-10-02T00:00:00+00:00"><meta property="article:modified_time" content="2025-10-02T08:34:05+00:00"><link rel=canonical href=/posts/flashing-jetson-orin-nano-in-virtualized-environments/><link rel=preload href=/fonts/fa-brands-400.woff2 as=font type=font/woff2 crossorigin><link rel=preload href=/fonts/fa-regular-400.woff2 as=font type=font/woff2 crossorigin><link rel=preload href=/fonts/fa-solid-900.woff2 as=font type=font/woff2 crossorigin><link rel=stylesheet href=/css/coder.min.c8e4eea149ae1dc7c61ba9b0781793711a4e657f7e07a4413f9abc46d52dffc4.css integrity="sha256-yOTuoUmuHcfGG6mweBeTcRpOZX9+B6RBP5q8RtUt/8Q=" crossorigin=anonymous media=screen><link rel=stylesheet href=/css/coder-dark.min.a00e6364bacbc8266ad1cc81230774a1397198f8cfb7bcba29b7d6fcb54ce57f.css integrity="sha256-oA5jZLrLyCZq0cyBIwd0oTlxmPjPt7y6KbfW/LVM5X8=" crossorigin=anonymous media=screen><link rel=icon type=image/svg+xml href=/images/favicon.svg sizes=any><link rel=icon type=image/png href=/images/favicon-32x32.png sizes=32x32><link rel=icon type=image/png href=/images/favicon-16x16.png sizes=16x16><link rel=apple-touch-icon href=/images/apple-touch-icon.png><link rel=apple-touch-icon sizes=180x180 href=/images/apple-touch-icon.png><link rel=manifest href=/site.webmanifest><link rel=mask-icon href=/images/safari-pinned-tab.svg color=#5bbad5></head><body class="preload-transitions colorscheme-auto"><div class=float-container><a id=dark-mode-toggle class=colorscheme-toggle><i class="fa-solid fa-adjust fa-fw" aria-hidden=true></i></a></div><main class=wrapper><nav class=navigation><section class=container><a class=navigation-title href=/>Eric X. Liu's Personal Page
|
|
</a><input type=checkbox id=menu-toggle>
|
|
<label class="menu-button float-right" for=menu-toggle><i class="fa-solid fa-bars fa-fw" aria-hidden=true></i></label><ul class=navigation-list><li class=navigation-item><a class=navigation-link href=/posts/>Posts</a></li><li class=navigation-item><a class=navigation-link href=https://chat.ericxliu.me>Chat</a></li><li class=navigation-item><a class=navigation-link href=https://git.ericxliu.me/user/oauth2/Authenitk>Git</a></li><li class=navigation-item><a class=navigation-link href=https://coder.ericxliu.me/api/v2/users/oidc/callback>Coder</a></li><li class=navigation-item><a class=navigation-link href=/>|</a></li><li class=navigation-item><a class=navigation-link href=https://sso.ericxliu.me>Sign in</a></li></ul></section></nav><div class=content><section class="container post"><article><header><div class=post-title><h1 class=title><a class=title-link href=/posts/flashing-jetson-orin-nano-in-virtualized-environments/>Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive</a></h1></div><div class=post-meta><div class=date><span class=posted-on><i class="fa-solid fa-calendar" aria-hidden=true></i>
|
|
<time datetime=2025-10-02T00:00:00Z>October 2, 2025
|
|
</time></span><span class=reading-time><i class="fa-solid fa-clock" aria-hidden=true></i>
|
|
11-minute read</span></div></div></header><div class=post-content><h1 id=flashing-jetson-orin-nano-in-virtualized-environments-a-technical-deep-dive>Flashing Jetson Orin Nano in Virtualized Environments: A Technical Deep Dive
|
|
<a class=heading-link href=#flashing-jetson-orin-nano-in-virtualized-environments-a-technical-deep-dive><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h1><h2 id=introduction>Introduction
|
|
<a class=heading-link href=#introduction><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><p>Flashing NVIDIA Jetson devices remotely presents unique challenges when the host machine is virtualized. This article documents the technical challenges, failures, and eventual success of flashing a Jetson Orin Nano Super developer kit using NVIDIA SDK Manager in various virtualized environments, specifically focusing on QEMU/KVM virtual machines and LXC containers on Proxmox VE.</p><p><img src="http://localhost:4998/attachments/image-7c88938eaa4db1b7eafc437b9067b8790998fc71.png?client=default&bucket=obsidian" alt="S3 File"></p><h3 id=the-constraint-hypervisor-only-infrastructure>The Constraint: Hypervisor-Only Infrastructure
|
|
<a class=heading-link href=#the-constraint-hypervisor-only-infrastructure><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>This project operated under a specific constraint: the only available x86_64 machines were homelab servers running Proxmox VE as bare-metal hypervisors. There was no x86 laptop available, and the primary workstation was an Apple M4 Mac (ARM64 architecture incompatible with SDK Manager).</p><p>Installing SDK Manager directly on the Proxmox host OS was explicitly ruled out for several reasons:</p><ol><li><p><strong>Hypervisor Stability</strong>: The Proxmox hosts run critical infrastructure (Kubernetes clusters, Ceph storage, network services). Installing development tools and potentially conflicting dependencies directly on the hypervisor risks system stability.</p></li><li><p><strong>Dependency Conflicts</strong>: SDK Manager requires numerous dependencies (QEMU, specific Python versions, USB libraries) that could conflict with Proxmox’s carefully managed package versions.</p></li><li><p><strong>Clean Separation</strong>: Best practices dictate keeping hypervisor hosts minimal, with all workloads running in VMs or containers. This separation simplifies maintenance, updates, and disaster recovery.</p></li><li><p><strong>Repeatability</strong>: A solution confined to a VM or container can be easily replicated, backed up, and destroyed without affecting the host system.</p></li></ol><p>This constraint made the flashing process significantly more complex, as it required finding a virtualization method that could reliably handle the Jetson’s USB communication requirements without installing anything on the Proxmox host beyond standard virtualization features.</p><h2 id=background-jetson-flashing-requirements>Background: Jetson Flashing Requirements
|
|
<a class=heading-link href=#background-jetson-flashing-requirements><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><p>NVIDIA Jetson devices use a specialized flashing process that requires:</p><ol><li><strong>USB Connection</strong>: The device must be connected in Force Recovery Mode (APX mode, USB ID <code>0955:7523</code>)</li><li><strong>Initrd Flash Method</strong>: Modern Jetson devices boot a temporary Linux kernel over USB (<code>0955:7035</code>) that establishes USB networking</li><li><strong>USB Network Communication</strong>: The host system must establish network connectivity (typically <code>192.168.55.1</code> or IPv6 <code>fc00:1:1::1</code>) with the Jetson during the flash process</li><li><strong>SDK Manager</strong>: NVIDIA’s SDK Manager orchestrates the entire process, requiring specific kernel modules and capabilities</li></ol><p>The initrd flash method is particularly sensitive to timing and USB device handling, making it challenging in virtualized environments.</p><table><thead><tr><th>Method</th><th>USB Passthrough</th><th>Network Namespace</th><th>Timing Sensitivity</th><th>Result</th></tr></thead><tbody><tr><td>QEMU VM (device-level)</td><td>Emulated</td><td>VM-isolated</td><td>High latency</td><td>❌ Failed (USB timeout)</td></tr><tr><td>LXC Container</td><td>Host devices</td><td>Host namespace</td><td>Near-native</td><td>❌ Failed (network isolation)</td></tr><tr><td>QEMU VM (PCI-level)</td><td>Direct hardware</td><td>VM-isolated</td><td>Native</td><td>✅ Success</td></tr></tbody></table><h2 id=first-attempt-qemukvm-virtual-machine-with-usb-passthrough>First Attempt: QEMU/KVM Virtual Machine with USB Passthrough
|
|
<a class=heading-link href=#first-attempt-qemukvm-virtual-machine-with-usb-passthrough><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><h3 id=configuration>Configuration
|
|
<a class=heading-link href=#configuration><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>Given the constraint of not having an x86 laptop, initial attempts used a QEMU/KVM virtual machine running Ubuntu 22.04 x86_64 on an Apple M4 Mac via UTM (a QEMU frontend for macOS). This approach allowed running SDK Manager on an emulated x86_64 system while connecting the Jetson device via USB passthrough configured through UTM’s USB settings.</p><p>While this satisfied the requirement of having an x86_64 environment without using the Proxmox hosts, it introduced additional virtualization overhead as the entire x86_64 instruction set was being emulated on ARM64 hardware.</p><h3 id=issues-encountered>Issues Encountered
|
|
<a class=heading-link href=#issues-encountered><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>The flash process consistently failed during the USB communication phase with the error:</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-fallback data-lang=fallback><span style=display:flex><span>ERROR: might be timeout in USB write.
|
|
</span></span></code></pre></div><h3 id=root-cause-analysis>Root Cause Analysis
|
|
<a class=heading-link href=#root-cause-analysis><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>QEMU/KVM’s USB passthrough implementation has known reliability issues with complex USB protocols. The Jetson’s initrd flash process requires:</p><ol><li>Rapid USB re-enumeration when switching between recovery mode and initrd mode</li><li>High-throughput data transfer for writing the root filesystem</li><li>Bidirectional USB network communication with strict timing requirements</li></ol><p>Individual USB device passthrough in QEMU emulates USB at the device level, introducing latency and potential timing issues. The Jetson’s USB networking during initrd boot is particularly sensitive to these delays, causing the timeout errors.</p><h3 id=conclusion>Conclusion
|
|
<a class=heading-link href=#conclusion><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>This approach was abandoned due to fundamental limitations in QEMU’s USB emulation layer. USB passthrough at the device level is insufficient for the Jetson flash process.</p><h2 id=second-attempt-lxc-container-on-proxmox>Second Attempt: LXC Container on Proxmox
|
|
<a class=heading-link href=#second-attempt-lxc-container-on-proxmox><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><h3 id=rationale>Rationale
|
|
<a class=heading-link href=#rationale><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>After the Mac-based VM approach failed, attention shifted to the Proxmox infrastructure. LXC containers provide near-native performance with minimal virtualization overhead compared to full VMs. Unlike running SDK Manager directly on the Proxmox host (which was ruled out for stability reasons), an LXC container offers:</p><ol><li><strong>Isolation</strong>: Complete separation from the host OS with its own filesystem and process space</li><li><strong>Near-Native Performance</strong>: Containers share the host kernel, eliminating instruction emulation overhead</li><li><strong>Easy Management</strong>: Containers can be created, destroyed, and backed up without affecting the host</li><li><strong>USB Access</strong>: Proxmox supports passing USB devices to containers via cgroup device permissions</li></ol><p>The hypothesis was that an LXC container with proper USB device access would provide the necessary USB timing characteristics while maintaining the clean separation requirement.</p><h3 id=configuration-progression>Configuration Progression
|
|
<a class=heading-link href=#configuration-progression><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>The LXC container (ID 106, Ubuntu 22.04) required extensive configuration on the Proxmox host (<code>/etc/pve/lxc/106.conf</code>):</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Enable mknod capability for creating device nodes</span>
|
|
</span></span><span style=display:flex><span>features: <span style=color:#79c0ff>nesting</span><span style=color:#ff7b72;font-weight:700>=</span>1,mknod<span style=color:#ff7b72;font-weight:700>=</span><span style=color:#a5d6ff>1</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># USB device passthrough (Bus 003)</span>
|
|
</span></span><span style=display:flex><span>lxc.cgroup2.devices.allow: c 189:* rwm
|
|
</span></span><span style=display:flex><span>lxc.mount.entry: /dev/bus/usb/003 dev/bus/usb/003 none bind,optional,create<span style=color:#ff7b72;font-weight:700>=</span>dir <span style=color:#a5d6ff>0</span> <span style=color:#a5d6ff>0</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Loop device access for mounting disk images</span>
|
|
</span></span><span style=display:flex><span>lxc.cgroup2.devices.allow: b 7:* rwm
|
|
</span></span><span style=display:flex><span>lxc.mount.entry: /dev/loop0 dev/loop0 none bind,optional,create<span style=color:#ff7b72;font-weight:700>=</span>file <span style=color:#a5d6ff>0</span> <span style=color:#a5d6ff>0</span>
|
|
</span></span><span style=display:flex><span>lxc.mount.entry: /dev/loop1 dev/loop1 none bind,optional,create<span style=color:#ff7b72;font-weight:700>=</span>file <span style=color:#a5d6ff>0</span> <span style=color:#a5d6ff>0</span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># ... (loop2-7)</span>
|
|
</span></span><span style=display:flex><span>lxc.mount.entry: /dev/loop-control dev/loop-control none bind,optional,create<span style=color:#ff7b72;font-weight:700>=</span>file <span style=color:#a5d6ff>0</span> <span style=color:#a5d6ff>0</span>
|
|
</span></span></code></pre></div><h3 id=issues-encountered-and-resolutions>Issues Encountered and Resolutions
|
|
<a class=heading-link href=#issues-encountered-and-resolutions><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><h4 id=1-mknod-permission-errors>1. mknod Permission Errors
|
|
<a class=heading-link href=#1-mknod-permission-errors><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p><strong>Error</strong>: <code>mknod: .../rootfs/dev/random: Operation not permitted</code></p><p><strong>Cause</strong>: LXC containers lack <code>CAP_MKNOD</code> capability by default, required by L4T flash scripts to create device nodes in the rootfs.</p><p><strong>Solution</strong>: Enable <code>mknod=1</code> feature on the Proxmox host:</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span>pct set <span style=color:#a5d6ff>106</span> -features <span style=color:#79c0ff>nesting</span><span style=color:#ff7b72;font-weight:700>=</span>1,mknod<span style=color:#ff7b72;font-weight:700>=</span><span style=color:#a5d6ff>1</span>
|
|
</span></span></code></pre></div><h4 id=2-arm64-binary-execution>2. ARM64 Binary Execution
|
|
<a class=heading-link href=#2-arm64-binary-execution><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p><strong>Error</strong>: <code>chroot: failed to run command 'dpkg': Exec format error</code></p><p><strong>Cause</strong>: The L4T rootfs contains ARM64 binaries that cannot execute on x86_64 without emulation.</p><p><strong>Solution</strong>: Install and enable <code>qemu-user-static</code> and <code>binfmt-support</code> on the <strong>Proxmox host</strong> (not the container):</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span>apt-get install qemu-user-static binfmt-support
|
|
</span></span><span style=display:flex><span>update-binfmts --enable qemu-aarch64
|
|
</span></span></code></pre></div><h4 id=3-loop-device-access>3. Loop Device Access
|
|
<a class=heading-link href=#3-loop-device-access><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p><strong>Error</strong>: <code>losetup: cannot find an unused loop device</code></p><p><strong>Cause</strong>: The L4T flash scripts use loop devices to mount disk images. LXC containers don’t have loop device access by default.</p><p><strong>Solution</strong>: Add loop device permissions and mount entries to the container configuration.</p><h4 id=4-usb-networking-failure>4. USB Networking Failure
|
|
<a class=heading-link href=#4-usb-networking-failure><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p><strong>Error</strong>: <code>Device failed to boot to the initrd flash kernel</code></p><p><strong>Cause</strong>: This was the most complex issue. When the Jetson boots into initrd mode (<code>0955:7035</code>), it creates a USB network interface (<code>enx*</code> or <code>usb0</code>). However, in LXC containers, this interface appeared in the <strong>host’s network namespace</strong>, not the container’s namespace.</p><p><strong>Attempted Solution</strong>:</p><ol><li>Loaded USB networking kernel modules on the Proxmox host:</li></ol><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span>modprobe rndis_host cdc_ether cdc_ncm cdc_subset
|
|
</span></span><span style=display:flex><span>echo <span style=color:#a5d6ff>"rndis_host"</span> >> /etc/modules
|
|
</span></span><span style=display:flex><span>echo <span style=color:#a5d6ff>"cdc_ether"</span> >> /etc/modules
|
|
</span></span><span style=display:flex><span>echo <span style=color:#a5d6ff>"cdc_ncm"</span> >> /etc/modules
|
|
</span></span><span style=display:flex><span>echo <span style=color:#a5d6ff>"cdc_subset"</span> >> /etc/modules
|
|
</span></span></code></pre></div><ol start=2><li>Created udev rules to automatically move USB network interfaces to the container:</li></ol><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># /etc/udev/rules.d/99-jetson-usb-network.rules</span>
|
|
</span></span><span style=display:flex><span><span style=color:#79c0ff>ACTION</span><span style=color:#ff7b72;font-weight:700>==</span><span style=color:#a5d6ff>"add"</span>, <span style=color:#79c0ff>SUBSYSTEM</span><span style=color:#ff7b72;font-weight:700>==</span><span style=color:#a5d6ff>"net"</span>, <span style=color:#79c0ff>KERNEL</span><span style=color:#ff7b72;font-weight:700>==</span><span style=color:#a5d6ff>"enx*"</span>, <span style=color:#79c0ff>RUN</span><span style=color:#ff7b72;font-weight:700>+=</span><span style=color:#a5d6ff>"/usr/local/bin/handle-jetson-usb-network.sh %k"</span>
|
|
</span></span></code></pre></div><ol start=3><li>Created handler script to move interfaces into container namespace:</li></ol><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-weight:700;font-style:italic>#!/bin/bash
|
|
</span></span></span><span style=display:flex><span><span style=color:#8b949e;font-weight:700;font-style:italic></span><span style=color:#79c0ff>INTERFACE</span><span style=color:#ff7b72;font-weight:700>=</span><span style=color:#79c0ff>$1</span>
|
|
</span></span><span style=display:flex><span><span style=color:#79c0ff>CONTAINER_ID</span><span style=color:#ff7b72;font-weight:700>=</span><span style=color:#a5d6ff>106</span>
|
|
</span></span><span style=display:flex><span><span style=color:#79c0ff>CONTAINER_PID</span><span style=color:#ff7b72;font-weight:700>=</span><span style=color:#ff7b72>$(</span>pct exec <span style=color:#79c0ff>$CONTAINER_ID</span> -- pidof systemd | awk <span style=color:#a5d6ff>'{print $1}'</span><span style=color:#ff7b72>)</span>
|
|
</span></span><span style=display:flex><span>ip link set <span style=color:#a5d6ff>"</span><span style=color:#79c0ff>$INTERFACE</span><span style=color:#a5d6ff>"</span> netns <span style=color:#a5d6ff>"ct</span><span style=color:#79c0ff>$CONTAINER_ID</span><span style=color:#a5d6ff>"</span>
|
|
</span></span><span style=display:flex><span>pct exec <span style=color:#79c0ff>$CONTAINER_ID</span> -- ip link set dev <span style=color:#79c0ff>$INTERFACE</span> up
|
|
</span></span><span style=display:flex><span>pct exec <span style=color:#79c0ff>$CONTAINER_ID</span> -- dhclient <span style=color:#79c0ff>$INTERFACE</span>
|
|
</span></span></code></pre></div><h3 id=fundamental-lxc-limitation>Fundamental LXC Limitation
|
|
<a class=heading-link href=#fundamental-lxc-limitation><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>Despite all configuration efforts, the LXC container could not properly handle USB network interfaces due to <strong>network namespace isolation</strong>. LXC containers have separate network namespaces from the host, and moving USB network interfaces between namespaces proved unreliable and often failed to establish proper connectivity.</p><p>The initrd flash process has strict timing requirements:</p><ol><li>Jetson boots into initrd mode</li><li>USB network interface must appear and be configured within seconds</li><li>SSH connection must establish for flash commands</li></ol><p>Even when the interface was successfully moved to the container’s namespace, DHCP configuration often failed, causing the flash process to timeout.</p><h3 id=conclusion-1>Conclusion
|
|
<a class=heading-link href=#conclusion-1><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>LXC containers, despite their near-native performance, have fundamental limitations for this use case due to network namespace isolation. USB networking devices created dynamically during the flash process cannot be reliably handled.</p><h2 id=final-solution-proxmox-vm-with-pci-usb-controller-passthrough>Final Solution: Proxmox VM with PCI USB Controller Passthrough
|
|
<a class=heading-link href=#final-solution-proxmox-vm-with-pci-usb-controller-passthrough><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><h3 id=architecture-change>Architecture Change
|
|
<a class=heading-link href=#architecture-change><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><p>With both the Mac-based VM (due to QEMU USB emulation issues) and the LXC container (due to network namespace isolation) ruled out, the final approach combined the best aspects of both previous attempts while working within the Proxmox infrastructure constraint:</p><ol><li><strong>Use a VM</strong> (not a container) to provide proper network namespace isolation for USB networking</li><li><strong>Pass through the entire USB controller at the PCI level</strong> (not individual USB devices) to eliminate emulation overhead and any potential timing issues</li><li><strong>Keep the host OS clean</strong> by running SDK Manager only within the disposable VM</li></ol><p>This approach leverages Proxmox’s PCI passthrough capability—a feature designed exactly for scenarios where VMs need direct hardware access without installing drivers or tools on the hypervisor host.</p><h3 id=implementation>Implementation
|
|
<a class=heading-link href=#implementation><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><h4 id=1-identify-usb-controller>1. Identify USB Controller
|
|
<a class=heading-link href=#1-identify-usb-controller><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Find which USB controller the Jetson is connected to</span>
|
|
</span></span><span style=display:flex><span>lsusb -t | grep -B5 <span style=color:#a5d6ff>"0955:7523"</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Map USB buses to PCI addresses</span>
|
|
</span></span><span style=display:flex><span><span style=color:#ff7b72>for</span> bus in <span style=color:#ff7b72;font-weight:700>{</span>1..8<span style=color:#ff7b72;font-weight:700>}</span>; <span style=color:#ff7b72>do</span>
|
|
</span></span><span style=display:flex><span> <span style=color:#79c0ff>pci</span><span style=color:#ff7b72;font-weight:700>=</span><span style=color:#ff7b72>$(</span>readlink /sys/bus/usb/devices/usb<span style=color:#79c0ff>$bus</span> 2>/dev/null | grep -oE <span style=color:#a5d6ff>'[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9]'</span><span style=color:#ff7b72>)</span>
|
|
</span></span><span style=display:flex><span> echo <span style=color:#a5d6ff>"USB Bus </span><span style=color:#79c0ff>$bus</span><span style=color:#a5d6ff> → PCI </span><span style=color:#79c0ff>$pci</span><span style=color:#a5d6ff>"</span>
|
|
</span></span><span style=display:flex><span><span style=color:#ff7b72>done</span>
|
|
</span></span></code></pre></div><p>Result: Jetson on Bus 4, controlled by PCI device <code>0000:22:00.3</code></p><p>Verification that no other critical devices shared this controller:</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span>lsusb | grep <span style=color:#a5d6ff>"Bus 003"</span> <span style=color:#8b949e;font-style:italic># Empty except root hub</span>
|
|
</span></span><span style=display:flex><span>lsusb | grep <span style=color:#a5d6ff>"Bus 004"</span> <span style=color:#8b949e;font-style:italic># Only Jetson device</span>
|
|
</span></span></code></pre></div><h4 id=2-create-vm-with-pci-passthrough>2. Create VM with PCI Passthrough
|
|
<a class=heading-link href=#2-create-vm-with-pci-passthrough><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Create VM</span>
|
|
</span></span><span style=display:flex><span>qm create <span style=color:#a5d6ff>200</span> --name jetson-flash --memory <span style=color:#a5d6ff>4096</span> --cores <span style=color:#a5d6ff>4</span> <span style=color:#79c0ff>\
|
|
</span></span></span><span style=display:flex><span><span style=color:#79c0ff></span> --net0 virtio,bridge<span style=color:#ff7b72;font-weight:700>=</span>vmbr0 --scsihw virtio-scsi-pci
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Set machine type to q35 (required for PCIe passthrough)</span>
|
|
</span></span><span style=display:flex><span>qm set <span style=color:#a5d6ff>200</span> --machine q35
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Import Ubuntu cloud image</span>
|
|
</span></span><span style=display:flex><span>qm importdisk <span style=color:#a5d6ff>200</span> ubuntu-22.04-server-cloudimg-amd64.img local-lvm
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Configure disk and cloud-init</span>
|
|
</span></span><span style=display:flex><span>qm set <span style=color:#a5d6ff>200</span> --scsi0 local-lvm:vm-200-disk-0 --boot <span style=color:#79c0ff>order</span><span style=color:#ff7b72;font-weight:700>=</span>scsi0 <span style=color:#79c0ff>\
|
|
</span></span></span><span style=display:flex><span><span style=color:#79c0ff></span> --ide2 local-lvm:cloudinit
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Configure cloud-init</span>
|
|
</span></span><span style=display:flex><span>qm set <span style=color:#a5d6ff>200</span> --ciuser sdkmanager --cipassword sdkmanager <span style=color:#79c0ff>\
|
|
</span></span></span><span style=display:flex><span><span style=color:#79c0ff></span> --ipconfig0 <span style=color:#79c0ff>ip</span><span style=color:#ff7b72;font-weight:700>=</span>dhcp --sshkeys ~/.ssh/authorized_keys
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Add PCI passthrough for USB controller</span>
|
|
</span></span><span style=display:flex><span>qm set <span style=color:#a5d6ff>200</span> --hostpci0 0000:22:00.3,pcie<span style=color:#ff7b72;font-weight:700>=</span><span style=color:#a5d6ff>1</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Resize disk for JetPack installation</span>
|
|
</span></span><span style=display:flex><span>qm resize <span style=color:#a5d6ff>200</span> scsi0 +30G
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Start VM</span>
|
|
</span></span><span style=display:flex><span>qm start <span style=color:#a5d6ff>200</span>
|
|
</span></span></code></pre></div><h4 id=3-critical-usb-networking-kernel-modules>3. Critical: USB Networking Kernel Modules
|
|
<a class=heading-link href=#3-critical-usb-networking-kernel-modules><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p>The Ubuntu cloud image does not include USB networking kernel modules by default. This is critical because when the Jetson boots into initrd mode, it requires the host to have these modules loaded immediately.</p><p><strong>Solution</strong>: Install and load modules before starting the flash:</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Install extra kernel modules</span>
|
|
</span></span><span style=display:flex><span>apt-get install linux-modules-extra-<span style=color:#ff7b72>$(</span>uname -r<span style=color:#ff7b72>)</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Load USB networking modules</span>
|
|
</span></span><span style=display:flex><span>modprobe rndis_host
|
|
</span></span><span style=display:flex><span>modprobe cdc_ether
|
|
</span></span><span style=display:flex><span>modprobe cdc_ncm
|
|
</span></span><span style=display:flex><span>modprobe cdc_subset
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Verify modules loaded</span>
|
|
</span></span><span style=display:flex><span>lsmod | grep -E <span style=color:#a5d6ff>'rndis|cdc'</span>
|
|
</span></span></code></pre></div><p>When the Jetson transitions to initrd mode (<code>0955:7035</code>), the USB network interface (<code>usb0</code>) now appears immediately in the VM’s network namespace.</p><h4 id=4-network-configuration>4. Network Configuration
|
|
<a class=heading-link href=#4-network-configuration><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h4><p>The Jetson’s initrd uses <strong>IPv6</strong> for USB networking by default:</p><div class=highlight><pre tabindex=0 style=color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-bash data-lang=bash><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Interface appears automatically</span>
|
|
</span></span><span style=display:flex><span>ip addr show usb0
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Output:</span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># usb0: inet6 fc00:1:1::1/64 scope global</span>
|
|
</span></span><span style=display:flex><span>
|
|
</span></span><span style=display:flex><span><span style=color:#8b949e;font-style:italic># Test connectivity</span>
|
|
</span></span><span style=display:flex><span>ping6 -c <span style=color:#a5d6ff>3</span> fc00:1:1:0::2 <span style=color:#8b949e;font-style:italic># Jetson's IPv6 address</span>
|
|
</span></span></code></pre></div><p>The SDK Manager automatically detects and uses IPv6 connectivity for SSH and flash operations.</p><h3 id=flash-process-timeline>Flash Process Timeline
|
|
<a class=heading-link href=#flash-process-timeline><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><ol><li><strong>07:49:38</strong> - Flash component started, 30-second pre-check wait</li><li><strong>07:50:12</strong> - Board detected as <code>jetson-orin-nano-devkit-super</code></li><li><strong>07:51:25</strong> - System image created (rootfs populated)</li><li><strong>07:52:24</strong> - Converting to sparse image format</li><li><strong>07:54:54</strong> - Device rebooted into initrd mode (<code>0955:7035</code>)</li><li><strong>07:55:05</strong> - USB network interface <code>usb0</code> appeared immediately</li><li><strong>07:55:16</strong> - SSH connection established via IPv6</li><li><strong>07:55:16-07:59:05</strong> - QSPI flash (boot firmware) written</li><li><strong>07:59:05-08:00:28</strong> - eMMC flash (system partitions) written</li><li><strong>08:00:28</strong> - Flash successful, device rebooted to normal mode (<code>0955:7020</code>)</li><li><strong>08:00:28-08:02:58</strong> - First-boot auto-configuration</li><li><strong>08:03:00</strong> - Installation completed successfully</li></ol><p>Total flash time: <strong>~13 minutes</strong></p><h3 id=why-pci-passthrough-succeeded>Why PCI Passthrough Succeeded
|
|
<a class=heading-link href=#why-pci-passthrough-succeeded><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h3><ol><li><strong>Direct Hardware Access</strong>: The VM has complete control over the USB controller with no emulation layer</li><li><strong>Timing Precision</strong>: USB protocol timing is maintained at hardware level</li><li><strong>Network Namespace</strong>: The VM’s network stack directly handles USB network interfaces</li><li><strong>No Virtualization Overhead</strong>: USB transactions happen at native speed</li></ol><h2 id=key-lessons-learned>Key Lessons Learned
|
|
<a class=heading-link href=#key-lessons-learned><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><ol><li><p><strong>USB Device Passthrough vs Controller Passthrough</strong>: Passing through individual USB devices adds emulation overhead. PCI-level controller passthrough provides native hardware access.</p></li><li><p><strong>LXC Network Namespace Limitations</strong>: LXC containers cannot reliably handle dynamically created USB network interfaces due to network namespace isolation. Even with udev rules to move interfaces, timing and configuration issues persist.</p></li><li><p><strong>Kernel Module Requirements</strong>: USB networking kernel modules must be loaded <strong>before</strong> the Jetson enters initrd mode. Cloud images and minimal installations often lack these modules.</p></li><li><p><strong>IPv6 Support</strong>: Modern Jetson initrd images prefer IPv6 for USB networking. Ensure the host system has IPv6 enabled and properly configured.</p></li><li><p><strong>Timing Sensitivity</strong>: The Jetson’s initrd flash process has strict timing requirements. USB network interfaces must appear and be configured within seconds of the mode transition.</p></li><li><p><strong>PCI Passthrough Machine Type</strong>: QEMU/KVM requires <code>q35</code> machine type for PCIe device passthrough. The default <code>i440fx</code> machine type does not support it.</p></li></ol><h2 id=recommendations>Recommendations
|
|
<a class=heading-link href=#recommendations><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><p>For flashing Jetson devices in production or automated environments:</p><ol><li><p><strong>Use PCI USB Controller Passthrough</strong>: If virtualization is required, pass through the entire USB controller at the PCI level to a VM.</p></li><li><p><strong>Pre-load USB Networking Modules</strong>: Ensure <code>rndis_host</code>, <code>cdc_ether</code>, <code>cdc_ncm</code>, and <code>cdc_subset</code> kernel modules are loaded before starting the flash process.</p></li><li><p><strong>Verify USB Controller Isolation</strong>: Before passthrough, ensure no other critical devices share the USB controller.</p></li><li><p><strong>Use Physical Machines When Possible</strong>: For development and testing, a physical Linux machine provides the most reliable flashing experience.</p></li><li><p><strong>Monitor USB Device Transitions</strong>: Use <code>lsusb</code> and <code>dmesg</code> to monitor device state transitions:</p><ul><li><code>0955:7523</code> = Recovery mode (APX)</li><li><code>0955:7035</code> = Initrd flash mode</li><li><code>0955:7020</code> = Normal operation mode</li></ul></li></ol><h2 id=references>References
|
|
<a class=heading-link href=#references><i class="fa-solid fa-link" aria-hidden=true title="Link to heading"></i>
|
|
<span class=sr-only>Link to heading</span></a></h2><ul><li>NVIDIA Jetson Linux Developer Guide: <a href=https://docs.nvidia.com/jetson/ class=external-link target=_blank rel=noopener>https://docs.nvidia.com/jetson/</a></li><li>NVIDIA SDK Manager Documentation: <a href=https://developer.nvidia.com/sdk-manager class=external-link target=_blank rel=noopener>https://developer.nvidia.com/sdk-manager</a></li><li>Proxmox VE PCI Passthrough: <a href=https://pve.proxmox.com/wiki/PCI_Passthrough class=external-link target=_blank rel=noopener>https://pve.proxmox.com/wiki/PCI_Passthrough</a></li><li>Linux USB Networking Drivers: <a href=https://www.kernel.org/doc/html/latest/usb/ class=external-link target=_blank rel=noopener>https://www.kernel.org/doc/html/latest/usb/</a></li><li>QEMU USB Documentation: <a href=https://www.qemu.org/docs/master/system/devices/usb.html class=external-link target=_blank rel=noopener>https://www.qemu.org/docs/master/system/devices/usb.html</a></li><li>LXC Container Configuration: <a href=https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html class=external-link target=_blank rel=noopener>https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html</a></li></ul></div><footer><div id=disqus_thread></div><script>window.disqus_config=function(){},function(){if(["localhost","127.0.0.1"].indexOf(window.location.hostname)!=-1){document.getElementById("disqus_thread").innerHTML="Disqus comments not available by default when the website is previewed locally.";return}var t=document,e=t.createElement("script");e.async=!0,e.src="//ericxliu-me.disqus.com/embed.js",e.setAttribute("data-timestamp",+new Date),(t.head||t.body).appendChild(e)}(),document.addEventListener("themeChanged",function(){document.readyState=="complete"&&DISQUS.reset({reload:!0,config:disqus_config})})</script></footer></article><link rel=stylesheet href=https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.css integrity=sha384-vKruj+a13U8yHIkAyGgK1J3ArTLzrFGBbBc0tDp4ad/EyewESeXE/Iv67Aj8gKZ0 crossorigin=anonymous><script defer src=https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/katex.min.js integrity=sha384-PwRUT/YqbnEjkZO0zZxNqcxACrXe+j766U2amXcgMg5457rve2Y7I6ZJSm2A0mS4 crossorigin=anonymous></script><script defer src=https://cdn.jsdelivr.net/npm/katex@0.16.4/dist/contrib/auto-render.min.js integrity=sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05 crossorigin=anonymous onload='renderMathInElement(document.body,{delimiters:[{left:"$$",right:"$$",display:!0},{left:"$",right:"$",display:!1},{left:"\\(",right:"\\)",display:!1},{left:"\\[",right:"\\]",display:!0}]})'></script></section></div><footer class=footer><section class=container>©
|
|
2016 -
|
|
2025
|
|
Eric X. Liu
|
|
<a href="https://git.ericxliu.me/eric/ericxliu-me/commit/832aabc">[832aabc]</a></section></footer></main><script src=/js/coder.min.6ae284be93d2d19dad1f02b0039508d9aab3180a12a06dcc71b0b0ef7825a317.js integrity="sha256-auKEvpPS0Z2tHwKwA5UI2aqzGAoSoG3McbCw73gloxc="></script><script defer src=https://static.cloudflareinsights.com/beacon.min.js data-cf-beacon='{"token": "987638e636ce4dbb932d038af74c17d1"}'></script></body></html> |