Windows-on-ARM dev loop (CAP-2.1)

Audience: Pavri engineers iterating on the Windows sensor (ETW observers, named-pipe transport, Windows PeerCred) on an Apple Silicon Mac.

What this gets you: edit Go code on the Mac, run make winlive-test, and see results from a real Windows NT kernel in ~10-15 seconds. No PR, no GitHub Actions wait. Mirrors the same package set that .github/workflows/cap-2-1-windows.yml runs against windows/amd64 on every PR, so local-pass implies CI-pass.

Architecture

┌──────────────────────┐     ssh / scp       ┌───────────────────────────┐
│  macOS (Apple        │ ───────────────────▶│  Windows 11 ARM64 in UTM  │
│  Silicon)            │                     │  (Apple Hypervisor)       │
│                      │  cross-compiled     │                           │
│  GOOS=windows        │  test binaries      │  C:\Users\pavri\         │
│  GOARCH=arm64        │  (one per pkg)      │    pavri-tests\*.exe     │
│  go test -c          │                     │                           │
│                      │ ◀───────────────────│  test runtime + ETW       │
│  ./scripts/          │  test output        │  + named pipe + sshd      │
│  winlive-test.sh     │                     │                           │
└──────────────────────┘                     └───────────────────────────┘
        ↑
        └── make winlive-test  (or:  scripts/winlive-test.sh ./pkg)

Why ARM64 locally (when production targets windows/amd64): Apple Hypervisor.framework gives near-native speed for ARM64 Windows. ETW provider semantics, named-pipe semantics, and Win32 syscall surface are identical between arm64 and amd64. CI continues to run windows/amd64 on a real GHA hosted runner so production-target validation still happens on every push.

Why no SMB / no in-VM Go: The plan originally proposed SMB-mounting the repo and running go test inside the VM. Cross-compiling on the Mac and scp'ing only the test binaries is simpler (no macOS Sharing setup, no Go install in the VM, no clipboard issues) and the per-iteration cost is the same ~10s. SMB and in-VM Go remain valid alternatives if a future test suite genuinely needs the source tree mounted; nothing in this loop prevents that.

One-time setup

1. Acquire the Windows VHDX

Download Windows 11 Pro ARM64 VHDX from Microsoft Insider Preview Downloads (requires a free Microsoft account). Save it to ~/VMs/win11-arm.vhdx.

2. Convert to qcow2 and install UTM

mkdir -p ~/VMs
brew install --cask utm
brew install qemu       # supplies qemu-img
qemu-img convert -p -f vhdx -O qcow2 ~/VMs/win11-arm.vhdx ~/VMs/win11-arm.qcow2

3. Create the UTM VM

In UTM, Create a New Virtual Machine → Virtualize → Windows:

Setting	Value
Memory	8192 MB
CPU Cores	4
Storage	delete the auto-created disk; New Drive → Import ~/VMs/win11-arm.qcow2, Interface: NVMe
Network Mode	Shared Network
Display	virtio-ramfb-gl

Why NVMe and not VirtIO for the boot disk: Microsoft's stock ARM64 VHDX doesn't ship a VirtIO driver, so VirtIO yields "no boot device" on first boot. Windows ARM has a native NVMe driver. After first boot you can install virtio-win drivers if you want to switch.

Boot the VM. At the OOBE Microsoft-account / passkey screen press Shift+F10 and run start ms-cxh:localonly (24H2+) or fall back to OOBE\BYPASSNRO to skip to local-account creation.

4. Install the VirtIO network driver

The ARM64 VHDX has no VirtIO NIC driver, so the VM has no network on first boot. On the Mac:

curl -fL -o ~/VMs/virtio-win.iso \
  "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso"

In UTM Edit → Drives → New Drive (CD/DVD, Removable) → Import the ISO, boot, then in Windows Device Manager → Other devices → Ethernet Controller → Update driver → Browse → D:\NetKVM\w11\ARM64 (or w10\ARM64 if no w11) → accept the Red Hat publisher prompt. Network comes up immediately.

5. Bootstrap SSH from the Mac

In Windows, open an admin PowerShell:

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Start-Service sshd
Set-Service -Name sshd -StartupType Automatic
ipconfig | Select-String IPv4

Note the VM's IPv4 (typically 192.168.64.x).

If Add-WindowsCapability fails with 0x800B010F, the cause is usually clock skew on the stale VHDX. Start-Service w32time; w32tm /resync /force; Get-Date and retry.

On the Mac, append a host alias to ~/.ssh/config:

Host pavri-win
  HostName 192.168.64.X      # the IP from ipconfig
  User pavri                # whatever local user you created
  IdentityFile ~/.ssh/id_ed25519
  StrictHostKeyChecking accept-new
  ServerAliveInterval 30

Install your Mac's public key into the Windows-admin location. Easiest is a one-shot HTTP server on the Mac:

mkdir -p /tmp/keyshare && cp ~/.ssh/id_ed25519.pub /tmp/keyshare/key.pub
cd /tmp/keyshare && python3 -m http.server 8765

Then in the VM admin PowerShell:

$key = (New-Object Net.WebClient).DownloadString('http://192.168.64.1:8765/key.pub').Trim()
$f = 'C:\ProgramData\ssh\administrators_authorized_keys'
Add-Content -Path $f -Value $key
icacls $f /inheritance:r
icacls $f /grant 'Administrators:F'
icacls $f /grant 'SYSTEM:F'

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell `
  -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" `
  -PropertyType String -Force | Out-Null

# UTM Shared Network usually lands as Public profile, which blocks LAN inbound.
Get-NetConnectionProfile | Where-Object NetworkCategory -ne 'Private' |
    Set-NetConnectionProfile -NetworkCategory Private -ErrorAction SilentlyContinue
Get-NetFirewallRule -DisplayName "OpenSSH*" |
    Set-NetFirewallRule -Profile Any -Enabled True
Restart-Service sshd

Why administrators_authorized_keys and not ~/.ssh/authorized_keys: Windows OpenSSH refuses ~/.ssh/authorized_keys for accounts in the Administrators group; it reads C:\ProgramData\ssh\administrators_authorized_keys instead, with a required ACL of (Administrators, SYSTEM) only.

Test from the Mac: ssh pavri-win whoami should print <hostname>\<user> without a password prompt.

Daily use

make winlive-test                                                       # all default packages
make winlive-test ARGS="./services/endpointd/internal/app/runtime/..."  # one package
GOARCH=amd64 make winlive-test                                          # match CI's arch
WINLIVE_HOST=other-vm make winlive-test                                 # alternate VM

What scripts/winlive-test.sh does, in order:

1. Greps for //go:build .*winlive files under each requested package and collects the func TestX(...) names defined in those files.
2. Cross-compiles one test binary per package (GOOS=windows GOARCH=arm64 go test -tags=winlive -c).
3. scp's the binaries to ~/pavri-tests/ on the VM.
4. Invokes each binary via cmd /c (the OpenSSH default-shell is PowerShell, which mangles flags like -test.v), with -test.run set to only the winlive-tagged test names so unrelated /tmp-hardcoded tests in the same package are skipped.

Troubleshooting

Symptom	Cause / fix
ssh pavri-win times out	UTM Shared Network landed in Public profile. Set-NetConnectionProfile -NetworkCategory Private.
Add-WindowsCapability errors 0x800B010F	Clock skew or DNS. Start-Service w32time; w32tm /resync /force. If clock is fine, force DNS to 1.1.1.1, 8.8.8.8.
flag provided but not defined: -test	Ran the binary directly from PowerShell. PS parses -test.v as a parameter name. Use cmd /c (winlive-test.sh does this automatically).
Tests fail with mkdtemp: GetFileAttributesEx /tmp:	Test code hardcodes /tmp. Pre-existing hygiene bug, not winlive-specific. The script auto-filters to winlive-tagged test names so these are skipped.
cmd.exe interprets | as a shell pipe	The script caret-escapes | → ^|. If you write a custom -test.run, do the same.
ETW tests report ErrInsufficientCapabilities	Test ran as non-admin. The first user created at OOBE is admin by default; if you created a non-admin user, run from an admin shell.
First boot says "no boot device"	Disk attached as VirtIO; switch to NVMe in UTM Edit → Drives.

Limits

• ETW provider semantics are architecturally identical between ARM64 and AMD64, but production-target verification still requires the windows-2022 GHA runner (which the CAP-2.1 workflow runs on every push).
• UTM clipboard / folder-sharing is unsupported in Apple Virtualization mode. We avoid the issue: edits stay on the Mac; only test binaries cross the boundary.
• Cross-compile + scp is the entire dev loop; we deliberately do not install Go on the VM. If a future test needs additional binaries, add them to the cross-compile + scp step in winlive-test.sh.