nest/DEVELOPER.md

UVM Developer Guide

This document is the developer handoff for the entire UVM codebase. Read it before changing lifecycle behavior, host networking, Firecracker launch semantics, persistent state, or the HTTP API.

uvm is a local, privileged Firecracker microVM manager. It is an application with a CLI and optional FastAPI management server. It is not a general-purpose Python library or a multi-host control plane.

1. Goals and Boundaries

UVM installs host/guest assets, provisions private disks, and manages named VMs and private IPv4 TCP allowlists on one host. The CLI offers retained-disk start/stop/restart, CPU/RAM/storage resizing, deletion, fixed-target bulk operations, inspection, SSH, logs, doctor, and explicit port edits/sync. The optional API and local dashboard expose health, installation, VM lifecycle controls, and managed port-policy controls, host diagnostics, bounded VM-log tails, and fixed-target bulk operations. Interactive SSH remains CLI-only.

There is no autostart or background reconciler, multi-host control plane, tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port filtering is not public-port publishing or a hardened multi-tenant boundary; see Current Production Gaps for the remaining scope limits.

Treat UVM as a privileged local systems tool. It manipulates /dev/kvm, TAP interfaces, bridge addresses, nftables/conntrack and existing iptables egress rules, host filesystems, and processes. For the complete operator command and flag reference, see USAGE.md.

2. First Run

All UVM-specific files live in this directory. Start here:

python3 -m pip install -e '.[test]'
python3 -B -m unittest discover -s tests -v
./uvm.py --help

The uvm console script is registered by pyproject.toml. During source development, ./uvm.py and python3 -B -m uvm are equivalent entry points.

Requirements for real host operations:

• Python 3.12+
• Linux with KVM enabled and /dev/kvm accessible
• Linux >= 5.3 with native bridge conntrack and safe pidfd support
• Root privileges for installation and lifecycle mutations
• iproute2, iptables, nftables, conntrack, coreutils, e2fsprogs, util-linux, openssl, and OpenSSH client tools
• An x86_64 or ARM64 (aarch64) host, including NVIDIA DGX Spark

Do not run real integration operations on a developer workstation unless the bridge, subnet, firewall changes, and guest workload are acceptable there. Packet integration tests must remain inside disposable isolated namespaces.

3. Project Layout

./
├── DEVELOPER.md              # This document
├── README.md                 # Project overview
├── USAGE.md                  # Operator command/deployment guide
├── pyproject.toml            # Package metadata and dependencies
├── uvm.py                    # Executable source-tree launcher
├── uvm/
│   ├── __main__.py           # `python -m uvm`
│   ├── cli.py                # argparse, CLI rendering, serve dispatch
│   ├── app.py                # Composition root
│   ├── config.py             # Settings and environment loading
│   ├── domain.py             # PortRule, VmSpec, VmRecord, State
│   ├── errors.py             # Expected error hierarchy
│   ├── validation.py         # Resources, names, credentials, ports, timeouts
│   ├── system.py             # CommandRunner, root/KVM/directory checks
│   ├── state.py              # JSON state, atomic writes, file locks
│   ├── integrity.py          # SHA-256 and integrity manifest logic
│   ├── install.py            # Apt, Firecracker, kernel, rootfs installation
│   ├── images.py             # Template lookup, private disks, and guest credentials
│   ├── known_hosts.py        # Deletion-only cleanup of local SSH host-key caches
│   ├── network.py            # IP/MAC/TAP/bridge/NAT/forward rules
│   ├── firewall.py           # Owned nft bridge tables and offline conntrack cleanup
│   ├── lifecycle.py          # Lifecycle, bulk operations, recovery, port intent
│   ├── api_models.py         # Pydantic HTTP request/response models
│   ├── server.py             # FastAPI factory and Uvicorn launcher
│   ├── dashboard/            # Packaged local dashboard HTML, CSS, and JS
│   ├── routers/
│   │   ├── dashboard.py      # Dashboard document route
│   │   ├── diagnostics.py    # Host diagnostic endpoint
│   │   ├── dependencies.py   # API application/token dependencies
│   │   ├── health.py         # Liveness endpoint
│   │   ├── installation.py   # Installation endpoint
│   │   └── vms.py            # VM lifecycle endpoints
│   └── firecracker/
│       ├── config.py         # Firecracker JSON and boot arguments
│       ├── api.py            # Unix-socket HTTP client
│       └── process.py        # Process launch, liveness, termination
└── tests/                    # Unit/API tests; integration/ has opt-in namespace packets

Keep feature code in the most specific existing module. Do not add a new abstraction layer merely to wrap one function. The current module boundaries are intentionally small and operationally focused.

4. Architecture Overview

flowchart LR
  CLI[CLI: uvm] --> APP[app.py composition root]
  HTTP[FastAPI server] --> APP
  APP --> INSTALL[Installer]
  APP --> LIFE[LifecycleService]
  APP --> STATE[StateStore]
  LIFE --> IMAGES[ImageStore]
  LIFE --> NET[NetworkManager]
  NET --> FW[FirewallManager]
  LIFE --> FC[Firecracker adapters]
  INSTALL --> RUNNER[CommandRunner]
  NET --> RUNNER
  FW --> RUNNER
  FC --> HOST[Linux host primitives]
  RUNNER --> HOST
  STATE --> DISK[(state.json)]
  IMAGES --> DISKSTORE[/var/lib/uvm/images and vms]
  FC --> KVM[/dev/kvm]

The public entry points are thin adapters. The lifecycle layer is the only place that coordinates multiple host resources for a VM. Lower-level modules must not import cli.py, server.py, or lifecycle.py.

Leaf modules are config.py, domain.py, errors.py, and validation.py. They must remain free of subprocess, network, CLI, and FastAPI side effects.

5. Entrypoints and Request Paths

CLI Path

cli.py builds the application, validates arguments, and calls lifecycle services. It owns confirmation, terminal/JSON output, exit handling, SSH execvp, local log streaming from a validated path, and --serve dispatch. It must not coordinate state files, interfaces, or Firecracker sockets itself.

The local surface is install/create/list/inspect/start/stop/restart/resize/delete (destroy alias)/ssh/logs/doctor and ports list/open/close/set/sync. Identifiers resolve by exact ID, name, or guest IP, never an ID prefix. Generated-ID-shaped inputs resolve only by ID; names cannot impersonate generated IDs. Ambiguous non-ID aliases raise ConflictError.

Create alone adds 22:TCP unless --no-default-ports is supplied. Port syntax is PORT[:TCP|HTTP|HTTPS], comma-separated or repeated via --ports; all labels use TCP. --password-stdin reads one line instead of using a password argument. _vm_output() is an explicit output whitelist, not VmRecord.to_dict(). JSON stays on stdout, while warnings, prompts, and host traces use stderr.

Bulk lifecycle actions and ports sync resolve/deduplicate IDs before prompting. Delete and mutating --all selections require a TTY confirmation or --yes/-y. Port set/close-all and removal of TCP/22 also prompt, except empty managed no-ops. --dry-run previews without lifecycle/policy mutations or confirmation; it is not a full preflight. Stop/restart/resize/delete accept --timeout; start does not. Logs follow does not hold the operation lock. Doctor runs readiness checks, not per-VM repair or guest application probes.

HTTP Path

Routers validate the token and Pydantic request, invoke application services, and render whitelisted response models. Lifecycle, managed port-policy, diagnostics, and bounded log-tail operations have explicit HTTP routes for the dashboard. Bulk operations have a dedicated fixed-target HTTP route; interactive SSH has no HTTP endpoint. The public dashboard shell does not bypass token validation: it stores a supplied token only for the active browser session and sends it with management requests.

Use only uvm --serve or ./uvm.py --serve to launch the API. The create_api() factory requires an explicit host specifically to prevent it from being used as an insecure generic Uvicorn factory.

6. Application Composition

app.py constructs one concrete object graph per CLI or server process:

Component	Constructor	Responsibility
Settings	Settings.from_environment()	Immutable runtime configuration
CommandRunner	CommandRunner	Logged/testable subprocess execution
StateStore	StateStore	Inventory, locks, JSON persistence
ImageStore	ImageStore	Template verification, VM disks, and offline credentials
KnownHostsManager	KnownHostsManager	Scoped local SSH cache cleanup before deleted IPs are released
NetworkManager	NetworkManager	Guest allocation and Linux networking
FirewallManager	Constructed by NetworkManager	Owned native bridge policy and scoped offline conntrack cleanup
FirecrackerProcessManager	FirecrackerProcessManager	VMM process control
LifecycleService	LifecycleService	Ordered VM resource changes and rollback
Installer	Installer	Host packages and artifacts

Composition can disable command tracing with emit=None; create_api() does so when it constructs its own application. The CLI supplies an emitter that prints + ... traces to stderr, including when passing its application to the server launcher.

7. Persistent Data and Models

Host Data Layout

Default runtime storage is /var/lib/uvm:

/var/lib/uvm/
├── bin/
│   ├── firecracker
│   └── jailer
├── images/
│   ├── vmlinux
│   └── ubuntu.ext4
├── vms/
│   └── vm-<uuid>/
│       ├── rootfs.ext4
│       ├── config.json
│       ├── firecracker.log
│       └── firecracker.sock
├── state.json
├── state.lock
├── operations.lock
└── integrity.json

Per-VM runtime directories are mode 0700; copied VM disks are mode 0600. state.json is mode 0600 because VM records contain plaintext guest passwords. Registry-backed commands therefore need access to the root-owned state file.

State Schema

domain.py is the schema authority. The default registry is /var/lib/uvm/state.json; its current schema version is 3. An empty registry:

{
  "schema_version": 3,
  "next_mac_index": 1,
  "vms": {}
}

VmSpec is the validated create request. VmRecord is the persisted local record. Do not make API models or CLI namespaces the persistence format.

Record Fields	Purpose
id, name, cpu, ram_mib, storage_mib	Stable identity, name, applied resources; storage is null for older disks
pending_resize	Null or a validated VmResources target with CPU, RAM MiB, and storage MiB
guest_ip, gateway, network, bridge, tap, mac	Retained network identity/configuration
socket, config, log, disk	Canonical per-VM runtime paths
username, password	Guest credentials; password stays out of public output
ports	Null for legacy/unmanaged, or a list of { "port": 80, "protocol": "HTTP" } objects
disk_ready, runtime_owned, tap_owned	Completed provisioning and resource ownership evidence
status, operation, spawn_pending, last_error	Durable state, pending intent, ambiguous-spawn guard, failure detail
pid, process_start_time, boot_id	VMM identity, including host reboot detection
created_at, updated_at	Registry timestamps

PortRule accepts TCP/HTTP/HTTPS labels and integer ports 1-65535. A port number is unique regardless of label. VmSpec.ports defaults to TCP/22, whereas VmRecord.ports=None deliberately preserves the legacy distinction. An explicit empty tuple/list is a managed empty TCP allowlist, not unmanaged mode. There is no persisted boolean asserting that a policy is enforced.

State Guarantees

• state.json writes use a sibling temporary file, fsync, atomic replace, and parent-directory fsync.
• state.json is mode 0600; it must be treated as a credential registry and excluded from logs and bug reports.
• state.lock serializes short JSON read-modify-write transactions.
• operations.lock serializes long lifecycle, bulk, policy, and installation operations.
• State reads are lock-free because atomic replacement gives readers either a complete old file or a complete new file.
• Legacy versions 0 (including unversioned state), 1, and 2 are loaded into schema 3 in memory and written as schema 3 on the next transaction. Newer schemas are rejected. Reads do not reconcile VMs or adopt process identities.
• Missing name, ports, and boot_id remain null. No policy or default SSH rule is silently imposed on a legacy VM. Use stop, explicit ports set, then start; first policy adoption requires a stopped, proven-gone VM.
• Missing disk and process start time remain absent. Legacy disk_ready is inferred only for running/stopped records with a recorded private disk; start still validates the path, file, and provisioning state.
• Legacy records own their canonical runtime directory independently of whether they had a private disk. Deletion never removes the shared rootfs template; explicit runtime_owned=False on a new interrupted creation stays authoritative.
• Records created before credential provisioning default to username root and password null; UVM does not apply a password retroactively.
• Missing storage_mib and pending_resize remain null. The first explicit resize infers an older disk's capacity and enforces the 30 GiB minimum; reads and schema migration never resize a disk. Schema 3 prevents older UVM versions from ignoring pending resource changes during a boot.
• next_mac_index is reconstructed from older locally administered MACs when absent.

When adding a persisted field, update VmRecord.to_dict, VmRecord.from_dict, the state schema tests, and this document. Never silently discard existing state fields without a migration plan.

8. VM Lifecycle

Stored and Observed Status

The persisted status is a string, not an enum, to keep legacy JSON readable.

Status	Meaning
starting	Identity reserved; host resources are being created
running	Firecracker configuration and InstanceStart succeeded
stopping	Stop mutation is in progress
stopped	Process/TAP removed; private disk remains
terminating	Destruction is in progress
failed	A lifecycle step or cleanup failed; last_error explains why
dead	Observed-only status when a stored running process is proven gone
unknown	Observed process identity cannot be established safely

Transition statuses can survive interruption. dead/observed unknown do not rewrite inventory. Delete removes the record rather than keeping a tombstone. Pending deletion remains authoritative even if a subsequent stop finishes runtime cleanup; start and policy edits must not resurrect it.

ListedVm separates the record, observed process status, and observed policy status. Policy can be enforced, missing, drift, quarantined, unknown, legacy unmanaged, or inactive for a cleanly stopped managed VM. Readback is a ruleset/attachment snapshot, not application health or a reachability test.

Create Order

LifecycleService.create follows this sequence:

1. Validate the request/root access and acquire operations.lock.
2. Check KVM, verified Firecracker, firewall capability, and installed templates.
3. Reserve ID/name/IP/MAC/ports with status=starting, operation=preparing.
4. Create the owned mode-0700 runtime directory, sparse-copy a mode-0600 private disk, resize ext4 to the requested storage capacity, provision credentials/host keys, and persist disk_ready=True.
5. Ensure bridge/egress networking and prepare an owned, attached TAP down.
6. Apply/read back the nftables bridge policy, then activate the TAP.
7. Write Firecracker config; persist operation=spawning, spawn_pending=True, and the host boot ID before Popen.
8. Publish PID/start time/boot ID through on_spawn before socket readiness or API configuration; clear the pending-spawn guard.
9. Configure Firecracker and InstanceStart, verify policy enforcement again, then save running with no pending operation.

This records both intent and acquired ownership across crash windows. Never make a managed TAP usable before enforcement or mark a partial disk ready.

Rollback Rules

Any Exception, KeyboardInterrupt, or SystemExit during create triggers compensation:

1. Attempt quarantine/TAP deactivation and confirm safe process exit.
2. Remove the owned TAP, clean up scoped conntrack entries, and remove its policy.
3. Remove the owned runtime directory/private disk only after confirmed exit.
4. Remove the reservation only if cleanup succeeds.
5. Otherwise retain a failed record with identity/intent and last_error; do not release its disk or IP for reuse.

Do not reorder these operations without considering leaked TAPs, orphaned VMM processes, and allocation reuse.

Start and Restart

To boot a stopped/proven-gone VM, start requires managed ports, a complete safe private disk, and compatible retained gateway, network, and bridge settings. It verifies the VMM and installed kernel; the original rootfs template is not needed. It never recopies the rootfs, reprovisions credentials, or regenerates guest SSH host keys. Name/IP/MAC, ports, and guest disk changes survive stop/start; guest RAM does not.

Start on a healthy running VM synchronizes policy without spawning another VMM. An ambiguous process or interrupted live boot/stop is not adopted as a successful start. Failed resume retains the private disk for retry. Restart performs stop and start under one operation lock, not two independent public calls with a race between them.

Resource Resizing

resize() accepts a partial CPU/RAM/storage update for one VM. New disks default to 30720 MiB (30 GiB), also the minimum accepted storage. Storage parsing uses GiB for bare numbers, binary B/K/M/G/T suffixes, and rounds up to MiB; RAM retains its MiB-default parsing. Capacities must fit the host's signed 64-bit file offsets.

One operations.lock spans preflight, durable pending_resize publication, _stop_locked(), offline resource application, and _start_locked(). Stop must prove process exit before any disk work. The complete target is a validated VmResources, while the record's CPU/RAM/storage fields remain the applied values. Only the resource fields change; identity, networking, credentials, ports, and guest SSH keys are retained. Even an initially stopped VM boots after resize.

ImageStore.resize_vm_disk() sparse-copies the private disk to a mode-0600 sibling, runs e2fsck -f -p, resizes ext4, verifies its geometry using dumpe2fs, fsyncs, then atomically replaces the original and fsyncs the directory. Growth extends the image before resize2fs; shrinking truncates it only after ext4 fits and its geometry matches. Failed staging leaves the original disk intact. CPU/RAM-only resizes avoid disk work when its configured capacity is already known.

After disk publication, one state write applies all resource fields and clears pending_resize; boot then follows the normal start path. A crash between disk publication and that state write is recoverable: the target is durable and the image layer recognizes an already-completed capacity. A stopped VM's start retries a pending resize before boot; a live pending resize requires resize or stop first. Stop retains the target and deletion never applies it. A failed boot after successful resizing retains the new resources for retry.

Stop and Delete

stop:
  persist stop intent -> quarantine/deactivate TAP -> confirm VMM exit
  -> delete owned TAP -> scoped conntrack cleanup -> remove owned nft table
  -> stopped (retain disk, credentials, name/IP and desired ports)

delete (destroy alias):
  persist delete intent -> same runtime cleanup -> remove owned runtime directory
  -> clean local known_hosts entries -> remove state record -> release name/IP

Stop signals the VMM, not a guest shutdown agent or ACPI-coordinated shutdown. SIGTERM is followed by SIGKILL when needed. --timeout is the per-VM TERM wait, default 5 seconds, finite and non-negative; it is not a total deadline. Even zero still requires confirmed kernel exit before cleanup. Application consistency requires separately quiescing the guest.

Runtime cleanup retains reservations when process exit, TAP ownership, or connection cleanup is uncertain. A pending spawn cannot be declared gone merely because its PID was never persisted or its socket has not appeared.

KnownHostsManager is an explicit lifecycle dependency, also used by API and bulk deletion. It runs after successful process/network/disk teardown and before the registry entry is removed, under the same operation lock. Cleanup errors retain operation=delete and the IP reservation; repeating delete retries it. Stop, restart, and resize retain the existing guest's trusted host-key entries.

Default caches are known_hosts and known_hosts2 under the effective user's and sudo invoker's account homes (SUDO_UID, with a SUDO_USER fallback). UVM_KNOWN_HOSTS_FILES overrides this discovery for service deployments using a colon-separated list of absolute paths. Only local files are managed. Matches are explicit guest IPs or UVM's uvm-<VM-ID> host-key alias, including OpenSSH's HMAC-SHA1 hashed form. Plain bracketed ports are recognized directly; hashed bracketed candidates use port 22 and the VM's recorded allowlist. Wildcard rules and other aliases sharing a line are retained.

File updates use pinned directory descriptors, regular/single-link checks, an advisory lock, and a sibling temporary file. Owner, group, mode, and extended attributes are preserved. A changed source snapshot prevents publication over concurrent edits. A matching-free file is not rewritten and missing files are not created. Tests inject the dependency to avoid accessing developers' caches, and verify hashed-entry interoperability using real OpenSSH on temporary files.

Policy and Bulk Operations

set_ports() takes a frozen registry ID, not a reusable name/IP; the CLI resolves the user's identifier before confirmation. It persists the new allowlist with operation=policy before applying it. Live updates apply atomically and retain conntrack; cleanly stopped edits are metadata-only. On apply/readback failure, preserve the new intent, attempt link-down/quarantine, and leave a failed pending record. Never restore an older, more permissive allowlist after a failed close. An explicit ports sync retries the desired policy and can reactivate a contained live VM.

run_many() holds one outer operation lock while processing the fixed ID snapshot produced before CLI confirmation. It reloads by exact ID, not a reusable name/IP. Results are succeeded, failed, skipped (already absent), or interrupted; ordinary failures do not undo other successes or stop later targets. Interruptions skip the remaining targets. Read-only views and API startup never reconcile; there is no autostart or background repair loop.

9. Firecracker Integration

The firecracker/ package has three separate responsibilities.

File	Responsibility
config.py	Build boot args and Firecracker JSON; atomically write config
api.py	Typed HTTP-over-Unix-socket requests
process.py	Spawn, socket readiness, liveness, signal safety, cleanup

Configuration Order

FirecrackerClient.configure_and_start() must keep this ordering:

1. PUT /machine-config
2. PUT /boot-source
3. PUT /drives/rootfs
4. PUT /network-interfaces/eth0
5. PUT /actions with InstanceStart

The guest receives a static IP via the kernel ip= boot argument. The netmask comes from Settings.network, not a hard-coded /24 value. ARM64 guests also receive keep_bootcon, as recommended by Firecracker's getting-started guide; both architectures use console=ttyS0.

Process Safety

Firecracker starts detached in a new session, appending stdout/stderr to the per-VM log. Identity consists of PID, /proc/<pid>/stat start time, and the host boot ID. observe() distinguishes alive, gone, and unknown; unreadable evidence is not proof of exit. A changed boot ID or start token can prove the old process gone without signaling a reused PID.

Start probes safe pidfd support before spawning. Termination opens a pidfd, rechecks identity, signals through that handle (never a bare PID), and confirms whole-process exit before unlinking the socket. Python bindings or libc bindings may supply pidfd support; there is no unsafe PID-signal fallback. A zombie leader alone does not prove its worker threads have exited.

Legacy adoption is deliberately narrow. During a locked mutation, adopt_legacy_identity() may fill a missing boot ID only when all evidence matches: a saved numeric start token, a live pidfd-pinned process, the configured Firecracker executable path and inode, the exact --api-sock command line, and root-owned/protected binary, socket, and ancestor directories. It checks that the process has not exited before returning, and the lifecycle layer persists the adopted boot ID. Listing/inspection never adopts it.

A live legacy PID without a saved start token, or with mismatched/inaccessible binary/socket evidence, remains unknown and fails closed. UVM does not guess an identity from the current occupant of a PID or delete an in-use disk to make recovery convenient. Proven-gone legacy processes can be cleaned up.

10. Guest Images and Integrity

Template Contract

images/ubuntu.ext4 is a template only. Each create operation copies it into the VM runtime directory before UVM modifies the requested existing account's shadow entry and OpenSSH password policy with debugfs. Firecracker only sees the resulting private copy. Never change that to a shared writable rootfs. Copies use coreutils cp --reflink=auto --sparse=always; the new disk is resized to the requested capacity before credential provisioning. Existing disk resizes do not reprovision credentials or host keys.

Guest images must provide /etc/passwd, /etc/shadow, OpenSSH's /etc/ssh/sshd_config, and the requested account. UVM does not create missing users. It replaces conventional RSA, ECDSA, and Ed25519 host keys in each private disk and removes the known public login key from Firecracker's bionic image. A VM must never rely on reusable SSH private keys from a shared template.

Integrity Model

integrity.py has two concepts:

• A configured trusted SHA-256 from the environment.
• A local integrity.json manifest recording artifact hashes and whether the installation was verified.

Strict installation requires all of these values:

UVM_FIRECRACKER_SHA256
UVM_KERNEL_SHA256
UVM_ROOTFS_SHA256

UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 is a local-development escape hatch. It writes a manifest marked verified: false; a later strict run will not treat that manifest as publisher trust. Use install --force with trusted digests to refresh an existing artifact without manually deleting managed files.

Retain the explicit opt-out on create/start/restart/resize (or the API server) when using an unverified development installation. It is not appropriate for untrusted workloads. A bare strict install requires the trusted digests.

The optional UVM_FIRECRACKER_BINARY_SHA256 validates a local Firecracker binary when no verified manifest is available.

11. Networking

Topology

guest eth0
    |
virtio-net / Firecracker
    |
per-VM TAP: uvm-<suffix>
    |
Linux bridge: uvm0 (10.42.0.1/24 by default)
    |
iptables forwarding + MASQUERADE
    |
host default uplink
    |
internet

network.py owns address allocation and host attachments; firewall.py owns native bridge filtering:

• IP allocation from Settings.network, excluding network, broadcast, and gateway addresses.
• Locally administered MAC generation: 02:fc:00:xx:xx:xx.
• Interface names limited to Linux IFNAMSIZ (15 characters).
• Bridge creation or validation that an existing interface is actually a Linux bridge.
• Gateway address setup and IPv4 forwarding.
• One egress NAT rule and forward/return rules for the default uplink.
• TAP ownership aliases (uvm:<id>), preparation while down, activation, deactivation, cleanup, and error reporting.

Network actions are idempotent where practical. Do not adopt an existing TAP: it may belong to another VM. A create-time name collision is an error. Cleanup checks the TAP kind and ownership alias, with a narrow unaliased-TAP allowance for legacy records; it does not treat an arbitrary interface name as ownership.

Native Bridge Policy

The supported path is a software Linux bridge with STP disabled. Each managed TAP's bridge-slave group_fwd_mask must include 0x4000 (the network layer sets 16384). This sends LLDP through the early filter instead of a pre-hook delivery/learning path. Firewall apply/status checks the bridge and port settings when the TAP exists. Do not use hardware/flow offloads or other paths that bypass these hooks. This is not an offload-isolation guarantee.

Native bridge conntrack requires Linux >= 5.3. Capability checks run conntrack --version, inspect the kernel release, and ask nft --json --check to validate a full policy without committing it. The nft CLI must support the rendered JSON. This layer needs no br_netfilter sysctl changes.

Each VM owns one bridge table named uvm_fw_<digest>, derived from the resolved UVM_BASE and exact VM ID, with a full ownership comment. Apply uses an atomic nft JSON transaction, delete-by-handle for the old owned table, and exclusive create for its replacement. Foreign ownership is an error, not an adoption opportunity. Readback normalizes presentation differences but compares all effective objects, ordered rules, hooks, identity, and the full port set; rule counts or an owner comment alone are not proof of enforcement. Normalization materializes implied EtherType constraints before folding transport-protocol dependencies. This accepts nftables' omitted redundant ether type ip checks while retaining the IPv4 scope of TCP rules.

The policy has prerouting/input/forward/output hooks. Early prerouting identity checks run at priority -310, before conntrack and bridge MAC learning; input paths recheck identity as well. Managed traffic is bound to the recorded MAC and private IPv4 address. Spoofed identity, IPv6, VLAN encapsulation, and other unsupported EtherTypes are dropped. Host-to-guest and peer-to-guest paths are covered, not just routed IP forwarding.

The current TCP allowlist is checked against conntrack's original service destination address/port in both directions, including established service flows. There is no blanket established/related accept before it. Consequently close blocks subsequent packets on existing service connections in both directions after commit, without promising a TCP RST or immediate application disconnect. Live close does not flush conntrack, which would risk reclassifying midstream service traffic as guest egress.

Guest-initiated IPv4 TCP/UDP and replies remain allowed, including outbound DNS. ARP, ICMP echo, and necessary related ICMP errors are supported. Other inbound UDP services are not; HTTP/HTTPS are only TCP labels. Empty managed ports means no inbound TCP service allowance, not quarantine or loss of all egress. Quarantine drops every scoped TAP path, including ARP.

Offline cleanup requires the TAP to be absent. It deletes and verifies absence of conntrack entries scoped to the guest IP in all four original/reply source and destination fields, including NAT reply tuples. Only then may teardown release the address. Rule removal also requires an absent TAP and verified table ownership; it never flushes global firewall or conntrack state. Cleanup uses conntrack's documented -D (delete) and -L (list) commands, including on conntrack-tools 1.4.8, where --list is not a supported option.

Coexistence and Recovery

Existing iptables egress MASQUERADE/forwarding remains unchanged. The new policy adds no DNAT, public listener, or host-port mapping and does not replace the host firewall manager. Public connectivity, DNS, guest reverse proxies/TLS, and integration with UFW/firewalld/Docker/libvirt remain out of scope. An administrator's external routing/NAT can still affect reachability; do not infer source-CIDR restrictions or complete public-exposure protection from the use of private addresses. Operators must choose a non-conflicting topology.

If an external actor loses or changes UVM rules, recovery is explicit ports sync, not a promised daemon repair. An enforced snapshot cannot prevent later root-level changes or prove that other rules allow traffic. FIREWALL.md retains the original proposal/backend evaluation; do not import its unimplemented topology checks into current documentation.

12. Installer and Host Commands

Installer is the only owner of apt installation, Firecracker artifact handling, downloads, and integrity-manifest publication.

Settings detects and normalizes the host architecture at construction time (amd64 to x86_64, arm64 to aarch64) and rejects unsupported architectures. The same architecture selects the Firecracker release archive, guest kernel, rootfs, and architecture-specific boot arguments. Default guest URLs use Firecracker's img/quickstart_guide/<architecture>/ assets: kernels/vmlinux.bin and rootfs/bionic.rootfs.ext4. Kernel and rootfs URL overrides are independent; custom assets and trusted digests must match the host architecture. ARM64 uses an uncompressed Linux Image (PE format) and an ARM64 ext4 rootfs.

The host package list includes nftables and conntrack as well as the existing iproute2/iptables, ext4, OpenSSL, OpenSSH, download, and util-linux tools. UVM adds no reconciliation daemon or tested external firewall-manager integration profile.

system.py centralizes host command execution:

• CommandRunner.run() converts subprocess failures into CommandError.
• require_root() is called by host-mutating operations.
• check_kvm() validates the KVM device.
• ensure_data_directories() creates the UVM storage layout.

Use CommandRunner rather than direct subprocess.run in installation or network code. This keeps command construction observable in the CLI and mockable in tests.

Downloads use temporary sibling files, optional checksum verification, and atomic replacement. Archive extraction rejects path traversal before using Python's safe tar extraction filter.

13. HTTP API

Server Startup

See USAGE.md for dependency installation, loopback deployment, TLS deployment, and authenticated curl examples. Launch through uvm --serve using the Python environment with the installed API dependencies. Every server requires UVM_API_TOKEN, even at 127.0.0.1. Non-loopback binds additionally require UVM_API_TLS_CERT and UVM_API_TLS_KEY; paired certificate/key files are loaded into an ssl.SSLContext before Uvicorn starts.

The token must contain visible ASCII characters without whitespace. It is never accepted as a UVM command-line argument, so it does not appear in UVM process arguments. If you type an export command interactively, your shell may record it in history; use a protected environment file, service manager, or secret manager when that matters.

API Contract

Method	Route	Router	Notes
GET	/	server.py	Redirect to /dashboard
GET	/dashboard	dashboard.py	Public static dashboard shell
GET	/health	health.py	Unauthenticated liveness response
GET	/diagnostics	diagnostics.py	Host readiness checks
POST	/install	installation.py	{ "force": false } body
GET	/vms	vms.py	List persisted VM records and observed status
POST	/vms	vms.py	cpu, ram, storage, guest_ip, username, password, name, ports, no_default_ports
POST	/vms/actions	vms.py	Fixed-ID snapshot bulk lifecycle action
GET	/vms/{identifier}	vms.py	Inspect by exact ID, name, or guest IP
POST	/vms/{identifier}/start	vms.py	Boot a retained private disk
POST	/vms/{identifier}/restart	vms.py	Restart; optional timeout query parameter
POST	/vms/{identifier}/resize	vms.py	Partial cpu/ram/storage update; stop, resize, boot; optional timeout
POST	/vms/{identifier}/stop	vms.py	Stop; optional timeout query parameter
DELETE	/vms/{identifier}	vms.py	Remove runtime files/state; optional timeout query parameter
POST	/vms/{vm_id}/ports	vms.py	Exact ID only; body has mode and ports
POST	/vms/{vm_id}/ports/sync	vms.py	Exact ID only; reapply managed policy
GET	/vms/{identifier}/logs	vms.py	Bounded non-following log tail

All management routes use get_authorized_application() and require:

X-UVM-Token: <UVM_API_TOKEN>

Request schemas reject unexpected fields. UvmError instances are rendered as JSON errors by server.py; Pydantic request validation uses FastAPI's normal 422 response format. Create accepts name: str | null, ports: list | null, and strict boolean no_default_ports (default false). Port objects contain a strict integer port in 1-65535 and uppercase protocol TCP/HTTP/HTTPS (TCP default). The router normalizes/deduplicates ports and adds TCP/22 unless default ports are disabled, even for null/empty input. It never creates an unmanaged VM from an omitted policy. Names share CLI validation.

Create accepts storage (default "30G"). Resize requires at least one non-null cpu, ram, or storage; identity/credential fields are rejected by its schema. Responses expose applied storage_mib and a nullable, resource-only pending_resize object so interrupted requests are visible without secrets.

VM responses include name, ports, policy_status, and operation alongside resource/status fields and username. The schema excludes password and disk/socket/config/log path fields; error messages may still contain paths. ports: null in a legacy response still means unmanaged. Start/restart/stop and port-policy mutations report the resulting policy status. Restart, stop, resize, and delete accept an optional finite non-negative timeout query parameter; there is no dry-run HTTP schema. Port mutation bodies use {"mode":"set"|"open"|"close", "ports":[...]} and always resolve the target by exact ID to prevent name/IP reuse. A legacy policy must be set while stopped before it can be started or synced. Typed not-found/conflict errors map to 404/409.

Log reads accept a tail query parameter from 0 through 1000 and never follow the file. LifecycleService.read_log_tail() reads at most the final 256 KiB and marks a response truncated when an oversized line or that bound prevents a full tail. Bulk bodies use an action plus either explicit identifiers or all_vms; the router resolves them before run_many() executes the fixed IDs under one operation lock. The API does not expose interactive SSH, raw Firecracker calls, arbitrary command execution, or host path access. Public /, /dashboard, /docs, /redoc, and /openapi.json do not perform privileged management requests.

14. Configuration Reference

The environment-variable table is in README.md; config.py is the implementation authority. Keep the same base/network/gateway/bridge settings across CLI and server operations. The network identity is retained per VM and checked before resume.

Settings validates the gateway, bridge-name length, API token format, and environment boolean values at process startup. Add configuration fields there, not as module-level global constants in feature code.

15. Error Handling Rules

Expected failures inherit from UvmError:

Error	Use
ConfigurationError	Invalid local settings or environment
ValidationError	Invalid CLI/API resource input
StateError	Unsafe or malformed persisted state
NotFoundError	Missing registry entry
ConflictError	Ambiguous identifier, allocation conflict, or unsafe/pending operation
CommandError	Required host command failure
FirecrackerError	VMM process or Unix API failure
TapCreationError	TAP setup failed after interface creation

CLI code catches UvmError and prints uvm: error: ... without a traceback. Invalid CLI input returns 2, ordinary operational failure 1, and interruption 130. Batch JSON retains per-target outcomes rather than pretending a partial success was atomic.

The FastAPI server maps the same class to an error envelope. New expected operational failures should use this hierarchy rather than leaking raw OSError, CalledProcessError, or implementation-specific exceptions.

16. Concurrency and Ownership Invariants

These invariants are critical:

1. Only LifecycleService may coordinate VM state, disks, TAPs, firewall, and Firecracker as one operation.
2. Only StateStore may read/write state.json.
3. All long host mutations acquire operations.lock.
4. All short inventory mutations acquire state.lock.
5. The installer shares operations.lock with lifecycle actions, preventing a Firecracker binary replacement between verification and launch.
6. A VM rootfs is private before it becomes writable.
7. Process signaling requires a safe pidfd and matching persisted start/boot identity; unknown is not gone. Legacy adoption must prove binary/socket identity.
8. A new VM never adopts a pre-existing TAP or activates it before verified policy.
9. A server must have an API token; a remote server must also have usable TLS.
10. Routers call application services; they do not run shell commands or touch state files directly.
11. Plaintext guest passwords remain in the mode-0600 registry and never enter Firecracker config, helper command arguments/traces, or CLI/API VM output.
12. Retained-disk start never overwrites guest data, credentials, or SSH host keys.
13. No disk/IP release before confirmed exit and scoped network cleanup; no old-policy rollback after a failed close, and no resurrection of pending deletion.
14. Policy state is observed from the actual owned rules, not a cached applied flag.

When a change would violate one of these rules, redesign the change rather than adding a special-case bypass.

17. Testing Strategy

Standard Unit Suite

python3 -m pip install -e '.[test]'
python3 -B -m unittest discover -s tests -v

Ordinary tests require no root, KVM, Firecracker, or host network mutations. They cover validation, schema migration/locking/permissions, artifact trust, private disks/credentials, network/firewall rendering and readback, conntrack semantics, pidfd/legacy identity, lifecycle recovery/resume/bulk targeting, CLI parsing/output, and server security. Mock/fixture policy checks do not prove the kernel's actual packet behavior.

FastAPI route tests are conditionally enabled when both fastapi and httpx are installed. They should be enabled in CI using the test extra. When nft and conntrack are installed, command-parser tests append an invalid final command/option so parsing is checked without executing any mutation. These checks supplement fixture-based readback and cleanup tests; they do not exercise the kernel packet path.

test_storage.py also exercises real e2fsprogs against temporary unmounted, sparse ext4 images when the tools are installed: default 30 GiB creation, growth, shrinking, geometry, and retained data/password-hash/host-key fixtures. Lifecycle resize tests use fake processes/networking to verify ordering, single-lock ownership, identity preservation, pending intent, and failure recovery.

Isolated Packet Tests

The opt-in packet harness must run only in disposable isolated user/network namespaces. It verifies namespace separation before creating links or changing rules, and never falls back to the parent host namespace. It may require privileges (or enabled unprivileged user namespaces) plus unshare, ip, bridge, nft, conntrack, and sysctl:

UVM_FIREWALL_INTEGRATION=1 python3 -B -m unittest discover -s tests/integration -v

The harness uses bridge-connected veths in place of guest TAPs and routed egress without NAT. It exercises host/peer service traffic, both-direction close of existing streams while preserving conntrack/egress, IPv6/spoof blocking, quarantine, and offline cleanup. It does not boot a microVM or validate real TAP/NAT deployment.

The packet test skipped in this workspace because namespace creation was denied (Operation not permitted). Document that permission limitation, not an assertion that real packets or a real host were validated. Unit tests and nft JSON parser checks are not substitutes for kernel packet-path execution.

Isolated Host Validation

No current test automatically boots a real microVM. Before a release, validate on an isolated Linux host:

1. Strict checked installation.
2. create and successful guest boot.
3. Guest IP reachability and egress.
4. SSH with unique guest host keys.
5. Stop/start/restart with retained guest data, SSH keys, and credentials.
6. Live private-port close in both directions, empty policy, and explicit sync after deliberately lost rules inside the isolated setup.
7. Delete and IP/TAP/policy/conntrack cleanup.
8. API launch with token and TLS on a non-loopback test address.
9. Firecracker crash, interrupted create/start/stop/delete, and legacy recovery.

18. Common Development Tasks

• CLI: keep business logic in services; wire arguments in build_parser() and dispatch/output in _run_command(). Test validation, prompts, dry runs, JSON, and service behavior; use UvmError for expected failures.
• HTTP: define strict models in api_models.py, use the appropriate router and get_authorized_application(), and register new routers in create_api(). Add API tests and update all route tables. New CLI commands do not implicitly authorize new HTTP endpoints; /health remains the only unauthenticated operational API route, while / and /dashboard only serve the public dashboard shell.
• Host primitives: use the specific network/firewall/install/process module and CommandRunner for external commands. Define ownership, idempotence, and compensation before invoking a primitive from lifecycle code. Use fake runners rather than requiring privileges in unit tests.
• State: update VmRecord/State, schema tests, cleanup, and documentation. Version incompatible interpretations and preserve older data or provide an explicit migration. Consider lock order and crash/retry behavior first.

19. Debugging Guide

Symptom	First places to inspect
guest assets missing	/var/lib/uvm/images, integrity variables, integrity.json
Firecracker exits early	VM firecracker.log, VM config, /dev/kvm access
VM is dead in list	PID/start token in state, /proc/<pid>, VM log
Cannot stop legacy VM	Missing start/boot identity or failed protected binary/socket match; never guess a PID
Guest has no egress	ip route show default, bridge address, TAP state, iptables rules
Policy missing/drift/unknown	Actual nft JSON and ownership, STP/group mask, permissions; explicit sync after fixing cause
Retained-disk start fails	disk_ready, safe private path, kernel/VMM integrity, process identity, retained network, pending operation
API server refuses to start	UVM_API_TOKEN, requested host, TLS cert/key files
API returns 401	X-UVM-Token header mismatch
API returns 422	Request schema or UVM input validation failure

Use inspect VM --json, logs VM, ports list --all, and doctor before examining host state. USAGE.md lists read-only bridge/nft/iptables debugging commands. Never share the credential registry or unredacted guest logs. Do not manually mutate UVM resources during a lifecycle operation; fix the service/host failure and retry through supported commands.

20. Current Production Gaps

The code is intentionally honest about current limits:

• Firecracker is launched directly, even though the Jailer binary is downloaded.
• No cgroup CPU, memory, I/O, or process limits are applied.
• Fractional CPU requests round vCPUs up without host quota enforcement.
• Guest health is inferred from VMM process state, not application readiness.
• No quotas, DHCP, host-port mapping, UDP/IPv6 services, or network tenancy.
• No guest snapshots, extra disks, production image builder, or SSH-key injection.
• Per-VM private TCP ingress policy exists, but public connectivity, DNS, guest reverse proxy/TLS, and external firewall integration remain out of scope.
• Existing egress iptables rules are incremental and unchanged; only the private bridge policy is atomically rendered per VM.
• No autostart, background reconciliation, or guest-coordinated shutdown.
• The JSON inventory is suitable for one host, not distributed coordination.
• Guest passwords are stored in plaintext in the root-only local registry.
• API authentication is a single bearer token, not a user/tenant model.

Do not market UVM as a hardened multi-tenant platform until these boundaries are deliberately addressed with architecture, tests, and operational review.

21. Contributor Checklist

Before submitting a change:

1. Run the complete unit suite.
2. Run git diff --check.
3. Confirm a new feature stays inside the uvm/ project directory.
4. Verify command/API errors use UvmError rather than raw tracebacks.
5. Verify host mutations have a lock owner and cleanup path.
6. Verify persistent state remains readable across upgrades.
7. Update README.md for operator-facing behavior.
8. Update this document for architectural, trust-boundary, or lifecycle changes.