nest/DEVELOPER.md

# Nest Developer Guide

This document is the developer handoff for Nest and its UVM backend. Read it
before changing lifecycle behavior, host networking, Firecracker launch
semantics, persistent state, or the HTTP API.

Nest owns the workspace dashboard and its server entry point. Its `uvm`
component is a local, privileged Firecracker microVM manager with a CLI and
standalone FastAPI management API. These are application components, not a
general-purpose Python library or a multi-host control plane.

## 1. Goals and Boundaries

UVM installs host/guest assets, provisions private disks, and manages named
VMs and private IPv4 TCP allowlists on one host. The CLI offers retained-disk
start/stop/restart, CPU/RAM/storage resizing, deletion, fixed-target bulk operations, inspection, SSH,
logs, doctor, and explicit port edits/sync. The UVM API and Nest dashboard
expose health, installation, VM lifecycle controls, and managed port-policy
controls, host diagnostics, bounded VM-log tails, and fixed-target bulk
operations, and bounded guest command execution over vsock. Interactive SSH remains CLI-only.

There is no autostart or background reconciler, multi-host control plane,
tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port
filtering is not public-port publishing or a hardened multi-tenant boundary;
see Current Production Gaps for the remaining scope limits.

Treat UVM as a privileged local systems tool. It manipulates `/dev/kvm`, TAP
interfaces, bridge addresses, nftables/conntrack and existing iptables egress
rules, host filesystems, and processes. For the complete operator command and
flag reference, see [USAGE.md](USAGE.md).

## 2. First Run

All Nest and UVM files live in this project. Start here:

```sh
python3 -m pip install -e '.[test]'
python3 -B -m unittest discover -s tests -v
./nest.py --help
./uvm.py --help
```

The `nest` and `uvm` console scripts are registered by `pyproject.toml`.
During source development, `./nest.py` and `python3 -B -m nest` are equivalent
Nest entry points; `./uvm.py` and `python3 -B -m uvm` launch UVM.

Requirements for real host operations:

- Python 3.12+
- Linux with KVM enabled and `/dev/kvm` accessible
- Linux >= 5.3 with native bridge conntrack and safe pidfd support
- Root privileges for installation and lifecycle mutations
- `iproute2`, `iptables`, `nftables`, `conntrack`, `coreutils`, `e2fsprogs`, `util-linux`,
  `openssl`, and OpenSSH client tools
- An x86_64 or ARM64 (`aarch64`) host, including NVIDIA DGX Spark

Do not run real integration operations on a developer workstation unless the
bridge, subnet, firewall changes, and guest workload are acceptable there.
Packet integration tests must remain inside disposable isolated namespaces.

## 3. Project Layout

```text
./
├── DEVELOPER.md              # This document
├── README.md                 # Project overview
├── USAGE.md                  # Operator command/deployment guide
├── pyproject.toml            # Package metadata and dependencies
├── nest.py                   # Executable Nest source-tree launcher
├── nest/
│   ├── __main__.py           # `python -m nest`
│   ├── cli.py                # Nest workspace serve command
│   ├── server.py             # Nest composition of the dashboard and UVM API
│   ├── sessions.py           # Live guest terminals and browser-independent session lifecycle
│   ├── session_store.py      # Private SQLite session history and bounded transcripts
│   ├── session_routes.py     # VM-scoped session HTTP and WebSocket endpoints
│   └── dashboard/
│       ├── router.py         # Nest dashboard document route
│       └── static/           # Packaged HTML, CSS, JavaScript, and fonts
├── uvm.py                    # Executable UVM source-tree launcher
├── uvm/
│   ├── __main__.py           # `python -m uvm`
│   ├── cli.py                # argparse, CLI rendering, serve dispatch
│   ├── app.py                # Composition root
│   ├── config.py             # Settings and environment loading
│   ├── domain.py             # PortRule, VmSpec, VmRecord, State
│   ├── errors.py             # Expected error hierarchy
│   ├── validation.py         # Resources, names, credentials, ports, timeouts
│   ├── system.py             # CommandRunner, root/KVM/directory checks
│   ├── state.py              # JSON state, atomic writes, file locks
│   ├── integrity.py          # SHA-256 and integrity manifest logic
│   ├── install.py            # Apt, Firecracker, kernel, rootfs installation
│   ├── images.py             # Template lookup, private disks, and guest credentials
│   ├── known_hosts.py        # Deletion-only cleanup of local SSH host-key caches
│   ├── guest_agent.py        # Standalone guest daemon, job scopes, and bounded wire protocol
│   ├── guest_exec.py         # Host-side guest sessions and execution results
│   ├── guest_terminal.py     # Async PTY transport over a process-pinned guest connection
│   ├── network.py            # IP/MAC/TAP/bridge/NAT/forward rules
│   ├── firewall.py           # Owned nft bridge tables and offline conntrack cleanup
│   ├── lifecycle.py          # Lifecycle, bulk operations, recovery, port intent
│   ├── api_models.py         # Pydantic HTTP request/response models
│   ├── server.py             # FastAPI factory and Uvicorn launcher
│   ├── routers/
│   │   ├── diagnostics.py    # Host diagnostic endpoint
│   │   ├── dependencies.py   # API application/token dependencies
│   │   ├── exit.py           # Authenticated server shutdown
│   │   ├── health.py         # Liveness endpoint
│   │   ├── installation.py   # Installation endpoint
│   │   └── vms.py            # VM lifecycle endpoints
│   └── firecracker/
│       ├── config.py         # Firecracker JSON and boot arguments
│       ├── api.py            # Unix-socket HTTP client
│       └── process.py        # Process launch, liveness, termination
└── tests/                    # Unit/API tests; integration/ has opt-in namespace packets
```

Keep feature code in the most specific existing module. Do not add a new
abstraction layer merely to wrap one function. The current module boundaries
are intentionally small and operationally focused.

## 4. Architecture Overview

```mermaid
flowchart LR
  DASH[Nest dashboard] --> NEST[nest/server.py workspace]
  NEST --> HTTP[UVM management API]
  CLI[CLI: uvm] --> APP[uvm/app.py composition root]
  HTTP --> APP
  APP --> INSTALL[Installer]
  APP --> LIFE[LifecycleService]
  APP --> STATE[StateStore]
  LIFE --> IMAGES[ImageStore]
  LIFE --> NET[NetworkManager]
  NET --> FW[FirewallManager]
  LIFE --> FC[Firecracker adapters]
  INSTALL --> RUNNER[CommandRunner]
  NET --> RUNNER
  FW --> RUNNER
  FC --> HOST[Linux host primitives]
  RUNNER --> HOST
  STATE --> DISK[(state.json)]
  IMAGES --> DISKSTORE[/var/lib/uvm/images and vms]
  FC --> KVM[/dev/kvm]
```

The public entry points are thin adapters. Nest composes the UVM API and adds
the workspace dashboard; UVM does not import Nest. The lifecycle layer is the
only place that coordinates multiple host resources for a VM. Lower-level modules
must not import `cli.py`, `server.py`, or `lifecycle.py`.

Leaf modules are `config.py`, `domain.py`, `errors.py`, and `validation.py`.
They must remain free of subprocess, network, CLI, and FastAPI side effects.

## 5. Entrypoints and Request Paths

### CLI Path

`cli.py` builds the application, validates arguments, and calls lifecycle
services. It owns confirmation, terminal/JSON output, exit handling, SSH
`execvp`, local log streaming from a validated path, and `--serve` dispatch.
It must not coordinate state files, interfaces, or Firecracker sockets itself.

The local surface is install/create/list/inspect/start/stop/restart/resize/delete
(`destroy` alias)/ssh/logs/doctor and ports list/open/close/set/sync. Identifiers
resolve by exact ID, name, or guest IP, never an ID prefix. Generated-ID-shaped
inputs resolve only by ID; names cannot impersonate generated IDs. Ambiguous
non-ID aliases raise `ConflictError`.

Create alone adds `22:TCP` unless `--no-default-ports` is supplied. Port syntax
is `PORT[:TCP|HTTP|HTTPS]`, comma-separated or repeated via `--ports`; all labels
use TCP. `--password-stdin` reads one line instead of using a password argument.
`_vm_output()` is an explicit output whitelist, not `VmRecord.to_dict()`.
JSON stays on stdout, while warnings, prompts, and host traces use stderr.

Bulk lifecycle actions and ports sync resolve/deduplicate IDs before prompting.
Delete and mutating `--all` selections require a TTY confirmation or `--yes`/`-y`.
Port set/close-all and removal of TCP/22 also prompt, except empty managed
no-ops. `--dry-run` previews without lifecycle/policy mutations or confirmation;
it is not a full preflight. Stop/restart/resize/delete accept `--timeout`; start does
not. Logs follow does not hold the operation lock. Doctor runs readiness
checks, not per-VM repair or guest application probes.

### HTTP Path

`nest/server.py` owns the workspace application. Its `create_app()` composes
`uvm.server.create_api()` with the dashboard document and static assets from
`nest/dashboard/`. The same-origin API paths and `X-UVM-Token` authentication
are shared with the standalone UVM API.

Routers validate the token and Pydantic request, invoke application services,
and render whitelisted response models. Lifecycle, managed port-policy,
diagnostics, and bounded log-tail operations have explicit HTTP routes for the
dashboard. Bulk operations have a dedicated fixed-target HTTP route; interactive
SSH has no HTTP endpoint. The public dashboard shell does not bypass token
validation: it stores a supplied token only for the active browser session and
sends it with management requests.

Launch the workspace with `nest --serve` or `./nest.py --serve`. The standalone
backend uses `uvm --serve` or `./uvm.py --serve`. Both `create_app()` and
`create_api()` require an explicit host so validation covers the intended bind.
Nest passes its factory to the shared UVM server runner, which enforces token
and TLS requirements and wires graceful server shutdown.

## 6. Application Composition

[`app.py`](uvm/app.py) constructs one concrete object graph per CLI or server
process:

| Component | Constructor | Responsibility |
|---|---|---|
| Settings | `Settings.from_environment()` | Immutable runtime configuration |
| CommandRunner | `CommandRunner` | Logged/testable subprocess execution |
| StateStore | `StateStore` | Inventory, locks, JSON persistence |
| ImageStore | `ImageStore` | Template verification, VM disks, and offline credentials |
| KnownHostsManager | `KnownHostsManager` | Scoped local SSH cache cleanup before deleted IPs are released |
| NetworkManager | `NetworkManager` | Guest allocation and Linux networking |
| FirewallManager | Constructed by `NetworkManager` | Owned native bridge policy and scoped offline conntrack cleanup |
| FirecrackerProcessManager | `FirecrackerProcessManager` | VMM process control |
| LifecycleService | `LifecycleService` | Ordered VM resource changes and rollback |
| Installer | `Installer` | Host packages and artifacts |

Composition can disable command tracing with `emit=None`; `create_api()` does
so when it constructs its own application. The CLI supplies an emitter that
prints `+ ...` traces to stderr, including when passing its application to
the server launcher.

## 7. Persistent Data and Models

### Host Data Layout

Default runtime storage is `/var/lib/uvm`:

```text
/var/lib/uvm/
├── bin/
│   ├── firecracker
│   └── jailer
├── images/
│   ├── vmlinux
│   ├── ubuntu.ext4
│   └── ubuntu-build.log
├── vms/
│   └── vm-<uuid>/
│       ├── rootfs.ext4
│       ├── config.json
│       ├── firecracker.log
│       ├── firecracker.sock
│       └── vsock.sock
├── state.json
├── state.lock
├── operations.lock
└── integrity.json
```

Per-VM runtime directories are mode `0700`; copied VM disks are mode `0600`.
`state.json` is mode `0600` because VM records contain plaintext guest
passwords. Registry-backed commands therefore need access to the root-owned
state file.

### State Schema

[`domain.py`](uvm/domain.py) is the schema authority. The default registry is
`/var/lib/uvm/state.json`; its current schema version is **3**. An empty registry:

```json
{
  "schema_version": 3,
  "next_mac_index": 1,
  "vms": {}
}
```

`VmSpec` is the validated create request. `VmRecord` is the persisted local
record. Do not make API models or CLI namespaces the persistence format.

| Record Fields | Purpose |
|---|---|
| `id`, `name`, `cpu`, `ram_mib`, `storage_mib` | Stable identity, name, applied resources; storage is null for older disks |
| `pending_resize` | Null or a validated `VmResources` target with CPU, RAM MiB, and storage MiB |
| `guest_ip`, `gateway`, `network`, `bridge`, `tap`, `mac` | Retained network identity/configuration |
| `socket`, `config`, `log`, `disk` | Canonical per-VM runtime paths |
| `username`, `password` | Guest credentials; password stays out of public output |
| `ports` | Null for legacy/unmanaged, or a list of `{ "port": 80, "protocol": "HTTP" }` objects |
| `disk_ready`, `runtime_owned`, `tap_owned` | Completed provisioning and resource ownership evidence |
| `status`, `operation`, `spawn_pending`, `last_error` | Durable state, pending intent, ambiguous-spawn guard, failure detail |
| `pid`, `process_start_time`, `boot_id` | VMM identity, including host reboot detection |
| `created_at`, `updated_at` | Registry timestamps |

`PortRule` accepts TCP/HTTP/HTTPS labels and integer ports 1-65535. A port
number is unique regardless of label. `VmSpec.ports` defaults to TCP/22,
whereas `VmRecord.ports=None` deliberately preserves the legacy distinction.
An explicit empty tuple/list is a managed empty TCP allowlist, not unmanaged
mode. There is no persisted boolean asserting that a policy is enforced.

### State Guarantees

- `state.json` writes use a sibling temporary file, `fsync`, atomic replace,
  and parent-directory `fsync`.
- `state.json` is mode `0600`; it must be treated as a credential registry and
  excluded from logs and bug reports.
- `state.lock` serializes short JSON read-modify-write transactions.
- `operations.lock` serializes long lifecycle, bulk, policy, and installation operations.
- State reads are lock-free because atomic replacement gives readers either a
  complete old file or a complete new file.
- Legacy versions 0 (including unversioned state), 1, and 2 are loaded into schema
  3 in memory and written as schema 3 on the next transaction. Newer schemas
  are rejected. Reads do not reconcile VMs or adopt process identities.
- Missing `name`, `ports`, and `boot_id` remain null. No policy or default SSH
  rule is silently imposed on a legacy VM. Use stop, explicit `ports set`,
  then start; first policy adoption requires a stopped, proven-gone VM.
- Missing `disk` and process start time remain absent. Legacy `disk_ready` is
  inferred only for running/stopped records with a recorded private disk;
  start still validates the path, file, and provisioning state.
- Legacy records own their canonical runtime directory independently of whether
  they had a private disk. Deletion never removes the shared rootfs template;
  explicit `runtime_owned=False` on a new interrupted creation stays authoritative.
- Records created before credential provisioning default to username `root`
  and password `null`; UVM does not apply a password retroactively.
- Missing `storage_mib` and `pending_resize` remain null. The first explicit
  resize infers an older disk's capacity and enforces the 30 GiB minimum; reads
  and schema migration never resize a disk. Schema 3 prevents older UVM versions
  from ignoring pending resource changes during a boot.
- `next_mac_index` is reconstructed from older locally administered MACs when
  absent.

When adding a persisted field, update `VmRecord.to_dict`, `VmRecord.from_dict`,
the state schema tests, and this document. Never silently discard existing
state fields without a migration plan.

## 8. VM Lifecycle

### Stored and Observed Status

The persisted status is a string, not an enum, to keep legacy JSON readable.

| Status | Meaning |
|---|---|
| `starting` | Identity reserved; host resources are being created |
| `running` | Firecracker configuration and `InstanceStart` succeeded |
| `stopping` | Stop mutation is in progress |
| `stopped` | Process/TAP removed; private disk remains |
| `terminating` | Destruction is in progress |
| `failed` | A lifecycle step or cleanup failed; `last_error` explains why |
| `dead` | Observed-only status when a stored running process is proven gone |
| `unknown` | Observed process identity cannot be established safely |

Transition statuses can survive interruption. `dead`/observed `unknown` do
not rewrite inventory. Delete removes the record rather than keeping a
tombstone. Pending deletion remains authoritative even if a subsequent stop
finishes runtime cleanup; start and policy edits must not resurrect it.

`ListedVm` separates the record, observed process status, and observed policy
status. Policy can be `enforced`, `missing`, `drift`, `quarantined`, `unknown`,
legacy `unmanaged`, or `inactive` for a cleanly stopped managed VM. Readback is
a ruleset/attachment snapshot, not application health or a reachability test.

### Create Order

[`LifecycleService.create`](uvm/lifecycle.py) follows this sequence:

1. Validate the request/root access and acquire `operations.lock`.
2. Check KVM, verified Firecracker, firewall capability, and installed templates.
3. Reserve ID/name/IP/MAC/ports with `status=starting`, `operation=preparing`.
4. Create the owned mode-0700 runtime directory, sparse-copy a mode-0600 private
   disk, resize ext4 to the requested storage capacity, provision credentials/host
   keys and the managed guest network profile, then persist `disk_ready=True`.
5. Ensure bridge/egress networking and prepare an owned, attached TAP **down**.
6. Apply/read back the nftables bridge policy, then activate the TAP.
7. Write Firecracker config; persist `operation=spawning`, `spawn_pending=True`,
   and the host boot ID before `Popen`.
8. Publish PID/start time/boot ID through `on_spawn` before socket readiness
   or API configuration; clear the pending-spawn guard.
9. Configure Firecracker and `InstanceStart`, verify policy enforcement again,
   then save `running` with no pending operation.

This records both intent and acquired ownership across crash windows. Never
make a managed TAP usable before enforcement or mark a partial disk ready.

### Rollback Rules

Any `Exception`, `KeyboardInterrupt`, or `SystemExit` during create triggers
compensation:

1. Attempt quarantine/TAP deactivation and confirm safe process exit.
2. Remove the owned TAP, clean up scoped conntrack entries, and remove its policy.
3. Remove the owned runtime directory/private disk only after confirmed exit.
4. Remove the reservation only if cleanup succeeds.
5. Otherwise retain a `failed` record with identity/intent and `last_error`;
   do not release its disk or IP for reuse.

Do not reorder these operations without considering leaked TAPs, orphaned VMM
processes, and allocation reuse.

### Start and Restart

To boot a stopped/proven-gone VM, start requires managed ports, a complete safe
private disk, and compatible retained gateway, network, and bridge settings.
It verifies the VMM and installed **kernel**; the original rootfs template is
not needed. It never recopies the rootfs, reprovisions credentials, or
regenerates guest SSH host keys. Name/IP/MAC, ports, and guest disk changes
survive stop/start; guest RAM does not.

After runtime cleanup confirms process exit and any pending resize finishes,
start specializes an original UVM Ubuntu network profile with the retained
guest address/gateway. This one-time migration repairs older guests that lost
their default route. It uses a checked sparse copy of the retained private
disk; it never replaces that disk with the shared template. Guest-edited
profiles are preserved. The image layer's network checks and writes run only
under the lifecycle operation lock, before the TAP and VMM are started.

Start on a healthy running VM validates its retained network, synchronizes
policy, then restores shared bridge/NAT/forwarding rules without spawning another
VMM. A shared-network setup failure retains the running VM for retry. An ambiguous
process or interrupted live boot/stop is not adopted as a successful start.
Failed resume retains the private disk for retry. Restart
performs stop and start under one operation lock, not two independent public
calls with a race between them.

### Resource Resizing

`resize()` accepts a partial CPU/RAM/storage update for one VM. New disks default
to 30720 MiB (30 GiB), also the minimum accepted storage. Storage parsing uses
GiB for bare numbers, binary B/K/M/G/T suffixes, and rounds up to MiB; RAM retains
its MiB-default parsing. Capacities must fit the host's signed 64-bit file offsets.

One `operations.lock` spans preflight, durable `pending_resize` publication,
`_stop_locked()`, offline resource application, and `_start_locked()`. Stop must
prove process exit before any disk work. The complete target is a validated
`VmResources`, while the record's CPU/RAM/storage fields remain the applied values.
Only the resource fields change; identity, networking, credentials, ports, and
guest SSH keys are retained. Even an initially stopped VM boots after resize.

`ImageStore.resize_vm_disk()` sparse-copies the private disk to a mode-0600 sibling,
runs `e2fsck -f -p`, resizes ext4, verifies its geometry using `dumpe2fs`, fsyncs,
then atomically replaces the original and fsyncs the directory. Growth extends
the image before `resize2fs`; shrinking truncates it only after ext4 fits and its
geometry matches. Failed staging leaves the original disk intact. CPU/RAM-only
resizes avoid disk work when its configured capacity is already known.

After disk publication, one state write applies all resource fields and clears
`pending_resize`; boot then follows the normal start path. A crash between disk
publication and that state write is recoverable: the target is durable and the
image layer recognizes an already-completed capacity. A stopped VM's `start`
retries a pending resize before boot; a live pending resize requires `resize`
or `stop` first. Stop retains the target and deletion never applies it. A failed
boot after successful resizing retains the new resources for retry.

### Stop and Delete

```text
stop:
  persist stop intent -> quarantine/deactivate TAP -> confirm VMM exit
  -> delete owned TAP -> scoped conntrack cleanup -> remove owned nft table
  -> stopped (retain disk, credentials, name/IP and desired ports)

delete (destroy alias):
  persist delete intent -> same runtime cleanup -> remove owned runtime directory
  -> clean local known_hosts entries -> remove state record -> release name/IP
```

Stop signals the **VMM**, not a guest shutdown agent or ACPI-coordinated
shutdown. SIGTERM is followed by SIGKILL when needed. `--timeout` is the
per-VM TERM wait, default 5 seconds, finite and non-negative; it is not a total
deadline. Even zero still requires confirmed kernel exit before cleanup.
Application consistency requires separately quiescing the guest.

Runtime cleanup retains reservations when process exit, TAP ownership, or
connection cleanup is uncertain. A pending spawn cannot be declared gone
merely because its PID was never persisted or its socket has not appeared.

`KnownHostsManager` is an explicit lifecycle dependency, also used by API and
bulk deletion. It runs after successful process/network/disk teardown and before
the registry entry is removed, under the same operation lock. Cleanup errors
retain `operation=delete` and the IP reservation; repeating delete retries it.
Stop, restart, and resize retain the existing guest's trusted host-key entries.

Default caches are `known_hosts` and `known_hosts2` under the effective user's
and sudo invoker's account homes (`SUDO_UID`, with a `SUDO_USER` fallback).
`UVM_KNOWN_HOSTS_FILES` overrides this discovery for service deployments using a
colon-separated list of absolute paths. Only local files are managed. Matches
are explicit guest IPs or UVM's `uvm-<VM-ID>` host-key alias, including OpenSSH's
HMAC-SHA1 hashed form. Plain bracketed ports are recognized directly; hashed
bracketed candidates use port 22 and the VM's recorded allowlist. Wildcard rules
and other aliases sharing a line are retained.

File updates use pinned directory descriptors, regular/single-link checks,
an advisory lock, and a sibling temporary file. Owner, group, mode, and extended
attributes are preserved. A changed source snapshot prevents publication over
concurrent edits. A matching-free file is not rewritten and missing files are
not created. Tests inject the dependency to avoid accessing developers' caches,
and verify hashed-entry interoperability using real OpenSSH on temporary files.

### Policy and Bulk Operations

`set_ports()` takes a frozen registry ID, not a reusable name/IP; the CLI resolves
the user's identifier before confirmation. It persists the new allowlist with `operation=policy` before
applying it. Live updates apply atomically and retain conntrack; cleanly
stopped edits are metadata-only. On apply/readback failure, preserve the new
intent, attempt link-down/quarantine, and leave a failed pending record. Never
restore an older, more permissive allowlist after a failed close. An explicit
`ports sync` retries the desired policy and can reactivate a contained live VM.

`run_many()` holds one outer operation lock while processing the fixed ID
snapshot produced before CLI confirmation. It reloads by **exact ID**, not a
reusable name/IP. Results are `succeeded`, `failed`, `skipped` (already absent),
or `interrupted`; ordinary failures do not undo other successes or stop later
targets. Interruptions skip the remaining targets. Read-only views and API
startup never reconcile; there is no autostart or background repair loop.

## 9. Firecracker Integration

The `firecracker/` package has three separate responsibilities.

| File | Responsibility |
|---|---|
| `config.py` | Build boot args and Firecracker JSON; atomically write config |
| `api.py` | Typed HTTP-over-Unix-socket requests |
| `process.py` | Spawn, socket readiness, liveness, signal safety, cleanup |

### Configuration Order

`FirecrackerClient.configure_and_start()` must keep this ordering:

1. `PUT /machine-config`
2. `PUT /boot-source`
3. `PUT /drives/rootfs`
4. `PUT /network-interfaces/eth0`
5. `PUT /vsock` with a guest CID and a per-VM Unix socket path
6. `PUT /entropy` with `{}` (a host-backed virtio RNG device)
7. `PUT /actions` with `InstanceStart`

Every generated config includes `entropy`, including retained-disk boots.
An empty object enables the device with default settings; do not skip it with
a truthiness check. A missing/invalid configuration or an entropy API failure
must prevent `InstanceStart`. Without guest entropy, systemd's random-seed
service and OpenSSH can remain in startup while the VM process and IP respond.

The guest receives a static IP via the kernel `ip=` boot argument. The netmask
comes from `Settings.network`, not a hard-coded `/24` value. The Ubuntu
systemd-networkd profile also receives explicit `Address=` and `Gateway=`
settings so it can maintain the same configuration after boot or reconfiguration.
ARM64 guests also receive `keep_bootcon`, as recommended by Firecracker's
getting-started guide; both architectures use `console=ttyS0`.

### Process Safety

Firecracker starts detached in a new session, appending stdout/stderr to the
per-VM log. Identity consists of PID, `/proc/<pid>/stat` start time, and the
host boot ID. `observe()` distinguishes `alive`, `gone`, and `unknown`;
unreadable evidence is not proof of exit. A changed boot ID or start token can
prove the old process gone without signaling a reused PID.

Start probes safe pidfd support before spawning. Termination opens a pidfd,
rechecks identity, signals through that handle (never a bare PID), and confirms
whole-process exit before unlinking the socket. Python bindings or libc
bindings may supply pidfd support; there is no unsafe PID-signal fallback.
A zombie leader alone does not prove its worker threads have exited.

Legacy adoption is deliberately narrow. During a locked mutation,
`adopt_legacy_identity()` may fill a missing boot ID only when all evidence
matches: a saved numeric start token, a live pidfd-pinned process, the configured
Firecracker executable path **and inode**, the exact `--api-sock` command line,
and root-owned/protected binary, socket, and ancestor directories. It checks
that the process has not exited before returning, and the lifecycle layer
persists the adopted boot ID. Listing/inspection never adopts it.

A live legacy PID without a saved start token, or with mismatched/inaccessible
binary/socket evidence, remains unknown and fails closed. UVM does not guess
an identity from the current occupant of a PID or delete an in-use disk to
make recovery convenient. Proven-gone legacy processes can be cleaned up.

## 10. Guest Images and Integrity

### Template Contract

`images/ubuntu.ext4` is a template only. Each create operation copies it into
the VM runtime directory before UVM provisions the requested account, its
shadow entry, and OpenSSH password policy with `debugfs`. Firecracker only sees
the resulting private copy. Never change that to a shared writable rootfs.
Copies use coreutils `cp --reflink=auto --sparse=always`; the new disk is resized
to the requested capacity before credential provisioning. Existing disk resizes
do not reprovision credentials or host keys.

Guest images must provide `/etc/passwd`, `/etc/shadow`, and OpenSSH's
`/etc/ssh/sshd_config`. Existing accounts retain their identity, groups, home,
shell, and password-aging policy. Missing custom users are created offline,
using `/etc/group` and an available `/bin/sh` login shell. UVM allocates unused
UIDs/GIDs from 1000-60000, accounting for primary GIDs in passwd as well as
group entries, and reuses a same-named group when present. New group entries
also update `/etc/gshadow` if it exists. Account-file metadata is preserved.
The new `/home/USERNAME` directory is mode `0700`, owned by the new UID/GID;
`/home` is created if absent and existing home paths are not overwritten.
Directory ownership/modes and account-file writes are read back before the
disk can be marked ready. The default `root` account must already exist.

Custom login users are VM administrators. Provisioning requires a root-owned,
setuid `/usr/bin/sudo` and `/etc/sudoers`, and appends a user-scoped `PASSWD: ALL`
rule with root/target/runas-password defaults disabled. Sudo authenticates with
the login user's password; root's separate shadow entry is not changed. Global
sshd directives precede managed and existing `Match` blocks so standard Ubuntu
`UsePAM`/`Subsystem` settings remain valid.

UVM replaces conventional RSA, ECDSA, and Ed25519 host keys in each private
disk and removes the known public login key from Firecracker's bionic image.
A VM must never rely on reusable SSH private keys from a shared template.

### Guest Network Profile

`GUEST_NETWORK_TEMPLATE` in `images.py` is shared with the Ubuntu builder. It
uses `KeepConfiguration=yes`: in Ubuntu 22.04's systemd-networkd, `static`
retains permanent addresses and `proto static` routes, but the kernel's
`ip=`-configured default route uses `proto boot` and is otherwise removed.
This can leave SSH to the private IP working while all off-subnet destinations
fail locally with `Network is unreachable`.

`ImageStore.configure_guest_network()` fills in the assigned address/prefix and
gateway during creation and stopped-disk resume. It only specializes the exact
shipped profile from recognized UVM Jammy v1/v2/v3/v4 images. Missing, customized,
symlinked, hard-linked, or unexpectedly owned/mode-set profiles are left intact.
An already-specialized profile is a no-op. Custom images retain responsibility
for their own guest networking.

For migration, the image layer sparse-copies the private disk and runs
`e2fsck -f -p` on the copy. It rechecks the profile after journal replay so
committed guest edits cannot be overwritten based on stale offline reads.
The updated profile is read back and the filesystem checked before fsync and
atomic disk replacement. Failed staging keeps the original disk available for
retry. The built template's recipe marker is v4 (v3 introduced the route fix);
cached older templates are
also handled by per-VM provisioning without requiring an installation refresh.

### Integrity Model

[`integrity.py`](uvm/integrity.py) has two concepts:

- A configured trusted SHA-256 from the environment.
- A local `integrity.json` manifest recording artifact hashes and whether the
  installation was verified.

Strict installation requires these values for downloaded artifacts:

```text
UVM_FIRECRACKER_SHA256
UVM_KERNEL_SHA256
```

The default rootfs is built from signature-verified Ubuntu packages and its
resulting digest is recorded in the same manifest. `UVM_ROOTFS_URL` explicitly
selects a prebuilt ext4 download and additionally requires `UVM_ROOTFS_SHA256`.
A rootfs checksum without a URL is rejected during installation to avoid using
a stale Bionic-image checksum for a local build.

`UVM_ALLOW_UNVERIFIED_DOWNLOADS=1` is a local-development escape hatch. It
writes a manifest marked `verified: false`; a later strict run will not treat
that manifest as publisher trust. Use `install --force` with trusted digests to
refresh an existing artifact without manually deleting managed files.

Retain the explicit opt-out on create/start/restart/resize (or the API server) when
using an unverified development installation. It is not appropriate for
untrusted workloads. A bare strict install requires the trusted digests.

The optional `UVM_FIRECRACKER_BINARY_SHA256` validates a local Firecracker
binary when no verified manifest is available.

## 11. Networking

### Topology

```text
guest eth0
    |
virtio-net / Firecracker
    |
per-VM TAP: uvm-<suffix>
    |
Linux bridge: uvm0 (10.42.0.1/24 by default)
    |
iptables forwarding + MASQUERADE
    |
host IPv4 route for the destination (current uplink)
    |
internet
```

[`network.py`](uvm/network.py) owns address allocation and host attachments;
[`firewall.py`](uvm/firewall.py) owns native bridge filtering:

- IP allocation from `Settings.network`, excluding network, broadcast, and
  gateway addresses.
- Locally administered MAC generation: `02:fc:00:xx:xx:xx`.
- Interface names limited to Linux `IFNAMSIZ` (15 characters).
- Bridge creation or validation that an existing interface is actually a
  Linux bridge.
- Gateway address setup and IPv4 forwarding.
- Egress NAT and forward/return rules scoped to the guest subnet/bridge.
- TAP ownership aliases (`uvm:<id>`), preparation while down, activation,
  deactivation, cleanup, and error reporting.

Network actions are idempotent where practical. Do not adopt an existing TAP:
it may belong to another VM. A create-time name collision is an error. Cleanup
checks the TAP kind and ownership alias, with a narrow unaliased-TAP allowance
for legacy records; it does not treat an arbitrary interface name as ownership.

Internet sharing is configured even with no host default route. Rules never
pin an uplink interface: Linux routing selects it, and MASQUERADE uses that
interface's address. Offline-to-online transitions, reconnects and route changes
therefore work without another UVM command or a monitoring daemon. Existing
connections may need to reconnect after the upstream address changes. Guest
DNS in the Ubuntu template uses the public resolvers documented in README.

`uvm-egress`-commented iptables rules are checked before insertion, with a bounded
xtables-lock wait. Scoped exceptions are inserted at the start of the host chains
so pre-existing terminal drops cannot shadow them. Outbound forwarding requires
the UVM bridge and source subnet; routed replies require the destination bridge,
subnet and established/related conntrack state. Native bridge policy still checks
identity, service ports and permitted protocols. Outbound forwarding also covers
peer traffic if the host already uses `br_netfilter`, while MASQUERADE excludes
the UVM bridge to retain private peer source addresses. Global chain policies
and unrelated rules remain administrator-owned.

### Native Bridge Policy

The supported path is a **software Linux bridge** with STP disabled. Each
managed TAP's bridge-slave `group_fwd_mask` must include `0x4000` (the network
layer sets 16384). This sends LLDP through the early filter instead of a
pre-hook delivery/learning path. Firewall apply/status checks the bridge and
port settings when the TAP exists. Do not use hardware/flow offloads or other
paths that bypass these hooks. This is not an offload-isolation guarantee.

Native bridge conntrack requires Linux >= 5.3. Capability checks run
`conntrack --version`, inspect the kernel release, and ask `nft --json --check`
to validate a full policy without committing it. The nft CLI must support
the rendered JSON. This layer needs no `br_netfilter` sysctl changes.

Each VM owns one `bridge` table named `uvm_fw_<digest>`, derived from the
resolved `UVM_BASE` and exact VM ID, with a full ownership comment. Apply uses
an atomic nft JSON transaction, delete-by-handle for the old owned table, and
exclusive create for its replacement. Foreign ownership is an error, not an
adoption opportunity. Readback normalizes presentation differences but compares
all effective objects, ordered rules, hooks, identity, and the full port set;
rule counts or an owner comment alone are not proof of enforcement.
Normalization materializes implied EtherType constraints before folding
transport-protocol dependencies. This accepts nftables' omitted redundant
`ether type ip` checks while retaining the IPv4 scope of TCP rules.

The policy has prerouting/input/forward/output hooks. Early prerouting identity
checks run at priority -310, before conntrack and bridge MAC learning; input
paths recheck identity as well. Managed traffic is bound to the recorded MAC
and private IPv4 address. Spoofed identity, IPv6, VLAN encapsulation, and other
unsupported EtherTypes are dropped. Host-to-guest and peer-to-guest paths are
covered, not just routed IP forwarding.

The current TCP allowlist is checked against conntrack's **original service
destination address/port in both directions**, including established service
flows. There is no blanket established/related accept before it. Consequently
close blocks subsequent packets on existing service connections in both
directions after commit, without promising a TCP RST or immediate application
disconnect. Live close does not flush conntrack, which would risk
reclassifying midstream service traffic as guest egress.

Guest-initiated IPv4 TCP/UDP and replies remain allowed, including outbound DNS.
ARP, ICMP echo, and necessary related ICMP errors are supported. Other inbound
UDP services are not; HTTP/HTTPS are only TCP labels. Empty managed ports
means no inbound TCP service allowance, not quarantine or loss of all egress.
Quarantine drops every scoped TAP path, including ARP.

Offline cleanup requires the TAP to be absent. It deletes and verifies absence
of conntrack entries scoped to the guest IP in all four original/reply source
and destination fields, including NAT reply tuples. Only then may teardown
release the address. Rule removal also requires an absent TAP and verified
table ownership; it never flushes global firewall or conntrack state.
Cleanup uses conntrack's documented `-D` (delete) and `-L` (list) commands,
including on conntrack-tools 1.4.8, where `--list` is not a supported option.

### Coexistence and Recovery

Shared `iptables` egress MASQUERADE/forwarding is separate from private bridge
policy. The policy adds no DNAT, public listener, or host-port mapping and does
not replace the host firewall manager. Public inbound connectivity, DNS
publication, guest reverse proxies/TLS, and integration with
UFW/firewalld/Docker/libvirt remain out of scope. An
administrator's external routing/NAT can still affect reachability; do not
infer source-CIDR restrictions or complete public-exposure protection from the
use of private addresses. Operators must choose a non-conflicting topology.

If an external actor loses or changes private policy, recovery is explicit
`ports sync`; `start` also restores shared networking for a running VM. An
`enforced` snapshot cannot prevent later root-level changes or prove that other
rules allow traffic.
[FIREWALL.md](FIREWALL.md) retains the original proposal/backend evaluation;
do not import its unimplemented topology checks into current documentation.

## 12. Installer and Host Commands

`Installer` is the only owner of apt installation, Firecracker artifact
handling, downloads, and integrity-manifest publication.

`Settings` detects and normalizes the host architecture at construction time
(`amd64` to `x86_64`, `arm64` to `aarch64`) and rejects unsupported architectures.
The same architecture selects the Firecracker release archive, guest kernel,
native rootfs packages, and architecture-specific boot arguments. The default
kernel URL is pinned to Firecracker's published
`firecracker-ci/v1.15/<architecture>/vmlinux-6.1.155` artifacts. The CI artifact
series is independent of the VMM release. Both architecture configs include
`CONFIG_HW_RANDOM=y` and `CONFIG_HW_RANDOM_VIRTIO=y`; custom kernels need these
built-in drivers for UVM's entropy device. The previous ARM quickstart image
was Linux 4.14.174 without the built-in virtio RNG driver. Updating the device
configuration alone cannot repair that kernel; existing installs need a forced
asset refresh with a matching new trusted kernel checksum, then a fresh VM boot.
An empty `Settings.rootfs_url` selects the built Ubuntu 22.04 (Jammy) template.
Explicit kernel/rootfs URLs and trusted digests must match the host architecture. ARM64
uses an uncompressed Linux `Image` (PE format) and an ARM64 ext4 rootfs.

The default rootfs builder installs `debootstrap` and `ubuntu-keyring`, uses
`--force-check-gpg`, and bootstraps only the base release's main component with
CA certificates and the Ubuntu keyring. Do not pass updates/security as
`--extra-suites`: debootstrap can download base archives but import newer
pre-dependency metadata, producing an empty `dpkg --install` invocation.
After bootstrap, apt sees main/universe plus updates/security, runs update
with `APT::Update::Error-Mode=any`, upgrades the base, and installs guest packages.
amd64 uses `archive.ubuntu.com/ubuntu`; arm64 uses `ports.ubuntu.com/ubuntu-ports`.
Both phases execute in private mount/PID namespaces. The apt phase has private
dev/proc and read-only sys mounts, and recreates `policy-rc.d` to suppress service
startup after Ubuntu's debootstrap script removes it. Cleanup tolerates an
already-removed policy file. No mounts survive the build process.
The staged tree retains `/var/lib/dpkg` and apt metadata, validates installed
apt/sudo/sshd/systemd packages, enables SSH/networkd, preserves kernel-provided
static networking, and supplies guest DNS. Template host keys and machine IDs
are cleared for per-VM initialization. `mkfs.ext4 -d` creates a sparse 2 GiB
template with a guest-kernel-compatible feature set; `e2fsck` and fsync precede
atomic publication. Create still grows each private disk to at least 30 GiB.

Only a checksum-verified manifest and matching `/etc/uvm-image` recipe marker
permit reuse of a built template. Old demo images and unverified caches are
rebuilt, and `install --force` always rebuilds. A failed staging build retains
the old template. Installation holds the existing operation lock throughout;
existing VM disks are never upgraded or reprovisioned by installation.
Stage command output and debootstrap's detailed log are atomically saved to
`images/ubuntu-build.log` (mode `0600`) on success or failure before deleting
staging files. Errors name that durable log; the last lines contain the latest
stage's diagnostics and failure summary. The recipe marker is bumped whenever
the build workflow changes so an earlier cached template cannot skip it.

The host package list includes `nftables` and `conntrack` as well as the
existing iproute2/iptables, ext4, OpenSSL, OpenSSH, download, and util-linux
tools. UVM adds no reconciliation daemon or tested external firewall-manager
integration profile.

`system.py` centralizes host command execution:

- `CommandRunner.run()` converts subprocess failures into `CommandError`.
- `require_root()` is called by host-mutating operations.
- `check_kvm()` validates the KVM device.
- `ensure_data_directories()` creates the UVM storage layout.

Use `CommandRunner` rather than direct `subprocess.run` in installation or
network code. This keeps command construction observable in the CLI and
mockable in tests.

Downloads use temporary sibling files, optional checksum verification, and
atomic replacement. Archive extraction rejects path traversal before using
Python's safe tar extraction filter.

## 13. HTTP API

### Server Startup

See [USAGE.md](USAGE.md) for dependency installation, loopback deployment,
TLS deployment, and authenticated curl examples. Launch the Nest workspace
through `nest --serve`, or the standalone backend through `uvm --serve`, using
the Python environment with the installed API dependencies. Every server
requires `UVM_API_TOKEN`, even at `127.0.0.1`. Non-loopback binds additionally
require `UVM_API_TLS_CERT` and `UVM_API_TLS_KEY`; paired certificate/key files
are loaded into an `ssl.SSLContext` before Uvicorn starts.

The token must contain visible ASCII characters without whitespace. It is
never accepted as a UVM command-line argument, so it does not appear in UVM
process arguments. If you type an `export` command interactively, your shell
may record it in history; use a protected environment file, service manager,
or secret manager when that matters.

### API Contract

The root redirect, dashboard document, and terminal-session routes belong to
Nest. Both server entry points expose the UVM management routes below.

| Method | Route | Router | Notes |
|---|---|---|---|
| `GET` | `/` | `nest/server.py` | Redirect to `/dashboard` (Nest only) |
| `GET` | `/dashboard` | `nest/dashboard/router.py` | Public Nest dashboard shell (Nest only) |
| `GET` | `/health` | `health.py` | Unauthenticated liveness response |
| `POST` | `/exit` | `exit.py` | Request graceful Uvicorn shutdown |
| `GET` | `/diagnostics` | `diagnostics.py` | Host readiness checks |
| `POST` | `/install` | `installation.py` | `{ "force": false }` body |
| `GET` | `/vms` | `vms.py` | List persisted VM records and observed status |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports` |
| `POST` | `/vms/actions` | `vms.py` | Fixed-ID snapshot bulk lifecycle action |
| `GET` | `/vms/{identifier}` | `vms.py` | Inspect by exact ID, name, or guest IP |
| `POST` | `/vms/{identifier}/start` | `vms.py` | Boot a retained private disk |
| `POST` | `/vms/{identifier}/restart` | `vms.py` | Restart; optional `timeout` query parameter |
| `POST` | `/vms/{identifier}/resize` | `vms.py` | Partial `cpu`/`ram`/`storage` update; stop, resize, boot; optional `timeout` |
| `POST` | `/vms/{identifier}/stop` | `vms.py` | Stop; optional `timeout` query parameter |
| `DELETE` | `/vms/{identifier}` | `vms.py` | Remove runtime files/state; optional `timeout` query parameter |
| `POST` | `/vms/{vm_id}/ports` | `vms.py` | Exact ID only; body has `mode` and `ports` |
| `POST` | `/vms/{vm_id}/ports/sync` | `vms.py` | Exact ID only; reapply managed policy |
| `POST` | `/vms/{vm_id}/exec` | `vms.py` | Exact ID; `command`, strict `sudo`, bounded `timeout_seconds` |
| `POST` | `/vms/{vm_id}/agent/install` | `vms.py` | Exact ID; install/upgrade the agent on a stopped disk |
| `GET` | `/vms/{identifier}/logs` | `vms.py` | Bounded non-following log tail |
| `GET` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Saved session list (Nest only) |
| `POST` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Launch a guest PTY; optional rows/cols in a JSON object (Nest only) |
| `GET` | `/vms/{vm_id}/sessions/{session_id}` | `nest/session_routes.py` | VM-scoped session metadata (Nest only) |
| `POST` | `/vms/{vm_id}/sessions/{session_id}/close` | `nest/session_routes.py` | End a shell; retain history (Nest only) |
| `WS` | `/vms/{vm_id}/sessions/{session_id}/terminal` | `nest/session_routes.py` | Authenticated replay/input/resize (Nest only) |

All management routes use `get_authorized_application()` and require:

```http
X-UVM-Token: <UVM_API_TOKEN>
```

`POST /exit` is a management route. The shared `uvm.server.run_server()` creates
the Uvicorn server, stores it on the FastAPI application state, and runs it
in-process. The route returns `202` before a background task sets `should_exit`,
allowing Uvicorn to deliver the response and perform its normal graceful
shutdown for Nest or UVM. It does not stop or delete any managed VM.

Request schemas reject unexpected fields. `UvmError` instances are rendered as
JSON errors by `uvm/server.py`; Pydantic request validation uses FastAPI's normal
422 response format. Create accepts `name: str | null`, `ports: list | null`,
and strict boolean `no_default_ports` (default false). Port objects contain a
strict integer `port` in 1-65535 and uppercase `protocol` TCP/HTTP/HTTPS (TCP
default). The router normalizes/deduplicates ports and adds TCP/22 unless
default ports are disabled, even for null/empty input. It never creates an
unmanaged VM from an omitted policy. Names share CLI validation.

Create accepts `storage` (default `"30G"`). Resize requires at least one non-null
`cpu`, `ram`, or `storage`; identity/credential fields are rejected by its schema.
Responses expose applied `storage_mib` and a nullable, resource-only
`pending_resize` object so interrupted requests are visible without secrets.

VM responses include `name`, `ports`, `policy_status`, and `operation` alongside
resource/status fields and username. The schema excludes password and
disk/socket/config/log path fields; error messages may still contain paths.
`ports: null` in a legacy response still means unmanaged. Start/restart/stop
and port-policy mutations report the resulting policy status. Restart, stop,
resize, and delete accept an optional finite non-negative `timeout` query parameter;
there is no dry-run HTTP schema. Port mutation bodies use
`{"mode":"set"|"open"|"close", "ports":[...]}` and always resolve the
target by exact ID to prevent name/IP reuse. A legacy policy must be set while
stopped before it can be started or synced. Typed not-found/conflict errors map
to 404/409.

Log reads accept a `tail` query parameter from 0 through 1000 and never follow
the file. `LifecycleService.read_log_tail()` reads at most the final 256 KiB and
marks a response truncated when an oversized line or that bound prevents a full
tail. Bulk bodies use an action plus either explicit identifiers or `all_vms`;
the router resolves them before `run_many()` executes the fixed IDs under one
operation lock. The API does not expose interactive SSH, raw Firecracker calls,
arbitrary host command execution, or host path access. Public `/`, `/dashboard`,
`/docs`, `/redoc`, and `/openapi.json` do not perform privileged management
requests.

### Guest Execution

`LifecycleService.execute()` resolves an exact registry ID, validates canonical
paths and an observed running VMM with no pending operation, and connects under
`operations.lock`. `GuestSession` checks the Unix peer PID against the selected
VMM, sends Firecracker's `CONNECT 1070\n`, consumes its `OK` acknowledgement, and
checks the agent's protocol version, VM ID, and configured username. The lock is
released before submission/waiting; the established socket pins the original
VMM and is never reconnected across a concurrent restart or address reuse.

`guest_agent.py` is both a host-importable protocol module and a standalone
stdlib-only Python 3.10+ guest daemon. Keep it free of imports from the rest of
UVM. Version 1 uses length-prefixed JSON with bounds before allocation, absolute
socket deadlines, a 64 KiB UTF-8 command limit, and 1 MiB combined raw output.
The synchronous HTTP response exposes only exit code, stdout/stderr, timeout,
and truncation. Captured text uses UTF-8 replacement. Nonzero process exits are
HTTP 200 results; agent failures are 502, and failures before submission are 503.
Lost/malformed responses after submission carry `outcome_unknown: true`. There
is no command retry, durable job registry, streaming response, or reconnection
for these one-shot exec requests.

The root systemd agent accepts only host-CID connections and limits concurrent
connections/jobs to eight. Each request creates a delegated cgroup v2 job scope.
A fresh child interpreter joins the scope before initializing the chosen user's
supplementary groups, GID, UID, home, and environment, then execs `/bin/sh -c`.
`sudo` selects guest root directly; otherwise the recorded username is used.
Password authentication is not involved, and a recorded `root` login remains
root for both flag values. One-shot commands have closed stdin; Nest terminal
sessions use the separate PTY channel described below.

Both output pipes are drained concurrently. The guest enforces the deadline,
signals the process group, escalates via `cgroup.kill`, and confirms cleanup.
The leader is observed with `waitid(WNOWAIT)` until signaling finishes, preventing
PGID reuse before reap. Normal exits clean up remaining background descendants
too. An unconfirmed cleanup is an unknown outcome. Stopping the agent service
also kills its job scopes; dropping an HTTP connection does not cancel a job.

The v5 Ubuntu builder installs Python, the terminal-capable service, and an
ownership manifest. Creation provisions the VM ID/username in root-only
`/etc/uvm-agent.json` before marking the disk ready. Cached v4 templates are also
configured; their original manifest-less agent is recognized by its exact
published source digest. Earlier templates require an explicit stopped-disk
`agent/install` request.
Provisioning and upgrades stage a sparse copy, replay/check ext4, validate guest
path ownership and recorded agent-file hashes, enable the service, verify writes,
and publish atomically. A root-only guest manifest records managed file hashes
so later upgrades preserve guest edits. An older guest missing Python installs
it through apt inside the guest on its next boot; no guest binaries execute on
the host. This bootstrap requires working guest egress until the runtime is ready.

The guest needs systemd, Python 3.10+, built-in virtio-vsock, and cgroup v2 with
`cgroup.kill` (Linux 5.14+). Host socket paths are derived from Settings, so the
registry schema stays at 3. Each Firecracker instance has an independent vsock
device and uses CID 3. The process adapter retains both sockets while identity
is uncertain and removes them only after confirmed VMM exit. IPv4 port policies
do not govern this host-only management channel.

### Nest Terminal Sessions

Nest owns session HTTP/WebSocket routes and history. UVM owns the process-pinned
connection and guest PTY. `LifecycleService.open_terminal()` validates an exact
running VM ID and connects under the operation lock, then opens the PTY outside
the lock. It never reconnects that socket to a replacement VMM.

The existing version-1 hello and exec request stay compatible. A terminal request
uses `{"version":1,"terminal":{"rows":24,"cols":80}}`. The guest replies with
`{"type":"ready"}` and exchanges length-prefixed JSON output/input/resize/close
messages. Bytes are base64 encoded; raw chunks are at most 8192 bytes, frames
16384 bytes, and dimensions 2–200 rows / 2–400 columns. Partial frames have finite
deadlines while idle shells can remain connected indefinitely. An old agent
rejects the new request; the host reports the stopped-disk upgrade procedure.

The guest opens a real PTY and spawns a fresh interpreter in a new OS session.
The child joins its delegated job cgroup, establishes the controlling terminal,
assumes the recorded account, and execs that account's interactive shell with
`TERM=xterm-256color`. Input, terminal echo, Ctrl+C, foreground job control, and
window-size changes are handled by the guest terminal driver. Normal exit,
explicit close, or host connection loss cleans up the job cgroup and processes.

`SessionManager` owns background output readers independently of WebSocket
attachments. Creating a shell is shielded from an HTTP disconnect; its connection
is either tracked or closed. Browser navigation, reload, and logout detach only
the browser. Nest shutdown closes guest connections and marks live sessions
interrupted. In-flight transcript writes finish before releasing store ownership.

`SessionStore` keeps metadata and raw output in
`$UVM_BASE/nest/sessions.sqlite3`, separate from the VM registry. A private
directory, mode-0600 files, an exclusive owner lock, and SQLite transactions
protect history. On reopening after a server restart, unfinished records become
interrupted. Session lookups always include the exact VM ID. Output is capped at
the newest 8 MiB per session with byte offsets and an explicit truncation marker;
session metadata is retained. Only PTY output is recorded, not a separate input
or password log. Nest permits 32 active sessions; guest terminals share the
agent's eight connection slots with one-shot jobs.

HTTP routes use the existing `X-UVM-Token` dependency. Browser WebSockets check
the Origin and require `{"type":"auth","token":"...","offset":0}` as their
first frame within ten seconds. Tokens never enter query strings. Each output
message has its byte offset and base64 data; the browser acknowledges the next
offset after xterm renders the chunk. Only one chunk is outstanding, providing
backpressure. Reattachment resumes from an acknowledged offset; a truncated
prefix resets the viewer. Closed sessions use the same channel for read-only
replay. WebSocket transport support comes from the `websockets` dependency.

The dashboard's Sessions tab and nested hash routes belong to the selected VM.
`sessions.js` owns attachment teardown, replay, reconnects, and the terminal
widget. Vendored xterm 5.5.0 and fit addon 0.10.0 (with MIT licenses) are served
locally under `static/vendor/`. Terminal keyboard input bypasses dashboard search
shortcuts. The widget has a 10,000-line scrollback and a 64 KiB per-paste limit.

## 14. Configuration Reference

The environment-variable table is in [README.md](README.md#configuration);
[`config.py`](uvm/config.py) is the implementation authority. Keep the same
base/network/gateway/bridge settings across CLI and server operations. The
network identity is retained per VM and checked before resume.

`Settings` validates the gateway, bridge-name length, API token format, and
environment boolean values at process startup. Add configuration fields there,
not as module-level global constants in feature code.

## 15. Error Handling Rules

Expected failures inherit from `UvmError`:

| Error | Use |
|---|---|
| `ConfigurationError` | Invalid local settings or environment |
| `ValidationError` | Invalid CLI/API resource input |
| `StateError` | Unsafe or malformed persisted state |
| `NotFoundError` | Missing registry entry |
| `ConflictError` | Ambiguous identifier, allocation conflict, or unsafe/pending operation |
| `CommandError` | Required host command failure |
| `FirecrackerError` | VMM process or Unix API failure |
| `GuestAgentUnavailableError` | Agent connection/readiness failure before submission (HTTP 503) |
| `GuestExecutionError` | Execution failure or unknown outcome after submission (HTTP 502) |
| `TapCreationError` | TAP setup failed after interface creation |

CLI code catches `UvmError` and prints `uvm: error: ...` without a traceback.
Invalid CLI input returns 2, ordinary operational failure 1, and interruption
130. Batch JSON retains per-target outcomes rather than pretending a partial
success was atomic.

The FastAPI server maps the same class to an error envelope. New expected
operational failures should use this hierarchy rather than leaking raw
`OSError`, `CalledProcessError`, or implementation-specific exceptions.

## 16. Concurrency and Ownership Invariants

These invariants are critical:

1. Only `LifecycleService` may coordinate VM state, disks, TAPs, firewall,
   and Firecracker as one operation.
2. Only `StateStore` may read/write `state.json`.
3. All long host mutations acquire `operations.lock`.
4. All short inventory mutations acquire `state.lock`.
5. The installer shares `operations.lock` with lifecycle actions, preventing a
   Firecracker binary replacement between verification and launch.
6. A VM rootfs is private before it becomes writable.
7. Process signaling requires a safe pidfd and matching persisted start/boot
   identity; unknown is not gone. Legacy adoption must prove binary/socket identity.
8. A new VM never adopts a pre-existing TAP or activates it before verified policy.
9. A server must have an API token; a remote server must also have usable TLS.
10. Routers call application services; they do not run shell commands or touch
    state files directly.
11. Plaintext guest passwords remain in the mode-`0600` registry and never
    enter Firecracker config, helper command arguments/traces, or CLI/API VM output.
12. Retained-disk start never overwrites guest data, credentials, or SSH host keys.
13. No disk/IP release before confirmed exit and scoped network cleanup; no
    old-policy rollback after a failed close, and no resurrection of pending deletion.
14. Policy state is observed from the actual owned rules, not a cached applied flag.

When a change would violate one of these rules, redesign the change rather
than adding a special-case bypass.

## 17. Testing Strategy

### Standard Unit Suite

```sh
python3 -m pip install -e '.[test]'
python3 -B -m unittest discover -s tests -v
```

Ordinary tests require no root, KVM, Firecracker, or host network mutations.
They cover validation, schema migration/locking/permissions, artifact trust,
private disks/credentials, network/firewall rendering and readback, conntrack
semantics, pidfd/legacy identity, lifecycle recovery/resume/bulk targeting,
CLI parsing/output, Nest dashboard delivery, and server security. Mock/fixture
policy checks do not prove the kernel's actual packet behavior.

FastAPI route tests are conditionally enabled when both `fastapi` and `httpx`
are installed. They should be enabled in CI using the `test` extra.
`test_nest_server.py` checks the workspace launcher, relocated frontend assets,
authenticated machine controls, API documentation, and graceful shutdown.
`test_sessions.py` covers scoped persistence, interrupted-session recovery,
authentication, reconnects, output acknowledgements, and cancellation cleanup.
`test_guest_terminal.py` exercises actual PTYs, persistent shell state, resizing,
Ctrl+C, and child cleanup using temporary account homes and fake cgroup scopes.

Optional browser regression checks use Playwright against an isolated fixture
server with fake VMs and guest channels. With the project's test extra installed:

```sh
npm install --prefix /tmp/nest-browser playwright
/tmp/nest-browser/node_modules/.bin/playwright install chromium
NODE_PATH=/tmp/nest-browser/node_modules \
  NEST_TEST_PYTHON="$(command -v python)" node tests/browser_sessions.cjs
```

This checks launch/input, reload/reconnect, session switching, per-VM isolation,
closed-session replay, and the mobile layout. It does not boot a real microVM.

When `nft` and `conntrack` are installed, command-parser tests append an invalid
final command/option so parsing is checked without executing any mutation.
These checks supplement fixture-based readback and cleanup tests; they do not
exercise the kernel packet path.

`test_storage.py` also exercises real e2fsprogs against temporary unmounted,
sparse ext4 images when the tools are installed: default 30 GiB creation,
growth, shrinking, geometry, and retained data/password-hash/host-key fixtures.
`test_images.py` uses temporary ext4 images to verify custom-user creation,
UID/GID allocation, account/home metadata, existing accounts, and password hashes.
`test_install.py` simulates package bootstrap and exercises real ext4 formatting
and readback for retained dpkg metadata, sudo modes, template configuration,
staging failures, durable build diagnostics, and checksum provenance. The apt
shell script runs against stub mount/chroot commands to check argument handling
and stop-on-error behavior. Tests simulate Ubuntu removing `policy-rc.d` after
bootstrap. They do not run debootstrap or boot a VM.
Lifecycle resize tests use fake processes/networking to verify ordering,
single-lock ownership, identity preservation, pending intent, and failure recovery.
`test_guest_network.py` exercises real offline ext4 profiles, explicit guest
addresses/gateways, custom subnets, idempotent migration, journal-replay
rechecks, and retention of guest data, credentials and keys on failures.
Lifecycle tests require confirmed process exit before network-profile repair.

Guest execution tests use real child processes with fake cgroup scopes, local
Unix sockets emulating Firecracker's vsock handshake, API validation fixtures,
and real temporary ext4 upgrades. They cover credential selection, concurrent
output draining, timeout cleanup, connection loss, VM targeting/locking, and
upgrade rollback. `tests/integration/test_guest_exec.py` is an opt-in smoke test
against a prepared disposable VM with a non-root login and all TCP ports closed;
see [USAGE.md](USAGE.md#18-verification). Unit tests do not validate the real
guest vsock device or cgroup kernel path.

### Isolated Packet Tests

The opt-in packet harness must run **only in disposable isolated user/network
namespaces**. It verifies namespace separation before creating links or
changing rules, and never falls back to the parent host namespace. It may
require privileges (or enabled unprivileged user namespaces) plus `unshare`,
`ip`, `bridge`, `nft`, `iptables`, `conntrack`, and `sysctl`:

```sh
UVM_FIREWALL_INTEGRATION=1 python3 -B -m unittest discover -s tests/integration -v
```

The harness uses bridge-connected veths in place of guest TAPs and two simulated
uplinks. It exercises NAT with a pre-existing terminal FORWARD drop, offline
startup, uplink changes and reconnects for host/guest TCP and guest UDP/53.
It also checks host/peer service traffic, both-direction close of existing
streams while preserving conntrack/egress, IPv6/spoof blocking, quarantine,
and offline cleanup. It does not boot a microVM or validate a real TAP deployment.

The packet test skipped in this workspace because namespace creation was
denied (`Operation not permitted`). Document that permission limitation, not
an assertion that real packets or a real host were validated. Unit tests and
nft JSON parser checks are not substitutes for kernel packet-path execution.

### Isolated Host Validation

No current test automatically boots a real microVM. Before a release, validate
on an isolated Linux host:

1. Strict checked installation.
2. `create` and successful guest boot.
3. Guest IP reachability and egress, including the guest's default route after
   systemd-networkd startup and reconfiguration.
4. SSH with unique guest host keys, `sudo -i` for a custom login, and
   `sudo apt update && sudo apt install htop` in a newly built Ubuntu guest.
5. Stop/start/restart with retained guest data, SSH keys, and credentials.
6. Live private-port close in both directions, empty policy, and explicit sync
   after deliberately lost rules inside the isolated setup.
7. Delete and IP/TAP/policy/conntrack cleanup.
8. API launch with token and TLS on a non-loopback test address.
9. Firecracker crash, interrupted create/start/stop/delete, and legacy recovery.

## 18. Common Development Tasks

- CLI: keep business logic in services; wire arguments in `build_parser()` and
  dispatch/output in `_run_command()`. Test validation, prompts, dry runs,
  JSON, and service behavior; use `UvmError` for expected failures.
- HTTP: define strict models in `api_models.py`, use the appropriate router and
  `get_authorized_application()`, and register new routers in `create_api()`.
  Add API tests and update all route tables. New CLI commands do not implicitly
  authorize new HTTP endpoints; `/health` remains the only unauthenticated
  operational API route. Nest owns the public `/` and `/dashboard` routes.
- Dashboard: keep the document route and frontend assets in `nest/dashboard/`.
  Compose workspace-level routes in `nest/server.py`; machine-management
  operations continue through the UVM API and its application services.
- Host primitives: use the specific network/firewall/install/process module
  and `CommandRunner` for external commands. Define ownership, idempotence,
  and compensation before invoking a primitive from lifecycle code. Use fake
  runners rather than requiring privileges in unit tests.
- State: update `VmRecord`/`State`, schema tests, cleanup, and documentation.
  Version incompatible interpretations and preserve older data or provide an
  explicit migration. Consider lock order and crash/retry behavior first.

## 19. Debugging Guide

| Symptom | First places to inspect |
|---|---|
| `guest assets missing` | `/var/lib/uvm/images`, integrity variables, `integrity.json` |
| Firecracker exits early | VM `firecracker.log`, VM config, `/dev/kvm` access |
| VM is `dead` in `list` | PID/start token in state, `/proc/<pid>`, VM log |
| Cannot stop legacy VM | Missing start/boot identity or failed protected binary/socket match; never guess a PID |
| Guest has no egress | Host connectivity/routes, bridge/TAP state, `uvm-egress` iptables rules; `start` refreshes networking on a running VM |
| Guest says `Network is unreachable` | Guest `ip -4 route`; the default route must use the UVM bridge gateway. Restart once with updated code to migrate an original Ubuntu profile. |
| Policy missing/drift/unknown | Actual nft JSON and ownership, STP/group mask, permissions; explicit sync after fixing cause |
| Retained-disk start fails | `disk_ready`, safe private path, kernel/VMM integrity, process identity, retained network, pending operation |
| API server refuses to start | `UVM_API_TOKEN`, requested host, TLS cert/key files |
| API returns 401 | `X-UVM-Token` header mismatch |
| API returns 422 | Request schema or UVM input validation failure |

Use `inspect VM --json`, `logs VM`, `ports list --all`, and `doctor` before
examining host state. [USAGE.md](USAGE.md) lists read-only bridge/nft/iptables
debugging commands. Never share the credential registry or unredacted guest
logs. Do not manually mutate UVM resources during a lifecycle operation;
fix the service/host failure and retry through supported commands.

## 20. Current Production Gaps

The code is intentionally honest about current limits:

- Firecracker is launched directly, even though the Jailer binary is
  downloaded.
- No cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU requests round vCPUs up without host quota enforcement.
- Guest health is inferred from VMM process state, not application readiness.
- No quotas, DHCP, host-port mapping, UDP/IPv6 services, or network tenancy.
- No guest snapshots, extra disks, production image builder, or SSH-key injection.
- Per-VM private TCP ingress policy exists, but public inbound connectivity,
  DNS publication, guest reverse proxy/TLS, and external firewall integration
  remain out of scope.
- Shared egress `iptables` rules are incremental and idempotent; only the
  private bridge policy is atomically rendered per VM.
- No autostart, background reconciliation, or guest-coordinated shutdown.
- The JSON inventory is suitable for one host, not distributed coordination.
- Guest passwords are stored in plaintext in the root-only local registry.
- API authentication is a single bearer token, not a user/tenant model.

Do not market UVM as a hardened multi-tenant platform until these boundaries
are deliberately addressed with architecture, tests, and operational review.

## 21. Contributor Checklist

Before submitting a change:

1. Run the complete unit suite.
2. Run `git diff --check`.
3. Keep workspace/dashboard features in `nest/` and machine-management features in `uvm/`.
4. Verify command/API errors use `UvmError` rather than raw tracebacks.
5. Verify host mutations have a lock owner and cleanup path.
6. Verify persistent state remains readable across upgrades.
7. Update `README.md` for operator-facing behavior.
8. Update this document for architectural, trust-boundary, or lifecycle
   changes.