nest/README.md

# Nest

Nest is a local infrastructure workspace with a browser dashboard for your
machines, network, and host health. Launch it with `nest --serve`.

Its `uvm` component is a small, local Firecracker microVM manager for Ubuntu.
UVM creates one Firecracker process per VM, attaches it to a host TAP device,
and keeps a private disk and local registry for named VMs. The CLI supports
create, start, stop, restart, resize, delete, inspection, SSH, logs, diagnostics,
host-shared IPv4 internet access, and enforced private-IPv4 TCP port allowlists.

This directory is a self-contained project. Nest owns the dashboard and
workspace server; UVM provides the machine-management CLI and backend services.
All code, tests, packaging metadata, launchers, and documentation live here.

For architecture, lifecycle invariants, extension guidance, and contributor
workflows, see [DEVELOPER.md](DEVELOPER.md).

For end-to-end operator scenarios, CLI examples, and HTTP API calls, see
[USAGE.md](USAGE.md).

For the original firewall proposal and backend evaluation, see
[FIREWALL.md](FIREWALL.md). That proposal is design background, not the current
feature contract; these guides describe the implementation in the workspace.

## Quick Start

Run commands from this project root:

```sh
./nest.py --help
python3 -B -m nest --help
./uvm.py --help
python3 -B -m uvm --help
```

The `pyproject.toml` file defines installable `nest` and `uvm` console commands.
The project is an application, not a reusable Python SDK.

## Requirements

- Ubuntu or another compatible Linux host
- Python 3.12 or newer
- Root access for UVM commands that read or mutate the protected VM registry
- KVM enabled and accessible at `/dev/kvm`
- Linux 5.3 or newer with native bridge conntrack and safe pidfd support
- An x86_64 or ARM64 (`aarch64`) host, including NVIDIA DGX Spark
- Working host networking, `iproute2`, `iptables`, `nftables`, `conntrack`,
  `coreutils`, `e2fsprogs`, `util-linux`, `openssl`, and `openssh-client`
- An STP-disabled software bridge; no offload path that bypasses bridge hooks

`uvm install` installs the required host packages. It validates `/dev/kvm`
before downloading VM artifacts. Boot operations additionally check that the
native nftables bridge policy is supported; installing packages alone is not
a connectivity or isolation test.

Firecracker and the guest kernel are selected automatically for the host
architecture. The default rootfs is built locally from Ubuntu 22.04 (Jammy)
packages with OpenSSH, sudo, and a complete apt/dpkg database. On NVIDIA DGX
Spark, UVM uses ARM64 assets and packages.
`arm64` and `amd64` architecture names are normalized to `aarch64` and
`x86_64`, respectively. Custom `UVM_KERNEL_URL` and `UVM_ROOTFS_URL` overrides
must point to assets compatible with the host architecture.

The default guest kernel is Linux 6.1.155 from Firecracker's published `v1.15`
CI artifacts, with built-in `virtio-rng` support. UVM attaches a host-backed
entropy device before each guest boot so services such as SSH can initialize
their cryptographic random generator. Custom kernels must include
`CONFIG_HW_RANDOM=y` and `CONFIG_HW_RANDOM_VIRTIO=y` for this device.

Older installs used a Linux 4.14 ARM demo kernel and supplied no entropy
device. If SSH stays at **Starting OpenBSD Secure Shell server** while ping
works, see [SSH startup and guest entropy](USAGE.md#ssh-startup-and-guest-entropy)
for the kernel-refresh and retained-VM recovery steps.

## Trusted Installation

Installation requires trusted SHA-256 digests by default. Supply digests from
an independently trusted source for the selected architecture before running
the privileged installer:

```sh
sudo env \
  UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
  UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
  ./uvm.py install
```

The Ubuntu build uses `debootstrap`, authenticated Ubuntu archive signatures,
and native packages. It bootstraps the base release first, then uses apt to
apply updates/security fixes and install the guest packages. UVM installs the
builder prerequisites and records the built ext4 image's SHA-256 in the
integrity manifest. Building requires root
with mount/PID namespace support and internet access to Ubuntu mirrors.

Use `--force` to refresh downloads and rebuild the Ubuntu template:

```sh
sudo env \
  UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
  UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
  ./uvm.py install --force
```

To use a prebuilt ext4 image instead, set both `UVM_ROOTFS_URL` and its trusted
`UVM_ROOTFS_SHA256`. Leave both unset for the default Ubuntu build.

### Upgrade from the Stripped Bionic Demo Image

Run `install --force` with the updated code and with `UVM_ROOTFS_URL` and
`UVM_ROOTFS_SHA256` unset, then create a new VM. The replacement template
includes sudo and the package database that the old demo image lacked.
Existing VM disks retain their original OS and credentials; installation
updates the template used for future VMs. Restart a running API server to load
the updated provisioner before creating VMs through the dashboard.

If the build fails, its detailed bootstrap/package output is retained in
`/var/lib/uvm/images/ubuntu-build.log` (mode `0600`). Correct the build failure
and rerun installation before creating a custom-user VM; a failed build leaves
the previous template installed.

For local experimentation only, the old unverified quick-start behavior is
available through an explicit opt-out:

```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py install
```

An unverified manifest is not accepted as publisher trust in a later strict
run. Reinstall with trusted digests to establish a verified local manifest.
To boot VMs from an unverified local-development install, retain
`UVM_ALLOW_UNVERIFIED_DOWNLOADS=1` on later `create`, `start`, `restart`, and `resize`
commands and on the API server process.

## Commands

These are separate operation examples, not a script to run from top to bottom.

```sh
# Install after supplying the trusted checksum variables shown above.
# A bare `install` command fails under the default strict integrity policy.

# Create a named VM: 1 CPU, 512 MiB RAM, 30 GiB storage, root/root, and private TCP/22.
# Use a unique non-default password for anything beyond a throwaway VM.
sudo ./uvm.py create --name web

# Choose resources, a private guest IP, and additional service ports.
sudo ./uvm.py create --name app --cpu 1 --ram 1G --storage 50G \
  --guest-ip 10.42.0.10 --ports 80:HTTP,443:HTTPS

# Read a password from one line of stdin, rather than a --password argument.
# USAGE.md includes a non-echoing Bash prompt example.
sudo ./uvm.py create --name worker --password-stdin

# Explicitly allow no inbound service ports, including SSH.
sudo ./uvm.py create --name closed --no-default-ports

# Inspect persisted state and observed process/policy status.
sudo ./uvm.py list
sudo ./uvm.py list --status running --json
sudo ./uvm.py list -q
sudo ./uvm.py inspect web --json
sudo ./uvm.py logs web --tail 50 --follow
sudo ./uvm.py doctor --json

# Resolve VMs by exact ID, name, or guest IP.
sudo ./uvm.py ssh web
sudo ./uvm.py ssh app --port 22 -- uname -a
ssh root@10.42.0.10

# Stop and boot the same private disk, or restart selected VMs.
sudo ./uvm.py stop web --timeout 5
sudo ./uvm.py start web
sudo ./uvm.py restart web app --timeout 10

# Stop, change resources, and boot the same VM with its existing identity/data.
sudo ./uvm.py resize app --cpu 2 --ram 4G --storage 80G
sudo ./uvm.py resize app --ram 2G

# Preview bulk operations before confirming them.
sudo ./uvm.py stop --all --dry-run --json
sudo ./uvm.py stop --all -y
sudo ./uvm.py start --all -y
sudo ./uvm.py restart --all -y
sudo ./uvm.py delete --all --dry-run
sudo ./uvm.py delete web -y
# destroy is an alias for delete, with the same flags and destructive effect.
```

`start`, `stop`, `restart`, and `delete` accept one or more VM identifiers or
`--all`, plus `--json`, `--dry-run`, and `--yes`/`-y`. Deletion and mutating
`--all` selections require confirmation; non-interactive callers must use
`--yes`. `--timeout` is available on stop/restart/delete (and `destroy`), not
start. It is a finite non-negative SIGTERM wait in seconds, default `5`, not
a total command deadline or a guest shutdown timeout.

### Storage and Resizing

New VMs default to a **30 GiB private disk**, which is also the minimum.
`--storage` accepts values such as `30G`, `50G`, or `1T`; bare numbers use GiB.
As with RAM, `G`/`GB`/`GiB` are binary units. Capacities round up to whole MiB
and are returned as `storage_mib`. Disks are sparse: the configured capacity
does not immediately reserve that much physical host space.

`resize VM` accepts any combination of `--cpu`, `--ram`, and `--storage`, plus
`--timeout` and `--json`. It stops the VM, resizes the ext4 disk when needed,
updates the resource settings, and boots it. A previously stopped VM also boots
at the end. VM ID, IP/MAC, username/password, port rules, guest files, and SSH
host keys are retained. The dashboard exposes the same operation under
**Resize resources**; the API uses `POST /vms/{identifier}/resize`.

Storage can grow or shrink, provided ext4 can fit its existing data into the
requested capacity. Disk changes are prepared on a temporary sparse copy and
published after verification. Failed resizes retain a `pending_resize` target
for retry; `start` on the stopped VM completes that target before booting.
Older VMs report `storage_mib: null` until their first resize, which retains
their current capacity or grows it to the 30 GiB minimum when storage is omitted.

Stop sends SIGTERM and, if needed, SIGKILL to the Firecracker VMM. It is **not
a guest-coordinated shutdown**. Start boots the retained disk without resetting
guest data, SSH host keys, or credentials; it does not restore guest memory.
Delete permanently removes the disk and releases the VM's name and IP after
cleanup. See [USAGE.md](USAGE.md) for the complete flag and recovery reference.

Names are unique local identifiers, not DNS names. `--host-ip` remains an
alias for `--guest-ip`; both select the guest's private IPv4 address. JSON
output omits passwords, with warnings and host-command traces on stderr.

SSH uses `StrictHostKeyChecking=accept-new` and a VM-ID-specific host-key
alias by default. `--user`, `--key`, and `--port` customize the SSH client;
append a remote command after a literal `--`. `--insecure-host-key` disables
verification for one connection. SSH does not automatically supply the stored
password or open a closed port.

Deletion removes matching local SSH `known_hosts` entries for the guest IP and
UVM's VM-ID host-key alias before releasing the IP for reuse. It checks the
effective user's files and, under `sudo`, the invoking user's files, including
hashed entries. Unrelated hosts and file ownership/permissions are preserved.
For API services or custom SSH cache locations, set `UVM_KNOWN_HOSTS_FILES` to a
colon-separated list of absolute paths on the UVM host. See the SSH section in
[USAGE.md](USAGE.md#7-connect-with-ssh) for configuration and one-time cleanup of
entries left by older versions.

### Private Ports

```sh
sudo ./uvm.py ports list web --json
sudo ./uvm.py ports list --all
sudo ./uvm.py ports open web --ports 80:HTTP --ports 443:HTTPS,8080:TCP
sudo ./uvm.py ports close web --ports 80
sudo ./uvm.py ports set web --ports 22,443:HTTPS --dry-run --json
sudo ./uvm.py ports set web --ports 22,443:HTTPS -y
sudo ./uvm.py ports close web --all -y
sudo ./uvm.py ports set web --none -y
sudo ./uvm.py ports sync web
sudo ./uvm.py ports sync --all --dry-run
sudo ./uvm.py ports sync --all -y
```

Rules use `PORT[:TYPE]`, for example `22`, `80:HTTP`, `443:HTTPS`, or
`8080:TCP`, repeated or comma-separated. HTTP and HTTPS are **TCP labels**,
not proxies or TLS configuration. Create adds `22:TCP` even when other ports
are supplied, unless `--no-default-ports` is set. Later port edits do not
automatically add SSH back. `ports close VM --all` closes ports on that one
VM; only `ports list --all` and `ports sync --all` select all VMs.

Port mutations support `--dry-run` and `--json`; close/set/sync also support
`--yes`/`-y`. Set, close-all, and removal of TCP/22 require confirmation
(an already empty managed policy can be a no-op). Closing a service blocks
subsequent packets on existing service connections in **both directions**, as
well as new connections. This is packet dropping, not a guaranteed TCP RST or
immediate application disconnect.

These rules govern access to the **guest private IP**. They do not start a
guest service or provide public inbound connectivity, host-port mappings,
DNS publication, reverse proxies, or guest-service TLS. UDP and IPv6 service policies are not
yet supported. Guest-initiated IPv4 TCP/UDP egress, its replies, and required
ARP/ICMP traffic remain allowed; managed TAPs block IPv6.

## Nest Dashboard and Management API

Install the project dependencies before serving the API:

```sh
python3 -m pip install .
```

Launch the Nest workspace server using the environment where the project
dependencies were installed (use the full console-script path if needed):

```sh
export UVM_API_TOKEN='replace-with-a-long-random-secret'
sudo env UVM_API_TOKEN="$UVM_API_TOKEN" \
  nest --serve --host 127.0.0.1 --port 8000

# Equivalent when running the source checkout directly.
sudo env UVM_API_TOKEN="$UVM_API_TOKEN" \
  ./nest.py --serve --host 127.0.0.1 --port 8000
```

The default bind address is `127.0.0.1` and the default port is `8000`.
OpenAPI documentation is available at `/docs` once the server is running. The
root URL redirects to the Nest dashboard at `/dashboard`; enter the API token
there for the current browser session.

Nest serves the dashboard and UVM-backed management endpoints together, using
the existing `UVM_API_TOKEN` and `UVM_*` host configuration. The standalone UVM
API is available through `uvm --serve`, with its reference at `/docs`; the `/`
and `/dashboard` routes belong to the Nest server.

The Nest dashboard includes a fleet overview, searchable machine collection,
private network and port-policy view, host diagnostics, and machine provisioning.
Open a machine to manage its lifecycle, install developer tools, launch terminal sessions, edit or sync
its port rules, copy its SSH command, or read recent logs. Select machines in the
collection for bulk operations.
Use `Ctrl+K` / `⌘K` (or `/`) to find a machine or page. Fleet data refreshes every
five seconds while the tab is visible; host checks refresh every thirty seconds
and can also be run manually. Charts show observed machine states and allocated
resources, rather than guest CPU or memory utilization. All frontend assets,
including the font, are served locally; no frontend build step is required.

The server exposes these routes:

| Method | Route | Purpose |
|---|---|---|
| `GET` | `/` | Redirect to the dashboard (Nest server) |
| `GET` | `/dashboard` | Serve the Nest workspace dashboard (Nest server) |
| `GET` | `/health` | Unauthenticated liveness response |
| `POST` | `/exit` | Gracefully stop the management server |
| `GET` | `/diagnostics` | Run host readiness checks for the dashboard/system view |
| `POST` | `/install` | Install or refresh host assets; body: `{ "force": false }` |
| `GET` | `/vms` | List VM records and observed status |
| `POST` | `/vms` | Create; supports `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports`, `tools` |
| `POST` | `/vms/actions` | Run a fixed snapshot of selected or all VM lifecycle actions |
| `GET` | `/vms/{identifier}` | Inspect by exact ID, name, or guest IP |
| `POST` | `/vms/{identifier}/start` | Boot a retained private disk |
| `POST` | `/vms/{identifier}/restart` | Stop then boot a VM; optional `timeout` query parameter |
| `POST` | `/vms/{identifier}/resize` | Stop, resize CPU/RAM/storage, and boot; optional `timeout` query parameter |
| `POST` | `/vms/{identifier}/stop` | Stop a VM while retaining its private disk; optional `timeout` query parameter |
| `DELETE` | `/vms/{identifier}` | Delete a VM and release its resources; optional `timeout` query parameter |
| `POST` | `/vms/{vm_id}/ports` | Set, open, or close private port rules for an exact VM ID |
| `POST` | `/vms/{vm_id}/ports/sync` | Reapply a managed private port policy for an exact VM ID |
| `POST` | `/vms/{vm_id}/exec` | Execute a command/script inside a running VM over vsock, with optional root privileges |
| `POST` | `/vms/{vm_id}/agent/install` | Install/upgrade the execution agent on a stopped VM's private disk |
| `GET` | `/tools` | Guest tool catalog and supported Python versions |
| `GET` | `/vms/{vm_id}/tools` | Tool inventory and installation history; `refresh=true` probes the running guest |
| `POST` | `/vms/{vm_id}/tools/install` | Queue selected guest tools; HTTP 202 with a job ID |
| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | Per-tool progress, results, and bounded installation output |
| `GET` | `/vms/{identifier}/logs` | Read a bounded, non-following tail of a VM log |
| `GET` | `/vms/{vm_id}/sessions` | List this VM's saved sessions (Nest) |
| `POST` | `/vms/{vm_id}/sessions` | Launch a guest terminal; body: `{}` or `{"rows":24,"cols":80}` (Nest) |
| `GET` | `/vms/{vm_id}/sessions/{session_id}` | Session metadata and status (Nest) |
| `PATCH` | `/vms/{vm_id}/sessions/{session_id}` | Rename a session; body: `{"name":"Deployment"}` (Nest) |
| `DELETE` | `/vms/{vm_id}/sessions/{session_id}` | Close the session and delete its saved history; returns 204 (Nest) |
| `POST` | `/vms/{vm_id}/sessions/{session_id}/close` | End the shell and retain its history (Nest) |
| `WS` | `/vms/{vm_id}/sessions/{session_id}/terminal` | Authenticated terminal input, output, resize, and replay (Nest) |

API port rules are objects such as `{"port": 80, "protocol": "HTTP"}`;
`no_default_ports` is a boolean, default `false`. Create also accepts `storage`,
defaulting to `"30G"`; resize accepts optional `cpu`, `ram`, and `storage` fields.
Port changes use
`{"mode":"set"|"open"|"close", "ports":[...]}`. A legacy VM must be
stopped and use `set` to establish a managed policy. Responses include `name`,
`ports`, `policy_status`, and `operation`, but never passwords. See
[USAGE.md](USAGE.md) for request examples and strict schema details.
Log reads use an optional `tail` query parameter from `0` to `1000`, default
`100`, and are capped at the final 256 KiB of the log.

Bulk requests use an action (`start`, `stop`, `restart`, `delete`, or
`ports-sync`) plus either `identifiers` or `all_vms: true`; they return a result
for each fixed target and do not prompt for confirmation. Interactive SSH
remains CLI-only. VM lifecycle changes are explicit; accepted tool installation
jobs are processed in the background.

### One-click Guest Tools

Choose tools under **New machine → Pick your tools** to install them automatically
after boot. For an existing VM, open **Machines → select a VM → Tools** and click
**Install**, or select several tools and click **Install selected**. A stopped VM
offers **Start & install**. Installation works over vsock with SSH disabled.

The initial catalog targets Ubuntu 22.04 guests on x86_64 and ARM64:

| Tool | Default |
|---|---|
| Git, wget, curl, htop | Ubuntu packages |
| Node.js | 24 LTS (24.21.0), including npm and npx |
| Python | 3.11, 3.12, and 3.13 with pip and venv; versions can coexist |
| JDK | OpenJDK 21 LTS, including `java` and `javac` |
| PHP | 8.5 CLI from the signed Ondřej Surý PHP PPA |
| Conda | Miniforge 26.5.3-0, using conda-forge |
| micromamba | 2.9.0 |

Node, Python, Miniforge, and micromamba downloads use pinned release checksums.
Python uses Astral's standalone distributions, exposed as `python3.11`,
`python3.12`, and `python3.13`. The guest's OS Python continues serving the agent.
Downloaded tools live under `/opt/uvm-tools`, with executable links in
`/usr/local/bin`. Conda environments and caches are writable by the configured
guest user. **Get started** shows commands, including shell initialization for
Conda/micromamba activation in Nest's non-login terminals.

Jobs run independently of the browser and show progress and captured output per
tool. Already available tools are verified and skipped. A failed tool can be
retried on the created VM. Guest internet access is required for downloads.
Restart the management server to load the feature; older running VMs may need
**Restart & install** to apply the current guest-agent setup.

The most recent ten jobs and cached inventory are stored in the VM's private
`tools.json` file. Queued requests resume when the server starts. An interrupted
in-flight installation has an uncertain outcome and may finish inside the guest;
use **Check installed tools** before retrying. See
[API examples and recovery](USAGE.md#install-guest-tools) for details.

### MicroVM Terminal Sessions

Open **Machines → select a microVM → Sessions**, then click **Launch session**.
Each session opens an interactive guest shell in the dashboard, with terminal
colors, keyboard input, Ctrl+C, resizing, and persistent shell state such as the
working directory. Sessions run as the VM's configured guest user; normal guest
`sudo` commands can request that user's password in the terminal.

The history list belongs to the selected VM. Reopen an active session to
continue using the same shell, or select an ended session to view its saved
output. Navigation, browser reloads, and signing out leave the shell running.
**End session** closes the shell and its processes. Stopping/restarting the VM
or the Nest server ends live sessions; their saved history remains available.

Use **Rename** in a history row or the terminal toolbar to give any session a
custom name. Names are trimmed and accept 1–80 printable characters; they persist
across reloads and server restarts. The session keeps its identity and shell.
**Delete** removes a session and its saved terminal output after confirmation,
closing its shell first if it is active. Both actions are scoped to the session's
microVM.

Nest stores session metadata and the latest **8 MiB of output per session** in
`$UVM_BASE/nest/sessions.sqlite3` (default `/var/lib/uvm/nest/sessions.sqlite3`).
The viewer indicates truncated output. Raw keyboard input is not logged
separately; the transcript contains what the guest terminal outputs. History
files are private to the host account running Nest. One Nest server owns
sessions for a given VM registry at a time, with up to 32 active sessions; the
guest agent shares its eight connection slots between terminals and exec calls.

**Vsock and the guest agent are configured by default for every microVM.**
Creation installs and enables the agent on every compatible image, including
cached older templates and custom images. Each stopped-disk boot also installs
or updates it automatically; an already-current installation is a read-only
check. Existing running VMs receive the setup on their next restart.
The **Sessions → Guest terminal support → Repair / update terminal support**
control is available for manual repair on a stopped VM. The guest-agent/kernel
requirements described below apply to terminals.

Terminal support adds the Python `websockets` dependency. In your micromamba
environment, install the current project and launch with its interpreter:

```sh
micromamba activate nest
python -m pip install -e '.[test]'
sudo env UVM_API_TOKEN="$UVM_API_TOKEN" \
  "$CONDA_PREFIX/bin/python" -m nest --serve
```

### Execute Commands Without SSH

Use an exact VM ID from create/list:

```sh
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"command":"whoami","sudo":true,"timeout_seconds":60}' \
  "http://127.0.0.1:8000/vms/<vm-id>/exec"
```

The response contains `exit_code`, `stdout`, `stderr`, `timed_out`, and
`truncated`. Commands and multiline scripts run through `/bin/sh` inside the
guest. `sudo: false` (the default) uses the VM's configured username; `true`
uses guest root without a password prompt. A VM configured with username `root`
still runs as root when `sudo` is false.

Execution uses a guest systemd agent over Firecracker vsock. It works with SSH
disabled and with an empty TCP allowlist. Every Firecracker boot attaches the
vsock device, and creation/start/restart/resize ensure the guest service is
installed and enabled before boot. Restart the management server to load this
behavior, then restart older VMs once to apply it automatically.

Installation stages and verifies a private disk copy when changes are needed,
preserving guest data and credentials. If Python is missing, the installed
bootstrap service acquires it inside the guest on boot; this one-time step needs
guest internet access. The default Ubuntu template already includes Python.
The guest needs Python 3.10+, systemd with cgroup v2, and a vsock-capable Linux
5.14+ kernel; UVM's default Linux 6.1 kernel provides the required drivers.

See [execution examples and limits](USAGE.md#execute-commands-without-ssh) for
programmatic calls, local script submission, timeout behavior, and upgrade steps.

### API Security

The server controls privileged host operations. `UVM_API_TOKEN` is required
for every API server, including loopback-only deployments. All management
routes require it in the `X-UVM-Token` header. The dashboard shell, liveness
route, and FastAPI's informational `/docs`, `/redoc`, and `/openapi.json`
endpoints do not require a token; the dashboard stores the entered token only
for its current browser session and applies it to management requests.

A non-loopback bind additionally requires TLS certificate and key paths:

```sh
export UVM_API_TOKEN='replace-with-a-long-random-secret'
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'
sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
  UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
  ./nest.py --serve --host 0.0.0.0 --port 8000
```

Send the token on management requests:

```sh
curl \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  http://127.0.0.1:8000/vms
```

Stop the API server without stopping its managed VMs:

```sh
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  http://127.0.0.1:8000/exit
```

The endpoint returns `202 {"status":"stopping"}` and then gracefully terminates
the serving Nest or UVM process. It is a protected management endpoint; it does
not stop or delete managed VMs.

Do not expose this server directly to an untrusted network. For remote access,
keep TLS enabled and consider placing it behind an additional authenticated
reverse proxy.

## Guest Image Contract

The installed kernel and rootfs are templates. `create` copies the rootfs to
the VM runtime directory before Firecracker opens it read-write, so VMs do not
share a mutable root disk.

During creation, UVM updates the private ext4 disk with a password for the
requested guest account and enables OpenSSH password authentication. Missing
custom usernames are created with an unused regular-user UID, a primary group,
a private `/home/USERNAME` directory, and `/bin/sh` as their login shell. The
defaults are username `root` and password `root`. A custom image must provide:

- `sshd`
- `/etc/passwd`, `/etc/shadow`, and `/etc/ssh/sshd_config`
- `/etc/group` and `/bin/sh` when creating a new user
- A root-owned, setuid `/usr/bin/sudo` and `/etc/sudoers` for custom login users
- systemd and apt for the automatically installed vsock guest agent (Ubuntu 22.04+)

Custom login users receive password-authenticated sudo access. Inside a new VM:

```sh
sudo -i                 # Enter the same password used for this user's SSH login.
sudo apt update
sudo apt install htop
```

`su -` asks for root's separate password; creating a custom login does not set
root's password. Use `sudo -i` for an administrative shell. The default Ubuntu
template retains apt's package metadata and enables `main` and `universe`,
including updates/security repositories. Guest DNS defaults to `1.1.1.1` and
`8.8.8.8` in `/etc/resolv.conf`.

UVM specializes the Ubuntu guest's `/etc/systemd/network/10-uvm.network` with
its assigned `Address=` and `Gateway=` before boot. `KeepConfiguration=yes`
also retains the kernel-created default route while systemd-networkd starts.
Older templates used `KeepConfiguration=static`, which could retain the guest
IP but remove the default route, producing `ping: connect: Network is unreachable`.

The updated CLI repairs an older VM's original, unedited UVM profile on its
next stopped-disk boot, including `restart`. The repair is prepared and checked
on a sparse copy before publication; guest-edited profiles and custom images
are preserved. New VMs also receive the fix when using a cached older Ubuntu
template. See [guest route recovery](USAGE.md#guest-reports-network-is-unreachable)
for immediate recovery inside a running guest.

Existing accounts retain their UID, groups, home directory, and shell. A new
user reuses a same-named group if present; otherwise UVM creates one with an
unused GID. `/etc/gshadow` is updated when present. The default `root` account
must already exist in the image.

The known public demo key bundled in Firecracker's older Bionic image is
removed from each private disk. UVM also replaces the conventional RSA, ECDSA,
and Ed25519 SSH host keys in every private disk so cloned VMs have distinct
host identities. Never rely on SSH private keys baked into a shared template.

## Host Networking

By default, UVM manages:

```text
Network:  10.42.0.0/24
Gateway:  10.42.0.1
Bridge:   uvm0
TAPs:     uvm-<unique suffix>
```

VMs share the host's IPv4 internet connection by default. When the host has
internet access, guests can initiate outbound connections through it. When
the host loses connectivity, guest internet access is lost too; the private
VM network remains usable. Reconnecting the host restores guest egress.

UVM creates the bridge when absent, validates an existing bridge before using
it, enables IPv4 forwarding, and installs subnet-scoped `iptables` masquerade
and forward/return rules. These rules are installed even while the host is
offline and follow its current routing table, so switching uplinks does not
require restarting VMs. Existing connections may need application retries
after an outage or upstream address change. Guest IPs, gateway, bridge name,
and guest boot netmask derive from the configured CIDR.

Forwarding exceptions are inserted ahead of existing host-chain rules and
allow guest-originated traffic and its replies. Masquerading excludes traffic
back onto the UVM bridge, preserving private peer addresses.

To apply updated sharing rules to an existing running VM, use:

```sh
sudo ./uvm.py start web
```

On a running VM, `start` refreshes its private policy and shared host networking
without rebooting it. New VM boots set up the same networking automatically.
For an older guest missing its default route, use `restart` once to apply the
offline guest-profile repair, or follow the guest route recovery steps above.

Private VM creation and start require an RFC 1918 subnet (`10.0.0.0/8`,
`172.16.0.0/12`, or `192.168.0.0/16`). Retained VMs prevent incompatible
gateway, subnet, or bridge changes within the same registry. Private
port enforcement uses separate per-VM native nftables `bridge` tables, not
DNAT or host listeners. Policy is installed and read back before an owned TAP
is activated and before the guest starts. The software bridge must have STP
disabled; each TAP's bridge-slave `group_fwd_mask` is set to `0x4000` so LLDP
cannot bypass the early identity filter. Native bridge conntrack requires
Linux >= 5.3; no `br_netfilter` sysctl setup is required by this policy layer.

Host-to-VM and VM-to-VM traffic crossing these hooks is filtered. An allowed
port is not a reachability guarantee, and a private IP alone is not protection
against administrator-supplied external routing or NAT. Integration with UFW,
firewalld, Docker, or other external firewall managers remains deferred.
Do not offload traffic around the software bridge hooks.

There is no autostart service or daemon reconciliation. After externally lost
or changed private policy, inspect its status and explicitly run `ports sync`.
Use `start` to restore shared bridge/NAT/forwarding rules as well. Read-only
commands do not repair rules; port sync does not boot stopped VMs.

Use one authoritative registry per bridge/subnet. Independent `UVM_BASE`
directories do not coordinate IP or MAC allocation with each other.

## Persistent State

The default data directory is `/var/lib/uvm`:

```text
/var/lib/uvm/
├── bin/                 # Firecracker and Jailer binaries
├── images/              # Kernel and rootfs templates
├── vms/<vm-id>/         # Per-VM disk, config, log, and API socket
├── integrity.json       # Download/build checksums and verification provenance
└── state.json           # Root-only VM inventory and guest credentials
```

The default registry is `/var/lib/uvm/state.json`, schema version `3`. Writes
are locked and atomic. Lifecycle, port-policy, and installation mutations
share an operation lock; bulk operations use a fixed snapshot of VM IDs.

The registry stores guest usernames and passwords in plaintext, so `state.json`
is mode `0600`. Treat it as a secret, do not include it in bug reports, and
replace the default `root` password immediately.

Credential provisioning applies only to VMs created by this version. Existing
guest accounts, passwords, and SSH host keys are retained on subsequent boots.

Older registries are read and migrated on the next write. A missing/null
`ports` policy remains `None`/`unmanaged`, not an empty allowlist or an
implicitly enforced default. Migrate a legacy VM explicitly with `stop`,
`ports set VM --ports 22,80:HTTP -y` (choose the actual required ports), then
`start`. Resume requires a complete, safe private disk, a verified kernel and
VMM, and compatible network settings. Removing the original rootfs template
does not block start of a retained private disk.

Process identity includes PID, Linux process start time, and host boot ID.
A running legacy PID missing the boot ID is adopted only when its saved start
time, running binary, exact API-socket command line, and protected paths all
match. Missing or ambiguous evidence fails closed: UVM will not guess which
process to signal or release a disk while process exit is unconfirmed.

Schema 3 adds disk capacity and durable resize targets. Older registries migrate
on the next write; a recorded pending resize is completed before a stopped VM
can boot. A boot failure after a successful resize retains the new allocation
for a later start.

## Configuration

Host and API settings can be overridden through environment variables:

| Variable | Purpose |
|---|---|
| `UVM_BASE` | Data directory; defaults to `/var/lib/uvm` |
| `UVM_NETWORK` | Guest private IPv4 CIDR; defaults to `10.42.0.0/24` |
| `UVM_GATEWAY` | Bridge/guest gateway; defaults to `10.42.0.1` |
| `UVM_BRIDGE` | Linux bridge name; defaults to `uvm0` |
| `UVM_KNOWN_HOSTS_FILES` | Override local SSH cache files cleaned on deletion; colon-separated absolute paths |
| `UVM_KERNEL_URL` | Override the host-architecture default guest kernel URL |
| `UVM_ROOTFS_URL` | Optional prebuilt ext4 image URL; unset to build Ubuntu 22.04 locally |
| `UVM_FIRECRACKER_SHA256` | Trusted Firecracker release archive SHA-256 |
| `UVM_KERNEL_SHA256` | Trusted guest kernel SHA-256 |
| `UVM_ROOTFS_SHA256` | Trusted prebuilt rootfs SHA-256; use with `UVM_ROOTFS_URL` |
| `UVM_FIRECRACKER_BINARY_SHA256` | Optional trusted local Firecracker binary SHA-256 |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | Set to `1` only for unverified local development |
| `UVM_API_TOKEN` | Required for Nest and standalone UVM servers; sent as `X-UVM-Token` |
| `UVM_API_TLS_CERT` | TLS certificate path required for a non-loopback API bind |
| `UVM_API_TLS_KEY` | TLS private-key path required for a non-loopback API bind |

## Project Layout

Selected files:

```text
./
├── README.md
├── USAGE.md
├── DEVELOPER.md
├── pyproject.toml
├── nest.py              # Executable Nest workspace launcher
├── nest/                # Nest workspace application
│   ├── cli.py           # nest --serve entry point
│   ├── server.py        # Workspace server, composing the UVM management API
│   └── dashboard/       # Nest document route and local HTML, CSS, JS, and fonts
├── uvm.py               # Executable UVM compatibility launcher
├── uvm/                 # Machine-management implementation
│   ├── cli.py           # argparse, terminal output, SSH execution
│   ├── lifecycle.py     # Lifecycle, bulk operations, recovery, port-policy intent
│   ├── network.py       # IP/MAC/TAP/bridge/NAT management
│   ├── firewall.py      # Owned nftables bridge policies and offline conntrack cleanup
│   ├── routers/         # UVM management API routes
│   └── firecracker/     # Firecracker config, API, and process adapters
└── tests/               # Unit tests and opt-in isolated namespace packet tests
```

See [DEVELOPER.md](DEVELOPER.md) for the complete module map and ownership rules.

## Testing

Run the unit suite from this project root:

```sh
python3 -m pip install '.[test]'
python3 -B -m unittest discover -s tests -v
```

The unit tests do not need root, KVM, Firecracker, or host network changes.
They cover validation, state migration and locking, integrity manifests,
network/firewall rendering, Firecracker request ordering, process safety,
retained-disk resume, bulk operations, lifecycle rollback, dashboard delivery,
and API routes. API route tests run when the FastAPI and `httpx` test
dependencies are installed.

When `nft` and `conntrack` are installed, the suite also checks their command
parsers using deliberately invalid final commands/options that prevent mutations.
With e2fsprogs installed, offline disk tests grow and shrink temporary sparse ext4
images and check that data and credential/key fixtures survive; no VM is booted.

Opt-in packet checks run **only in disposable isolated user/network
namespaces**, which may require privileges and the installed network tools:

```sh
UVM_FIREWALL_INTEGRATION=1 python3 -B -m unittest discover -s tests/integration -v
```

The packet test was skipped in this workspace because namespace creation was
denied (`Operation not permitted`). A skip is not packet-path or real-host
validation. The harness uses veths in place of TAPs and simulated upstream
networks to check NAT, offline startup, reconnects, and uplink changes alongside
private port policies. It does not boot a real VM; validate KVM, TAP, SSH, and
guest connectivity on an isolated Linux host.

## Current Boundaries

- Fractional CPU requests still round Firecracker vCPUs up and are advisory.
- No quotas or cgroup CPU, memory, I/O, or process limits are applied.
- Firecracker is launched directly by this local CLI. Jailer/cgroup hardening
  is not implemented by the current command set.
- No VM autostart, background reconciliation, guest application health probing,
  snapshots, DHCP, public-port publishing, or multi-host scheduling.
- The API uses one shared token, not users, roles, or tenants. UVM is not a
  hardened multi-tenant platform.
- Run real KVM, TAP, and guest-connectivity checks on an isolated Linux host
  before relying on UVM for workloads.