nest/USAGE.md

# Nest and UVM Usage Guide

This guide explains how to install, operate, and troubleshoot Nest and its UVM
machine-management backend. It covers the Nest dashboard, UVM command-line
tool, FastAPI server, common scenarios, and expected limitations.

For the project architecture and contributor guidance, see
[DEVELOPER.md](DEVELOPER.md). For a concise overview, see [README.md](README.md).

## 1. What UVM Does

UVM creates small Linux microVMs through Firecracker and KVM on one Linux host.
Each VM receives:

- A unique VM ID and, optionally, a unique local name.
- A private writable root disk copied from a shared template.
- A private IP address and MAC address.
- A TAP interface connected to a host bridge.
- A dedicated Firecracker process, API socket, log, and config file.
- A managed private-IPv4 TCP allowlist, initially allowing port 22 unless
  default ports are explicitly disabled.

By default, guest networking uses:

```text
Guest network: 10.42.0.0/24
Gateway:       10.42.0.1
Bridge:        uvm0
Guest pool:    10.42.0.2 through 10.42.0.254
```

Guest IPv4 internet access shares the host's connection automatically. Host
offline means guest internet is offline; reconnecting the host restores guest
egress. UVM installs sharing rules even when booting offline, and follows the
host's current routes when uplinks change. The private network remains usable
without internet. Applications may need to reconnect after a network interruption.

The default locations on the host are under `/var/lib/uvm`. Stop/start and
restart reuse the private disk; delete removes it. UVM is a local CLI, not an
autostart service or a daemon that continuously repairs VM or firewall state.

## 2. Before You Start

### Host Requirements

You need:

- Linux, preferably Ubuntu or a compatible distribution.
- Python 3.12 or newer.
- KVM enabled and readable/writable at `/dev/kvm`.
- Linux >= 5.3 with native bridge conntrack and safe pidfd support.
- Root access for installation and VM lifecycle operations.
- An x86_64 or ARM64 (`aarch64`) host, including NVIDIA DGX Spark.
- Internet access if downloading Firecracker and guest assets.
- `iproute2`, `iptables`, `nftables`, `conntrack`, `coreutils`, `e2fsprogs`, `util-linux`,
  `openssl`, and `openssh-client`; `install` installs the host packages.
- An STP-disabled software bridge without traffic offload around its hooks.

Check virtualization before installing:

```sh
ls -l /dev/kvm
lscpu | grep -i virtualization
```

### Source Checkout Setup

From this project root:

```sh
./nest.py --help
./uvm.py --help
python3 -B -m uvm --help
```

To install the HTTP server dependencies and console commands:

```sh
python3 -m pip install .
nest --help
uvm --help
```

The rest of this guide uses `./uvm.py` so it works from a source checkout. If
you installed the package, replace `./uvm.py` with `uvm` and `./nest.py` with
`nest`. For the server, use the Python environment containing the installed
dependencies; under `sudo`, use
the full path to that environment's console script when necessary.

### Privilege Rules

| Operation | Root required? |
|---|---|
| `install`, `create`, `start`, `stop`, `restart`, `resize`, `delete`/`destroy` | Yes |
| `ports open/close/set/sync` | Yes |
| `list`, `inspect`, `ports list`, dry-run target lookup | Access to the default root-owned registry requires root |
| `ssh`, `logs` | Access to the protected registry/runtime files requires root |
| `doctor` | Run as root for meaningful privileged checks |
| API server that manages VMs | Yes |

The registry contains plaintext guest credentials and is mode `0600`. Direct
SSH to a known guest IP does not require root.

## 3. Command Overview

The following is syntax notation, not a shell script. `VM` means an exact VM
ID, local name, or guest IPv4 address. Flags are command-specific; in
particular, create has no `--dry-run`, and start has no `--timeout`.
Examples elsewhere are separate scenarios; substitute your VM identifiers
and do not run every destructive example as a single script.

```text
nest --serve [--host HOST] [--port PORT]
uvm install [--force]
uvm create [--name NAME] [--cpu CPU] [--ram RAM] [--storage STORAGE] [--guest-ip GUEST_IP]
           [--username USER] [--password PASSWORD | --password-stdin]
           [--ports PORTS]... [--no-default-ports] [--json]
uvm list [--status STATUS]... [--json | -q | --quiet]
uvm inspect VM [--json]
uvm start (VM [VM ...] | --all) [--yes | -y] [--dry-run] [--json]
uvm stop (VM [VM ...] | --all) [--timeout SECONDS] [--yes | -y] [--dry-run] [--json]
uvm restart (VM [VM ...] | --all) [--timeout SECONDS] [--yes | -y] [--dry-run] [--json]
uvm resize VM [--cpu CPU] [--ram RAM] [--storage STORAGE] [--timeout SECONDS] [--json]
uvm delete (VM [VM ...] | --all) [--timeout SECONDS] [--yes | -y] [--dry-run] [--json]
uvm destroy (VM [VM ...] | --all) [--timeout SECONDS] [--yes | -y] [--dry-run] [--json]
uvm ssh VM [--user USER] [--key PATH] [--port PORT] [--insecure-host-key] [-- COMMAND ...]
uvm logs VM [--tail N] [--follow | -f]
uvm doctor [--json]
uvm ports list (VM | --all) [--json]
uvm ports open VM --ports PORTS [--ports PORTS ...] [--dry-run] [--json]
uvm ports close VM (--ports PORTS [--ports PORTS ...] | --all) [--yes | -y] [--dry-run] [--json]
uvm ports set VM (--ports PORTS [--ports PORTS ...] | --none) [--yes | -y] [--dry-run] [--json]
uvm ports sync (VM [VM ...] | --all) [--yes | -y] [--dry-run] [--json]
uvm --serve [--host HOST] [--port PORT]
```

`--host-ip` is an alias for `--guest-ip`; neither refers to the host's public
or uplink address. `--ports` and `list --status` are repeatable flags, not
space-separated positional lists. `destroy` is an alias for `delete`.

JSON is written to stdout; warnings, errors, prompts, and host-command traces
go to stderr. VM JSON omits passwords. Lifecycle and sync results are arrays of
`id`, `action`, `outcome`, and `error`; dry runs return an object with
`dry_run`, `action`, and `targets` (port edits also include before/after
`changes`). `ports list --json` returns an array even for one VM. JSON does
not disable confirmations, and errors may still be plain text on stderr.

### Stop the Management API

The API can be stopped remotely through its authenticated shutdown endpoint.
This stops the serving Nest or standalone UVM process; it does not stop or
delete any VM:

```sh
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  http://127.0.0.1:8000/exit
```

The server first returns `202 {"status":"stopping"}`, then performs Uvicorn's
graceful shutdown. Use the actual API host and port when they differ from the
defaults. `GET /exit` is intentionally not supported.

## 4. Install UVM Assets

UVM installs host packages, Firecracker, Jailer, a guest kernel, and a guest
rootfs template. The default rootfs is built locally from authenticated Ubuntu
22.04 packages, including OpenSSH, sudo, and complete apt/dpkg metadata. The
downloaded Firecracker archive and kernel require trusted SHA-256 checksums.

### ARM64 Hosts (NVIDIA DGX Spark)

UVM detects the host architecture and selects matching Firecracker and kernel
downloads and native Ubuntu packages. NVIDIA DGX Spark uses `aarch64` artifacts
and `arm64` packages; the `arm64` architecture alias is also accepted:

- Kernel: `https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.15/aarch64/vmlinux-6.1.155`
- Rootfs: Ubuntu 22.04 built with `debootstrap` from `ports.ubuntu.com/ubuntu-ports`.

Use the installation commands below with checksums for the ARM64 kernel
and Firecracker release archive. `UVM_KERNEL_URL` overrides the kernel download;
`UVM_ROOTFS_URL` selects a prebuilt ext4 image instead of the local build. Custom images must
match the host architecture and the [guest image contract](README.md#guest-image-contract).
On ARM64, the kernel must be an uncompressed ARM64 Linux `Image` (PE format),
and the rootfs must be an ARM64 ext4 filesystem. UVM adds `keep_bootcon` to the
guest boot arguments for ARM64 console output.

Both architectures default to the pinned Linux 6.1.155 kernel from the published
`v1.15` CI artifact series, independently of the Firecracker binary version.
These kernels have `CONFIG_HW_RANDOM=y` and `CONFIG_HW_RANDOM_VIRTIO=y` built in.
UVM configures Firecracker's entropy device before boot; custom kernels also
need these drivers to receive host-backed randomness.

### Scenario: Strict First Installation

Obtain trusted checksums independently, then run:

```sh
sudo env \
  UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
  UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
  ./uvm.py install
```

Installation prints the Firecracker, kernel, and rootfs paths under
`/var/lib/uvm`. Checksums and verification provenance are recorded in
`/var/lib/uvm/integrity.json`.

The default build installs `debootstrap` and `ubuntu-keyring` on the host,
checks Ubuntu archive signatures, bootstraps the base release, then uses apt
to apply updates and install the guest packages. It builds inside private
mount/PID namespaces and requires native root execution with namespace support.
Leave `UVM_ROOTFS_URL` and `UVM_ROOTFS_SHA256` unset for this mode. To use a prebuilt
ext4 image, supply both its URL and SHA-256 instead.

### Scenario: Refresh Firecracker or Guest Assets

Use `--force` after changing a URL, replacing an image, or rotating checksums:

```sh
sudo env \
  UVM_FIRECRACKER_SHA256='<new-firecracker-archive-sha256>' \
  UVM_KERNEL_SHA256='<new-kernel-sha256>' \
  ./uvm.py install --force
```

`--force` stages downloads and builds in temporary paths. Downloaded artifacts
are checksum-verified; the built rootfs is checked for installed packages,
sudo permissions, and filesystem consistency before replacement.

To upgrade from the old stripped Bionic demo rootfs, run this refresh with
`UVM_ROOTFS_URL` and `UVM_ROOTFS_SHA256` unset, then create a new VM. Restart a
running API server to load the new provisioner. Existing VM disks retain their
installed OS, files, and credentials.

### Scenario: Local Experimentation Without Checksums

This mode is unsafe for production; use it only when deliberately accepting
unverified development artifacts:

```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py install
```

An unverified manifest is not publisher trust. Reinstall with trusted digests
to establish a verified installation. Until then, retain the explicit opt-out
on later create/start/restart/resize commands and on an API server that creates VMs:

```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py create
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py start VM
```

### Installation Failures

| Message | Meaning and action |
|---|---|
| `/dev/kvm does not exist` | Enable hardware/nested virtualization first. |
| `checksum is required` | Provide the requested `UVM_*_SHA256` variable. |
| `SHA-256 mismatch` | Stop and obtain the correct digest or artifact URL. |
| `unsupported host architecture` | Use an x86_64 or ARM64 Linux host. |
| `unshare: Operation not permitted` | Run the builder as root on a host permitting private mount/PID namespaces. |
| `Ubuntu rootfs build failed` | Read `images/ubuntu-build.log` under `UVM_BASE`, fix the reported bootstrap/apt error, and rerun installation. |

The latest image-build log is retained after temporary build directories are
cleaned up, including detailed debootstrap and apt output:

```sh
sudo tail -n 100 /var/lib/uvm/images/ubuntu-build.log
```

A failed build leaves the previous rootfs template installed. If creation then
reports missing sudo, complete the image installation successfully before
retrying creation. The username itself does not cause that error.

## 5. Create VMs

### Scenario: Create a Default VM

The default is one CPU, 512 MiB RAM, 30 GiB storage, username `root`, password `root`, and an
allowlist containing private TCP/22:

```sh
sudo ./uvm.py create
```

Output includes the ID, optional name, guest IP, resources, TAP, username, and
private port rules. The default password produces a warning on stderr. Use
`--json` for a password-free VM object instead of the human-readable summary.

### Names and Credentials

```sh
sudo ./uvm.py create --name web --username codemate-user --password '<guest-password>'
```

Names are optional, unique within the registry, and case-sensitive. They must
be 1-63 lowercase letters, digits, or hyphens, starting with a letter. Names
that impersonate generated IDs (`vm-` plus 32 hexadecimal characters) are
rejected. Names and IPs stay reserved while a VM is stopped. Names do not
create DNS records, and identifiers are not matched by ID prefix.

`--username` selects the guest login account. If a custom username is missing,
UVM creates it on the private disk with a primary group, `/home/USERNAME`, and
the `/bin/sh` login shell. Custom logins receive sudo access using their guest
password. Existing accounts keep their identity and home.

A `--password` value is visible in process arguments and may enter shell
history. Prefer `--password-stdin`, which reads one line, strips the trailing
line ending, and rejects empty passwords, control characters, or passwords
longer than 128 characters. It cannot be combined with `--password`.

This Bash example prompts without echoing the password and avoids putting a
literal secret in the command line or history. Do not enable shell tracing:

```bash
sudo -v
IFS= read -r -s -p 'Guest password: ' guest_password
printf '\n'
printf '%s\n' "$guest_password" | sudo ./uvm.py create --name app \
  --cpu 1 --ram 1G --guest-ip 10.42.0.10 \
  --ports 80:HTTP,443:HTTPS --password-stdin
unset guest_password
```

UVM still stores the password in plaintext in the root-only registry. Never
reuse a host/service password; replace `root` / `root` outside throwaway tests.

### Initial Ports

```sh
# Adds 80 and 443 to the default private TCP/22 rule.
sudo ./uvm.py create --name frontend --ports 80:HTTP --ports 443:HTTPS

# Exactly TCP/443, with no SSH allowance.
sudo ./uvm.py create --name tls-app --ports 443:HTTPS --no-default-ports

# Managed empty service allowlist, not an unmanaged VM.
sudo ./uvm.py create --name worker --no-default-ports
```

`--ports` uses colon syntax `PORT[:TYPE]`, not slash notation or host:guest
port mappings. See Private Port Policies below for live edits and network
scope. UVM does not install the services named by these labels.

### Scenario: Choose CPU, RAM, and Storage

```sh
sudo ./uvm.py create --cpu 2 --ram 2G --storage 50G
sudo ./uvm.py create --cpu 0.5 --ram 512
```

RAM is in MiB by default. Supported suffixes include `B`, `K`, `M`, `G`,
`KiB`, `MiB`, and `GiB`.

UVM enforces a minimum of 128 MiB. Firecracker uses whole vCPUs. A fractional
request such as `--cpu 0.5` is accepted but rounds the VM up to one Firecracker
vCPU; it does not currently apply a host CPU quota.

Storage defaults to **30 GiB**, with a **30 GiB minimum**. `--storage 50G` gives
the private ext4 root disk 50 GiB of capacity; bare numbers use GiB. Accepted
suffixes include `M`, `G`, `T`, `MiB`, `GiB`, and `TiB` (also `B` and `K`).
`GB` and `TB` use binary units, consistently with RAM. Fractional amounts round
up to whole MiB. The API and JSON output report capacity as `storage_mib`.
Sparse images consume host space as data is written, rather than reserving the
whole configured capacity up front.

### Scenario: Request a Specific Guest IP

```sh
sudo ./uvm.py create --guest-ip 10.42.0.20
# The legacy spelling --host-ip has the same meaning.
```

The address must be a private RFC1918 IPv4 address inside the configured guest
network. It cannot be the network address, broadcast address, or gateway, or
already belong to another persisted VM (including a stopped one).

### What Happens During Create

UVM performs these operations in order:

1. Checks root access, KVM, artifact integrity, and nftables/conntrack support.
2. Reserves identity, name/IP, and initial port policy in the registry.
3. Copies the template into a private disk, resizes ext4 to the requested capacity,
   and provisions credentials and SSH host keys.
4. Ensures bridge/egress networking and creates an owned TAP attached but down.
5. Installs and verifies the nftables bridge policy, then activates the TAP.
6. Starts Firecracker, persists process identity, and configures/boots the guest.
7. Confirms policy enforcement and records `running`.

On failure, UVM attempts containment and cleanup. It does not release the
disk or IP reservation if safe process exit or cleanup cannot be confirmed;
inspect the retained `failed` record and `last_error` before retrying.

After updating UVM to fix a create/cleanup error, restart the API server to
load the updated code. To discard a failed creation and release its reserved
IP, run `sudo ./uvm.py delete VM -y` with that failed VM's exact ID, or send
an authenticated `DELETE /vms/VM`. Deletion retries the outstanding cleanup
and removes the failed VM's private disk. Then retry the create request.

## 6. Inspect VMs

```sh
sudo ./uvm.py list
sudo ./uvm.py list --status running --status dead --json
sudo ./uvm.py list --status stopped -q
sudo ./uvm.py inspect web
sudo ./uvm.py inspect 10.42.0.10 --json
```

List includes names and policy status. `--status` filters by **observed**
status and may be repeated; `-q`/`--quiet` prints IDs only and is mutually
exclusive with `--json`. Inspect includes retained disk/log/config paths,
`disk_ready`, pending `operation`, and `last_error`, without the password.

Status meanings:

| Status | Meaning |
|---|---|
| `starting` | Create/start is in progress or was interrupted. |
| `running` | Firecracker started; this is not an SSH/application readiness probe. |
| `stopping` | Stop is in progress or was interrupted. |
| `stopped` | VMM and TAP are gone; disk remains. |
| `terminating` | Deletion is pending. |
| `failed` | A lifecycle operation or cleanup failed. |
| `dead` | Observed-only status: stored state says running but the VMM is proven gone. |
| `unknown` | Process identity cannot be established safely; do not guess a PID to kill. |

Stored `status` and `observed_status` can differ. Inspection does not restart,
adopt a legacy process, or reconcile a VM. Policy status is a separate snapshot:

| Policy Status | Meaning |
|---|---|
| `enforced` | The actual owned rules and required bridge attachment settings match. |
| `inactive` | Managed VM is cleanly stopped; saved rules apply on next start. |
| `unmanaged` | Legacy record has no managed policy. |
| `missing` | Expected managed nftables table is absent. |
| `drift` | Actual rules, ownership, or bridge attachment settings differ. |
| `quarantined` | The inspected quarantine policy blocks the VM's TAP traffic. |
| `unknown` | Inspection failed or metadata could not establish policy status. |

`enforced` is not a service-connectivity test or a lasting guarantee against
external rule changes. Dry-run target snapshots may show policy `unknown`
because a preview does not verify enforcement. Use inspect or ports list for
a current readback.

### Logs and Doctor

```sh
sudo ./uvm.py logs web
sudo ./uvm.py logs web --tail 50
sudo ./uvm.py logs web --tail 0 -f
sudo ./uvm.py doctor
sudo ./uvm.py doctor --json
```

Logs reads the local Firecracker stdout/stderr log, defaulting to the last
100 lines. `--tail` takes a non-negative integer; `--follow`/`-f` watches for
new output until interrupted. It is not an in-guest log API.

Doctor checks root/KVM access, guest templates and integrity, the non-committing
nftables capability probe, and required command availability. It returns a
nonzero status if any check fails. It does not repair VMs or rules, boot a
guest, or prove SSH/egress connectivity. Use inspect for per-VM policy health.

## 7. Connect with SSH

UVM provisions a password for the requested account, creating missing custom
users during VM creation. The default credentials are `root` / `root`. Your
guest image must include:

- An SSH server.
- Standard passwd and shadow files, plus `/etc/ssh/sshd_config`.
- `/etc/group` and `/bin/sh` for new users; the default `root` account must exist.
- Root-owned setuid `/usr/bin/sudo` and `/etc/sudoers` for a custom login user.

For a custom login, run these commands inside the VM:

```sh
sudo -i                 # Authenticate with the custom user's SSH password.
sudo apt update
sudo apt install htop
```

`su -` requests root's password, which is separate from the custom user's
password. Use `sudo -i` for administrative access. The default Ubuntu image
includes an intact `/var/lib/dpkg` database; an old Bionic demo image missing
that database needs a replacement template and a new VM for package management.

UVM replaces conventional RSA, ECDSA, and Ed25519 SSH host keys in the private
disk during creation and removes Firecracker's publicly known demo login key.

This is applied only while creating a new VM. Start/restart preserves the
existing disk, credentials, and host keys; legacy disks are not changed
retroactively.

```sh
sudo ./uvm.py ssh web
ssh root@10.42.0.2
sudo ./uvm.py ssh web --user ubuntu --key ~/.ssh/id_ed25519 --port 2222
sudo ./uvm.py ssh web -- uname -a
```

By default, UVM uses SSH `StrictHostKeyChecking=accept-new` and a stable
host-key alias derived from the VM ID. This lets a guest IP be reused later
without confusing it with the prior VM's known-hosts entry.

When a VM is deleted, UVM removes its explicit IP and `uvm-<VM-ID>` entries
from local SSH caches. This also covers direct `ssh user@GUEST_IP` connections
and hashed hostnames. Plain `[host]:port` entries are removed for the VM;
hashed port entries are checked for TCP/22 and the ports in the VM's allowlist.
Other hosts (including other aliases on a shared line), wildcard trust rules,
comments, and file ownership/mode are preserved. Missing caches are harmless.

The default files are `~/.ssh/known_hosts` and `~/.ssh/known_hosts2` for the
effective user and, when running under `sudo`, its invoking user. Homes are
resolved from the local account database. The API uses the user context that
launched its server. To select specific local files for a service or custom
SSH configuration, set this variable when launching UVM:

```sh
export UVM_KNOWN_HOSTS_FILES='/root/.ssh/known_hosts:/home/dgxdev/.ssh/known_hosts'
# Pass it explicitly through sudo when starting the API or running deletion.
sudo env UVM_KNOWN_HOSTS_FILES="$UVM_KNOWN_HOSTS_FILES" ./uvm.py delete VM -y
```

These paths are local to the UVM host. An SSH client on another computer manages
its own cache there. For a stale entry left by a VM deleted with an older UVM
version, remove that entry once from the SSH client's account:

```sh
ssh-keygen -f "$HOME/.ssh/known_hosts" -R '10.42.0.2'
```

The key must already be authorized by the guest; UVM does not inject it.
`--port` selects the guest SSH port (default 22), not a host mapping. The
guest's sshd and private allowlist must allow that port. SSH does not feed the
stored password to the client. A remote command must follow a literal `--`;
UVM SSH options can appear before or after the VM identifier.

Only for troubleshooting a guest you trust, disable host-key verification
for one connection:

```sh
sudo ./uvm.py ssh web --insecure-host-key
```

## 8. Start, Stop, Restart, Resize, and Delete

### Scenario: Stop a VM and Keep Its Disk

```sh
sudo ./uvm.py stop web --timeout 5
```

Stop quarantines/deactivates the TAP, terminates Firecracker, removes the TAP,
cleans up VM-scoped connection tracking, and removes the owned policy table.
The private disk, config, log, credentials, name, IP, and saved ports remain.

**Stop is not a guest-coordinated shutdown.** UVM sends SIGTERM to the VMM,
waits, and sends SIGKILL if needed. `--timeout` accepts finite non-negative
seconds, including fractional values, and defaults to `5`. It controls the
SIGTERM wait, not the whole command: confirmed exit after SIGKILL and resource
cleanup can take additional time. `--timeout 0` skips the TERM wait, not exit
confirmation. Flush/quiesce important guest workloads separately before stop,
restart, or delete; retaining a disk does not guarantee application consistency.

### Boot or Restart the Retained Disk

```sh
sudo ./uvm.py start web
sudo ./uvm.py restart web --timeout 10
```

Start boots the same complete private disk, without recopying the template,
resetting passwords, or regenerating SSH host keys. Guest changes survive;
guest RAM/process state does not. The verified kernel and VMM, KVM, firewall
support, and compatible network settings are still required. Removing the
original rootfs template does not prevent retained-disk start, although doctor
checks templates and can report it missing.

Start on a VM recorded as running with a verified process identity synchronizes
its managed port policy without rebooting. Restart is stop followed by start
under the same operation lock. Unknown process identity, an incomplete/unsafe
disk, or pending deletion blocks a new boot; inspect the error rather than
editing registry flags. An interrupted boot with a live VMM must be stopped
before start.

### Scenario: Resize an Existing VM

```sh
sudo ./uvm.py resize web --cpu 2 --ram 4G --storage 80G
sudo ./uvm.py resize web --ram 2G --timeout 10 --json
sudo ./uvm.py resize web --storage 30G
```

Supply at least one resource; omitted fields retain their current values (or
the pending target when retrying an interrupted resize). UVM validates the
request and boot prerequisites, records the target, stops the VM, applies the
changes, and boots it under one operation lock. A stopped VM also boots after
resize. The operation retains the VM ID, IP/MAC, name, username/password, port
rules, guest data, and SSH host keys. The dashboard provides these controls
under **Resize resources** on a machine's detail page.

CPU and RAM can increase or decrease. Storage can grow or shrink to at least
30 GiB, subject to the existing data fitting in ext4. Disk resizing uses a
temporary sparse copy; the checked and resized filesystem replaces the original
only after success. This may require additional physical host space for the copy.

A failed stop prevents resource changes. A failed disk resize leaves the VM
stopped with its retained disk and a `pending_resize` target; correct the cause
and retry `resize` (use a larger storage target if needed). `start` on the stopped
VM also retries the pending target. A boot failure after the new resources were
applied retains those values for the next start. `stop` preserves a pending
resize; `delete` removes the VM and its disk normally.

### Scenario: Permanently Remove a VM

```sh
sudo ./uvm.py delete web --dry-run
sudo ./uvm.py delete web -y
# Equivalent destructive command: sudo ./uvm.py destroy web -y
```

Delete performs the stop cleanup, removes the runtime directory/private disk,
cleans matching local SSH host-key entries, then removes the registry record,
releasing the name and IP. A cache cleanup error retains the pending deletion
and IP reservation so cleanup can be retried before the IP is reused. `destroy` is the same
command with the same flags. Back up data first. A failed deletion remains
pending; retry delete rather than trying to resurrect it with start.

### Multiple VMs and Confirmation

```sh
sudo ./uvm.py stop web app --timeout 10 --json
sudo ./uvm.py start web app
sudo ./uvm.py restart --all --dry-run --json
sudo ./uvm.py restart --all -y
sudo ./uvm.py stop --all -y
sudo ./uvm.py start --all -y
sudo ./uvm.py delete --all --dry-run
sudo ./uvm.py delete --all --timeout 5 -y --json
```

Start/stop/restart/delete accept explicit identifiers **or** `--all`, never both.
Targets are resolved and deduplicated to a fixed ID snapshot before any prompt.
Later-created VMs are not added, and a reused name/IP is not retargeted. If
initial lookup fails, nothing is attempted. Once running, the batch reports
each result and continues after ordinary per-VM failures; it is not all-or-nothing.
An interrupt stops the batch and marks remaining targets skipped.

Every delete and every mutating `--all` selection requires confirmation when
there are targets. Use `--yes`/`-y` for scripts; piping `yes` is not sufficient
because the prompt requires a TTY. `--dry-run` previews without prompting or
performing lifecycle mutations, but is not a boot/firewall preflight. Empty
`--all` selections succeed without prompting. Exit codes are `0` for success
or cancellation, `1` for operational/batch failure, `2` for invalid CLI input,
and `130` for interruption; inspect per-target JSON outcomes as well.

## 9. Private Port Policies

### Syntax and Scope

Use `PORT[:TYPE]`, such as `22`, `80:HTTP`, `443:HTTPS`, and `8080:TCP`.
Ports must be integers from 1 to 65535. CLI labels are case-insensitive and
normalized to `TCP`, `HTTP`, or `HTTPS`; all three use TCP transport. Repeat
`--ports` or separate entries with commas. Identical entries are deduplicated;
conflicting labels for the same port in one request are rejected. Slash
syntax, ranges, UDP labels, and host:guest mappings are not accepted.

Opening a port permits service traffic to the **guest private IPv4 address**
through the UVM bridge/TAP policy, including host-to-guest and VM-to-VM paths.
It does not start a listener, bypass the guest firewall, or configure HTTP or
TLS. Public connectivity, host-port mappings/DNAT, DNS publication, reverse
proxies, and guest-service TLS management are out of scope. External routing
or NAT can change reachability; a private IP is not by itself an exposure
guarantee, and UVM does not integrate with external firewall managers yet.

UDP and IPv6 service policies are not yet supported. Unsolicited inbound UDP
is blocked, as is IPv6 in both directions on managed TAPs. Guest-initiated IPv4
TCP/UDP (including DNS queries), replies to that egress, ARP, ICMP echo, and
required related ICMP errors remain allowed. An empty TCP allowlist is not a
complete network disconnect. Other VMs' policies still apply to guest-to-guest
egress.

### List, Open, Close, and Set

```sh
sudo ./uvm.py ports list web --json
sudo ./uvm.py ports list --all
sudo ./uvm.py ports open web --ports 80:HTTP --ports 443:HTTPS,8080:TCP
sudo ./uvm.py ports close web --ports 80
sudo ./uvm.py ports close web --ports 22 --dry-run --json
sudo ./uvm.py ports close web --ports 22 -y
sudo ./uvm.py ports set web --ports 22,443:HTTPS -y
sudo ./uvm.py ports close web --all -y
sudo ./uvm.py ports set web --none -y
```

- `list` inspects one VM or all VMs; it does not repair rules.
- `open` adds missing ports, preserving other ports and existing labels.
- `close` removes by port number, regardless of label; closing an absent port
  is harmless. `close VM --all` means all ports of **one VM**, not all VMs.
- `set` replaces the entire policy, including labels. `--none` is an explicit
  empty managed allowlist. It is not legacy/unmanaged mode.

Only create adds TCP/22 by default. Set/open/close never add SSH implicitly;
include `22` yourself when replacing a policy if you want to keep SSH.
Open/close/set support `--json` and `--dry-run`. Close/set accept `--yes`/`-y`:
set, close-all, and removing TCP/22 prompt unless approved (an already empty
managed policy can be a no-op). Ordinary open has no confirmation flag.

For a running VM, edits persist the desired policy and apply it atomically.
For a cleanly stopped VM, they update saved metadata for the next start.
After a successful close, subsequent packets on already established service
connections are dropped in **both directions**, as are new connections.
There is no guaranteed TCP RST, immediate socket close, or application-level
disconnect. Live edits retain conntrack state and preserve unrelated egress.

### Explicit Sync and Recovery

```sh
sudo ./uvm.py ports sync web app --dry-run --json
sudo ./uvm.py ports sync web
sudo ./uvm.py ports sync --all -y --json
```

Sync reapplies the saved managed policy to running VMs; it does not start
stopped VMs. It accepts multiple identifiers or `--all`, plus `--dry-run`,
`--json`, and `--yes`/`-y`. `--all` uses the same confirmation and fixed-target
batch behavior as lifecycle commands.

UVM has no background firewall reconciler. If another administrator/tool
removes or changes UVM rules, inspect status and explicitly run `ports sync`
after resolving the external conflict. A failed policy update retains
the new intent with a pending `operation` and attempts TAP deactivation and
quarantine; it does not restore older, more permissive rules. Inspect
`last_error`, fix the underlying failure, then retry sync. Read-only status
commands and starting the API server do not repair lost rules.

### Legacy VM Migration

The default registry is `/var/lib/uvm/state.json`, schema version `3`. Older
registries are accepted and migrated on the next write. Missing/null `ports`
stays `None`/`unmanaged`, not an empty policy and not default SSH enforcement.
Open/close/sync/start do not silently adopt a policy for such a VM:

```sh
sudo ./uvm.py inspect VM --json
sudo ./uvm.py stop VM
sudo ./uvm.py ports set VM --ports 22,80:HTTP -y
sudo ./uvm.py start VM
```

Choose the actual required ports, or use `ports set VM --none -y`. Policy must
first be set while the legacy VM is stopped. Start also requires a safe,
completely provisioned private disk; very old records without one cannot be
resumed by copying a new template over their data.

VMs created before configurable storage retain `storage_mib: null` until their
first resize. That resize uses the existing disk capacity, rounded up to MiB and
at least 30 GiB, unless an explicit storage target is supplied. Merely reading
or migrating the registry does not resize disks.

A live legacy PID missing only the host boot ID can be adopted during a
locked mutation only if its saved start time, running Firecracker executable,
exact API-socket command line, and protected binary/socket paths match. A
missing start token, inaccessible evidence, or any ambiguity fails closed;
UVM will not blindly signal that PID or delete a possibly in-use disk. Listing
does not perform adoption. Inspect such failures rather than manually
inventing identity fields in `state.json`.

## 10. Use the Nest Workspace and FastAPI Server

The API uses the same local lifecycle services and exposes health, installation,
VM lifecycle, diagnostics, bounded VM-log tails, and managed private-port
controls. Nest owns the local dashboard at `/dashboard` and serves it alongside
these UVM-backed endpoints. The API also supports fixed-target bulk actions;
interactive SSH remains CLI-only.
The server does not autostart VMs or reconcile rules, and is not safe to expose
directly to the public internet.

### Start a Local Server

After installing the project dependencies, start the Nest workspace as root:

```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'

sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  ./nest.py --serve --host 127.0.0.1 --port 8000
```

Liveness is at `http://127.0.0.1:8000/health`; the dashboard is at
`http://127.0.0.1:8000/dashboard`; OpenAPI docs are at
`http://127.0.0.1:8000/docs`. The default bind is `127.0.0.1:8000`.

The installed command is `nest --serve`; `python3 -B -m nest --serve` also
works from the source checkout. Nest uses `UVM_API_TOKEN` and the `UVM_*` host
settings for its machine-management backend. To launch the standalone UVM API,
use `./uvm.py --serve` with the same host/port options and configuration. That
server exposes the UVM management endpoints and `/docs`; Nest serves the
dashboard and terminal-session endpoints.

The API token is required even for a loopback server. All VM, diagnostics, and
installation routes require it. The dashboard shell, liveness route, and FastAPI's
informational `/docs`, `/redoc`, and `/openapi.json` endpoints are available
without a token; they do not perform host mutations. The dashboard keeps an
entered token only for the active browser session. Tokens must be visible ASCII
with no whitespace. Use a protected environment file or secret manager rather
than recording a real token in shell history; do not share tokens or guest
passwords in reports.

### Use Sessions Inside a MicroVM

1. Open **Machines** and select the microVM.
2. Start it if it is stopped.
3. Click **Launch session**, or open its **Sessions** tab and launch there.
4. Type directly in the embedded terminal. Each session keeps its own shell,
   current directory, environment, and foreground program.
5. Use **Session history** to switch between that VM's sessions. Active sessions
   reconnect to their existing shell; ended sessions replay saved output.
6. Use **End session** when finished, or run `exit` inside the shell.
7. Use **Rename** in the history row or terminal toolbar to save a custom label.
   Names accept 1–80 printable characters, with surrounding whitespace trimmed.
8. Use **Delete** to permanently remove a session and its saved output. Confirming
   deletion closes an active shell before removing its history. **End session**
   retains history for later viewing.

Sessions continue while you navigate elsewhere, reload, or sign out of the
dashboard. VM shutdown/restart or Nest shutdown ends the shell. Session metadata,
start/end times, exit status, and output survive Nest restarts in
`$UVM_BASE/nest/sessions.sqlite3`; files are mode `0600` in a mode-`0700` directory.
Only sessions belonging to the selected exact VM ID are shown. Reusing a VM name
does not attach another VM's history.
Renaming works for active and ended sessions and keeps their IDs and URLs stable.
Deleting the open session clears the terminal selection; other sessions continue
to appear under their own history entries.

The retained transcript is capped at its latest 8 MiB, and the terminal viewer
has a 10,000-line scrollback. A truncation note appears when older output is
discarded. Terminal input supports pastes up to 64 KiB at a time. Guest output
is saved even while no browser is attached.

Vsock and the guest agent are set up automatically for every microVM during
creation and on each stopped-disk boot. This includes compatible older and custom
images. Restart an older VM once to apply the setup; **Sessions → Guest terminal
support → Repair / update terminal support** remains available for manual repair
while stopped. If the agent is still starting or installing Python, wait for it
to become ready before launching a session.

The terminal transport uses the guest agent over vsock. It logs in as the VM's
configured username; `sudo` inside the terminal uses normal guest authentication.
Nest supports 32 active sessions per server; each guest agent shares eight
connection slots with its command-execution endpoint. Run one Nest session
manager per `UVM_BASE` registry. Reverse proxies must forward WebSocket upgrades.

### Start a TLS Server

Non-loopback binds require a valid TLS certificate and key. The files must be
readable by the server process:

```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'

sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
  UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
  ./nest.py --serve --host 0.0.0.0 --port 8443
```

For real remote use, use a certificate trusted by clients and restrict network
access with a firewall or reverse proxy. Do not use a self-signed development
certificate for an untrusted network.

### API Authentication

Set variables for the request examples:

```sh
export UVM_API_TOKEN='replace-with-the-server-token'
export UVM_API_URL='http://127.0.0.1:8000'
```

Every management request needs the `X-UVM-Token` header, as below.

For a TLS server, set `UVM_API_URL` to an `https://` URL and use your CA:

```sh
export UVM_API_URL='https://host.example:8443'
curl --cacert /path/to/ca.pem \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms"
```

## 11. API Endpoint Examples

| Method | Path | Operation |
|---|---|---|
| `GET` | `/` | Redirect to `/dashboard` (Nest server) |
| `GET` | `/dashboard` | Serve the Nest workspace dashboard (Nest server) |
| `GET` | `/health` | Unauthenticated liveness |
| `GET` | `/diagnostics` | Run host readiness checks |
| `POST` | `/install` | Install/refresh with a `force` boolean |
| `GET` | `/vms` | List VM responses |
| `POST` | `/vms` | Create and boot, HTTP 201 on success |
| `POST` | `/vms/actions` | Run selected/all VM lifecycle actions from a fixed snapshot |
| `GET` | `/vms/{identifier}` | Inspect by exact ID, name, or guest IP |
| `POST` | `/vms/{identifier}/start` | Start a retained VM |
| `POST` | `/vms/{identifier}/restart` | Restart a VM; accepts `timeout` query parameter |
| `POST` | `/vms/{identifier}/resize` | Stop, resize CPU/RAM/storage, and boot; accepts `timeout` query parameter |
| `POST` | `/vms/{identifier}/stop` | Stop and retain disk; accepts `timeout` query parameter |
| `DELETE` | `/vms/{identifier}` | Delete VM and private disk; accepts `timeout` query parameter |
| `POST` | `/vms/{vm_id}/ports` | Set, open, or close private ports for an exact VM ID |
| `POST` | `/vms/{vm_id}/ports/sync` | Reapply the policy for an exact VM ID |
| `POST` | `/vms/{vm_id}/exec` | Execute a guest command/script over vsock, with optional root privileges |
| `POST` | `/vms/{vm_id}/agent/install` | Install/upgrade the agent on a stopped VM's disk |
| `GET` | `/tools` | Authenticated tool catalog and Python version options |
| `GET` | `/vms/{vm_id}/tools` | Cached tool inventory/history; `refresh=true` checks the running guest |
| `POST` | `/vms/{vm_id}/tools/install` | Queue tool installation; HTTP 202 with a job ID |
| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | Installation progress, results, and captured output |
| `GET` | `/vms/{identifier}/logs` | Read a bounded, non-following log tail |
| `GET` | `/vms/{vm_id}/sessions` | List saved sessions for an exact VM ID (Nest) |
| `POST` | `/vms/{vm_id}/sessions` | Launch a terminal, with `{}` or optional `rows`/`cols` (Nest) |
| `GET` | `/vms/{vm_id}/sessions/{session_id}` | Inspect a session (Nest) |
| `PATCH` | `/vms/{vm_id}/sessions/{session_id}` | Rename with `{"name":"Deployment"}`; returns updated metadata (Nest) |
| `DELETE` | `/vms/{vm_id}/sessions/{session_id}` | End the shell and remove its record/output; HTTP 204 (Nest) |
| `POST` | `/vms/{vm_id}/sessions/{session_id}/close` | End a session and retain its output (Nest) |
| `WS` | `/vms/{vm_id}/sessions/{session_id}/terminal` | Authenticated live terminal/replay (Nest) |

Individual lifecycle requests do not expose a CLI dry-run flag and do not prompt
for confirmation. `timeout` is an optional finite, non-negative query parameter
on restart, stop, resize, and delete; it has the same SIGTERM wait semantics as the CLI.

### Health, List, and Inspect

```sh
# Health is unauthenticated and returns {"status":"ok"}.
curl "$UVM_API_URL/health"
curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/vms"
curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/vms/api-web"
curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/vms/10.42.0.30"
# An exact VM ID from create/list can replace the name or IP.
```

Each response contains `id`, `name`, `cpu`, `ram_mib`, `storage_mib`, `pending_resize`, `guest_ip`, `gateway`,
`mac`, `username`, `status`, `observed_status`, `ports`, `policy_status`,
`operation`, `pid`, timestamps, and `last_error`. The response schema excludes
passwords and disk/socket/config/log path fields. Operational error messages
can still contain host paths; review them before sharing.

### Create a VM

Create a named VM with a chosen guest address and private service ports:

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"name":"api-web","cpu":1,"ram":"1G","storage":"50G","guest_ip":"10.42.0.30","username":"root","password":"replace-me","ports":[{"port":80,"protocol":"HTTP"},{"port":443,"protocol":"HTTPS"}],"no_default_ports":false}' \
  "$UVM_API_URL/vms"
```

Create an explicit empty service policy, with no default SSH allowance:

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"name":"api-worker","password":"replace-me","ports":[],"no_default_ports":true}' \
  "$UVM_API_URL/vms"
```

Request fields are `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`,
`ports`, `no_default_ports`, and `tools`. The name has the same validation as the CLI.
`ports` is an array of objects (or null/omitted), not a CLI colon string.
Each object has a strict integer `port` from 1 to 65535 and an optional,
case-sensitive `protocol` of `TCP` (default), `HTTP`, or `HTTPS`.
`no_default_ports` is a strict boolean, default `false`; port 22 is added
unless that flag is true, even if `ports` is null, empty, or contains other
ports. Null create input does not create a legacy/unmanaged VM.
`storage` defaults to `"30G"`; numeric storage values use GiB, with the same
30 GiB minimum and suffixes as the CLI.

The response never includes the password. Omitting credentials still uses
`root` / `root`. Extra request fields, invalid ports/names, and conflicting
labels return HTTP `422`. Example passwords above are placeholders; real
secrets in curl arguments can enter shell history and process listings.

### Install Guest Tools

The dashboard provides tool checkboxes during creation and a **Tools** tab on
each VM. Choose **Install** for a single tool or **Install selected** for a batch.
**Start & install** also boots a stopped VM. The catalog covers Ubuntu 22.04 on
x86_64 and ARM64, with guest internet access for package downloads.

Use the same API selections during creation:

```sh
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"dev-tools","username":"developer","password":"replace-me","tools":[{"id":"git"},{"id":"nodejs"},{"id":"python","version":"3.12"}]}' \
  "$UVM_API_URL/vms"
```

The VM response is HTTP 201 after creation. Selected tools install asynchronously;
read the returned VM's `/tools` endpoint for its job. Omitted `tools` defaults to
an empty list. An explicit null is rejected.

For an existing VM, use its exact ID from create/list:

```sh
export VM_ID='<exact-vm-id>'
curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/tools"

curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"tools":[{"id":"jdk"},{"id":"conda"},{"id":"micromamba"}],"start_if_stopped":true,"request_id":"dev-tools-1"}' \
  "$UVM_API_URL/vms/$VM_ID/tools/install"

# Use the job ID from the HTTP 202 response.
export JOB_ID='<tools-job-id>'
curl -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/$VM_ID/tools/jobs/$JOB_ID"
curl -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/$VM_ID/tools?refresh=true"
```

Selections are objects with `id` and an optional `version`. IDs are `git`,
`nodejs`, `python`, `jdk`, `wget`, `curl`, `htop`, `php`, `conda`, and
`micromamba`. Python requires `version: "3.11"`, `"3.12"`, or `"3.13"`;
the other tools use the curated catalog default and accept no version override.
Up to twelve selections are accepted, with identical selections deduplicated.
Multiple Python versions can be installed together.

Install requests require a nonempty `tools` list. `start_if_stopped` and `restart`
are strict booleans, both default false. Set `restart: true` to restart first and
refresh guest-agent setup. In the dashboard, the **Restart & install** recovery
button appears when agent readiness fails. Unknown tools/options return 422;
unavailable VM states or a different active install return 409.

`request_id` is an optional idempotency key of 1–80 letters, digits, underscores,
or hyphens. Reusing it with the same request returns the retained job; different
selections/options return 409. Use a new key for an intentional retry. Idempotency
keys last as long as their job in the ten-job history. One batch per VM is active
at a time, with two VM workers by default and sequential tools within a batch.

Jobs move through `queued`, `starting`, `waiting`, and `installing`, then
`succeeded`, `failed`, or `interrupted`. Each item reports `installed`,
`already_installed`, `failed`, `interrupted`, or `skipped`, plus its detected
version, error, bounded output, and truncation flag. Ordinary recipe failures
allow subsequent tools to run; uncertain execution outcomes end that batch.
Output is captured as a 16 KiB tail per tool and returned when the step finishes.
Default limits are 120 seconds for agent readiness and 1,200 seconds per tool.

`GET /vms/{vm_id}/tools` returns cached inventory, the latest job, and summaries
of the last ten jobs. `refresh=true` probes all tools in a running VM and updates
`checked_at`; it does not install software or start a stopped VM. `stale` marks
old or stopped-VM observations. Installed binaries and their version checks run
as the configured guest user. Matching installed runtimes are accepted, including
tools installed outside Nest. The install operation is not a general updater.

Tool jobs survive browser navigation, reload, and logout. The active management
server owns the queue through `$UVM_BASE/tool-worker.lock`. After server restart,
queued work resumes; in-flight jobs become interrupted with
`outcome_unknown: true`. The last command may still finish inside the guest.
Refresh inventory before retrying; a guest-side lock blocks overlapping installs.
Stopping/restarting the VM also interrupts a running batch. Successful tools and
the VM's private disk remain available. Deleting the VM removes its tool history.

Use versioned Python commands, for example `python3.12 -m venv .venv`. For Conda
or micromamba activation in a Nest terminal, initialize the current POSIX shell:

```sh
eval "$(conda shell.posix hook)"
conda create -n myenv python=3.13
conda activate myenv

# In another shell, for micromamba:
eval "$(micromamba shell hook --shell posix)"
micromamba create -n myenv -c conda-forge python=3.13
micromamba activate myenv
```

The installer also adds login-shell profiles under `/etc/profile.d`. See
[the catalog](README.md#one-click-guest-tools) for default versions and sources.

### Start, Restart, Stop, and Delete

```sh
# Start a retained private disk.
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/api-web/start"

# Restart with an explicit SIGTERM wait.
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/api-web/restart?timeout=10"

curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/api-web/stop"

# Destructive: no interactive confirmation over HTTP.
curl -X DELETE \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/api-web"
```

Stop returns the VM response with `status: "stopped"`; delete returns its
exact `id` and `status: "terminated"`. Start and restart return the updated VM
response. Start preserves the retained private disk and does not restore guest
memory.

### Resize CPU, RAM, or Storage

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"cpu":2,"ram":"4G","storage":"80G"}' \
  "$UVM_API_URL/vms/api-web/resize?timeout=10"
```

The body accepts optional `cpu`, `ram`, and `storage` fields; at least one must
be non-null. For example, `{"ram":"2G"}` changes only memory. IP and credentials
are retained and cannot be edited through this endpoint. A successful response
is the updated running VM, including `storage_mib` and `pending_resize: null`.
During a failed/interrupted resize, `pending_resize` contains the full target
as `{ "cpu": 2, "ram_mib": 4096, "storage_mib": 81920 }` until it is applied.

### Bulk Lifecycle Actions

The dashboard batch toolbar uses this endpoint for selected VMs. The server
resolves the selected identifiers into a fixed ID snapshot before it invokes the
existing serialized bulk lifecycle service.

```sh
# Restart two named VMs with a shared TERM wait.
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"action":"restart","identifiers":["api-web","worker"],"timeout":10}' \
  "$UVM_API_URL/vms/actions"

# Sync managed policies for every VM currently in the registry.
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"action":"ports-sync","all_vms":true}' \
  "$UVM_API_URL/vms/actions"
```

Valid actions are `start`, `stop`, `restart`, `delete`, and `ports-sync`.
Provide `identifiers` or `all_vms: true`, but not both. The response contains
one `{id, action, outcome, error}` object per selected VM. HTTP bulk requests
do not prompt for confirmation or provide a dry-run; require an explicit
operator confirmation in a caller before destructive use.

### Manage Private Ports

The port mutation and sync routes require an **exact VM ID** returned by create
or list. They deliberately do not resolve a name or guest IP, preventing a
later name/IP reuse from changing the wrong policy. Port rules are strict JSON
objects, not CLI colon strings.

```sh
# Replace the full managed TCP allowlist.
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"mode":"set","ports":[{"port":22,"protocol":"TCP"},{"port":443,"protocol":"HTTPS"}]}' \
  "$UVM_API_URL/vms/<vm-id>/ports"

# Add or remove one or more rules without replacing the other rules.
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"mode":"open","ports":[{"port":80,"protocol":"HTTP"}]}' \
  "$UVM_API_URL/vms/<vm-id>/ports"
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"mode":"close","ports":[{"port":80,"protocol":"HTTP"}]}' \
  "$UVM_API_URL/vms/<vm-id>/ports"

# Reapply a policy after an external firewall change.
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/<vm-id>/ports/sync"
```

`set` can use an empty `ports` array to close every managed service. A legacy
VM with `ports: null` must be stopped and use `set` before it can start or use
the other policy operations. `open`, `close`, and `sync` reject a legacy policy.
Mutating a live policy is subject to the same containment and confirmation rules
as the CLI.

### Execute Commands Without SSH

`POST /vms/{vm_id}/exec` accepts an **exact VM ID** and a JSON body:

| Field | Contract |
|---|---|
| `command` | Required, non-empty command or multiline script; at most 65,536 UTF-8 bytes; no NUL characters |
| `sudo` | Strict boolean, default `false`; select guest root instead of the VM's configured login user |
| `timeout_seconds` | Finite JSON number greater than 0 and at most 3600; default 60 |

```sh
export VM_ID='<exact-vm-id-from-create-or-list>'
curl --fail-with-body -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"command":"whoami","sudo":true,"timeout_seconds":60}' \
  "$UVM_API_URL/vms/$VM_ID/exec"
```

Example response:

```json
{"exit_code":0,"stdout":"root\n","stderr":"","timed_out":false,"truncated":false}
```

The agent runs `/bin/sh -c COMMAND` in the selected user's home directory, with
that user's UID and groups and a fresh `HOME`/`USER`/`PATH` environment. Stdin is
closed and no TTY is allocated. The entire script, including pipes and
redirections, uses the selected identity. For a Bash script already in the VM,
use a command such as `bash /opt/jobs/process.sh`. The flag selects an execution
identity directly; it does not invoke an interactive `sudo` password prompt.
For VMs whose configured username is `root`, both flag values run as root.

Submit a local script's contents using `jq` to handle JSON quoting:

```sh
jq -n --rawfile command ./script.sh \
  '{command:$command,sudo:false,timeout_seconds:300}' |
  curl --fail-with-body -X POST \
    -H "X-UVM-Token: $UVM_API_TOKEN" \
    -H "Content-Type: application/json" \
    --data-binary @- "$UVM_API_URL/vms/$VM_ID/exec"
```

Or call it from Python using the standard library:

```python
import json
import os
from urllib.request import Request, urlopen

request = Request(
    f"{os.environ['UVM_API_URL']}/vms/{os.environ['VM_ID']}/exec",
    data=json.dumps({"command": "id && uname -a", "sudo": False}).encode(),
    headers={"X-UVM-Token": os.environ["UVM_API_TOKEN"],
             "Content-Type": "application/json"},
    method="POST",
)
with urlopen(request, timeout=75) as response:
    result = json.load(response)
print(result["stdout"])
print(result["exit_code"])
```

**Behavior and limits:**

- The VM must be running with a known process identity and no pending lifecycle
  operation. Guest readiness is checked separately by connecting to its agent.
- The call waits for completion. Nonzero command exits still return HTTP 200;
  inspect `exit_code`. Negative codes indicate a terminating signal.
- The deadline is enforced inside the guest. On timeout the agent terminates
  the job, including descendants in its cgroup, and returns partial output with
  `timed_out: true` and `exit_code: null`. Connection and cleanup time are additional
  to `timeout_seconds`; give the HTTP client about 15 extra seconds.
- The agent captures up to **1 MiB combined stdout/stderr**, then drains and
  discards excess data. `truncated` reports this limit; invalid UTF-8 is replaced.
- Up to eight connections/jobs per VM are admitted. Excess connections fail
  before submission. Execution does not hold the global lifecycle lock while
  waiting, so another request can stop or restart the VM.
- A normal script exit also cleans up background descendants in the job's
  cgroup. Use a guest service manager for workloads that should outlive the call.
- A lost connection after submission returns HTTP 502 with
  `error.outcome_unknown: true`. The command may already have produced effects;
  it is never automatically resubmitted. An HTTP client disconnect does not
  itself cancel the guest job; the guest deadline still applies.
- Results are returned in the response, not stored as durable jobs. Commands
  and their captured output are not copied to the VM registry or host traces.

#### Prepare New or Existing VMs

The default Ubuntu template includes Python 3 and `uvm-agent.service`. UVM also
installs the service automatically on compatible older/custom images during
creation and before booting a retained private disk. Every VM gets a Firecracker
vsock device by default. Agent files, VM identity configuration, the ownership
manifest, and the enabled service are checked; an already-current installation
does not rewrite the disk.

Restart a running Nest or UVM API server to load the new code. Existing VMs pick
up the setup on their next start or restart. For a currently running VM:

```sh
curl --fail-with-body -X POST -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/$VM_ID/restart"
```

The explicit `POST /vms/{vm_id}/agent/install` endpoint is available for manual
repair while stopped. It returns
`{"id":"...","status":"installed","restart_required":true}` and does not start
the VM itself. All agent installation requires a proven-offline private disk;
updates are checked on a sparse copy before publication. Guest files,
credentials, and SSH keys are retained; customized agent files are preserved
rather than overwritten. Repeating installation also upgrades unedited managed
agent files using their recorded checksums.

The guest image must provide systemd and apt (Ubuntu 22.04 or newer is the
supported baseline). If Python 3.10+ is missing, a bootstrap service installs
`python3` inside the VM on its next boot. This one-time install needs guest DNS,
internet access, and working apt repositories. The service retries failed startup;
execution returns 503 until it can connect. Current default templates already include
the runtime and do not need this download.

The guest kernel needs built-in `CONFIG_VSOCKETS=y`, `CONFIG_VIRTIO_VSOCKETS=y`,
and `CONFIG_CGROUPS=y`, plus cgroup v2 and `cgroup.kill` (Linux 5.14+). UVM's
default Linux 6.1 kernel supports these. Refresh an older installed kernel as
described in the installation/entropy recovery sections before booting.

All transport is via a per-VM Unix socket and Firecracker vsock port 1070. This
management channel is independent of the IPv4 port policy, including TCP/22;
the guest agent accepts connections from the host CID only. VM execution does
not need SSH credentials, SSH host-key caches, or an open guest TCP port.

### Diagnostics and Log Tails

The dashboard's System view calls the protected diagnostics endpoint. It runs
the same host-readiness checks as `uvm doctor`, but does not repair a host or
change VM state.

```sh
curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/diagnostics"

# Read the final 200 lines without opening a streaming follow request.
curl -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/api-web/logs?tail=200"
```

Log responses have `content` and `truncated` fields. `tail` is an integer from
0 through 1000, defaulting to 100. Reads are limited to the final 256 KiB of a
log; `truncated: true` means an oversized line or the read bound prevented a
complete requested tail. Use `uvm logs VM --follow` for continuous output.

### Install or Refresh Through the API

The server process must itself have checksum variables configured for strict
installation. Start the server with those environment variables before using
this endpoint.

For a deliberately unverified local-development server, the server process
must instead include `UVM_ALLOW_UNVERIFIED_DOWNLOADS=1`. Do not use that mode
on a host that runs untrusted workloads.

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"force": false}' \
  "$UVM_API_URL/install"
```

Use `{"force": true}` to refresh all downloaded artifacts.

## 12. API Errors

| Status | Typical meaning |
|---|---|
| `401` | Missing or incorrect `X-UVM-Token`. |
| `403` | UVM process lacks root privileges for a host mutation. |
| `404` | Requested VM was not found. |
| `409` | Name/identifier conflict, incompatible retained state, or an unsafe/pending operation. |
| `422` | Invalid JSON body, extra request fields, or invalid UVM input. |
| `500` | Host command, Firecracker, state, or unexpected operational failure. |
| `502` | Guest execution failure or lost/invalid execution response; inspect `error.outcome_unknown`. |
| `503` | Guest agent unavailable, not ready, or at its connection limit before command submission. |

UVM operation errors use this envelope:

```json
{"error": {"message": "human-readable explanation"}}
```

FastAPI request-schema errors use FastAPI's normal `detail` response format.
Guest execution errors additionally include a boolean `outcome_unknown` in the
`error` object; `true` means the caller must not assume the command did not run.

## 13. Configuration Scenarios

### Custom Storage and Network

For example, install under `/srv/uvm` with a different private subnet. All
binaries, images, VM runtime directories, locks, and state use the base path:

```sh
sudo env \
  UVM_BASE='/srv/uvm' \
  UVM_NETWORK='10.50.0.0/24' \
  UVM_GATEWAY='10.50.0.1' \
  UVM_BRIDGE='uvm50' \
  UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
  UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
  ./uvm.py install
```

Use the same base/network/gateway/bridge values for all later lifecycle,
inspection, port, and server commands. Start rejects conflicting retained
network settings. Use a non-overlapping private subnet; do not assume UVM
automatically detects every conflict with host routes or other managers.

### Environment Variables

| Variable | Use |
|---|---|
| `UVM_BASE` | Base data directory; default `/var/lib/uvm` |
| `UVM_NETWORK` | Guest subnet; default `10.42.0.0/24` |
| `UVM_GATEWAY` | Guest gateway / bridge IP; default `10.42.0.1` |
| `UVM_BRIDGE` | Linux bridge name; default `uvm0` |
| `UVM_KNOWN_HOSTS_FILES` | Override deletion's local SSH cache files; colon-separated absolute paths |
| `UVM_KERNEL_URL` | Override the host-architecture default kernel URL |
| `UVM_ROOTFS_URL` | Optional prebuilt ext4 URL; unset to build Ubuntu 22.04 locally |
| `UVM_FIRECRACKER_SHA256` | Strict Firecracker archive checksum |
| `UVM_KERNEL_SHA256` | Strict kernel checksum |
| `UVM_ROOTFS_SHA256` | Strict prebuilt rootfs checksum, used with `UVM_ROOTFS_URL` |
| `UVM_FIRECRACKER_BINARY_SHA256` | Optional local binary checksum |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | Set to `1` only for local development |
| `UVM_API_TOKEN` | Required API token |
| `UVM_API_TLS_CERT` | TLS certificate for remote server binds |
| `UVM_API_TLS_KEY` | TLS key for remote server binds |

## 14. Files, Logs, and State

The default host layout is:

```text
/var/lib/uvm/
├── bin/                 # Firecracker and Jailer binaries
├── images/              # Shared kernel and rootfs template
├── vms/<vm-id>/         # Private rootfs, config, socket, and log
├── integrity.json       # Artifact checksums and verification status
└── state.json           # Mode-0600 VM inventory and plaintext guest credentials
```

Registry writes are atomic, with a short `state.lock`; lifecycle, policy, and
installation mutations share `operations.lock`. Avoid editing `state.json`,
runtime files, TAPs, or UVM-owned nftables/iptables rules by hand. Use supported
commands so ownership, locking, and cleanup remain correct. Back up important
disks while stopped and keep the associated registry backup protected.

`state.json` contains plaintext guest passwords. Keep it root-only, never copy
it into logs or bug reports, and use unique non-default passwords outside local
throwaway environments. Prefer inspect/list to dumping the registry. Raw
Firecracker logs may include guest/application secrets; redact them before sharing.

## 15. Networking Troubleshooting

UVM enables IPv4 forwarding and installs `uvm-egress`-commented iptables
MASQUERADE and forward/return rules. These match the private subnet and bridge
without fixing an uplink, so guest traffic follows the host's current routes.
Scoped forwarding exceptions are inserted ahead of existing host-chain rules.
Private port filtering is a separate native nftables `bridge` policy, not
DNAT. It must be installed and read back before the TAP is active. The bridge
must have STP disabled and each TAP must have bridge-slave `group_fwd_mask`
`0x4000` (16384), so LLDP takes the early identity-filter path rather than
bypassing it before MAC learning. These are software-bridge requirements;
do not offload traffic around the hooks. Native bridge conntrack needs Linux
>= 5.3; this layer does not configure `br_netfilter` sysctls.

Check the bridge and TAP state:

```sh
ip -d link show uvm0
ip addr show dev uvm0
ip route show default
```

Check port policy and the separate egress NAT/forwarding rules:

```sh
sudo ./uvm.py doctor
sudo ./uvm.py inspect web --json
sudo ./uvm.py ports list --all
sudo nft --json list tables bridge
sudo iptables -t nat -S POSTROUTING
sudo iptables -S FORWARD
```

After upgrading UVM, apply the sharing rules to an already-running VM with:

```sh
sudo ./uvm.py start web
```

`start` on a running VM refreshes its private policy and shared host networking
without a reboot. Use the updated CLI checkout, or restart the API server to
load the updated code before using its start endpoint. Subsequent host
disconnects, reconnects, and route changes require no UVM command.

### Guest Reports `Network is Unreachable`

An immediate `ping: connect: Network is unreachable` inside the VM indicates
that the guest has no usable route to that destination. Inspect the guest's
networking from its SSH shell:

```sh
ip -4 address show dev eth0
ip -4 route
```

For the default UVM subnet, the route must include `default via 10.42.0.1 dev
eth0`. The guest gateway is the **UVM bridge address**, even when the host's
own uplink uses a different gateway.

Older UVM Ubuntu images used `KeepConfiguration=static`. Ubuntu's network
manager could preserve the guest address but remove its kernel-created
`proto boot` default route. Inside the running guest, restore it immediately:

```sh
sudo ip -4 route replace default via 10.42.0.1 dev eth0 proto static
ping -c 3 1.1.1.1
```

For a persistent repair, run the updated CLI **on the host**:

```sh
sudo ./uvm.py restart 10.42.0.2
```

This stops the VM, repairs its original UVM network profile on the retained
disk, and boots it with explicit address/gateway settings. A stopped VM's
`start` applies the same repair; `start` on an already-running VM refreshes
host rules only. Per-VM provisioning also repairs cached older Ubuntu
templates when creating new VMs. Use the configured `UVM_GATEWAY` and guest IP
in place of these defaults for a custom subnet.

Guest-edited profiles and custom images are preserved. For those, persist the
correct `Address=` and `Gateway=` in the guest's own network configuration.

### Verify Guest Internet Access

Check DNS and HTTPS from the guest:

```sh
sudo ./uvm.py ssh web -- getent ahostsv4 example.com
sudo ./uvm.py ssh web -- curl -4 --connect-timeout 5 --max-time 15 -I https://example.com
```

Common issues:

| Symptom | First checks |
|---|---|
| Guest cannot reach the internet | Confirm the host can reach the same destination, bridge/TAP are up, and `uvm-egress` NAT/FORWARD rules exist; use `start VM` to refresh sharing. |
| Guest immediately reports `Network is unreachable` | Inspect the guest's default route and apply the guest route recovery steps above. |
| IP connectivity works but DNS fails | Check guest `/etc/resolv.conf` and DNS reachability. The Ubuntu template uses `1.1.1.1` and `8.8.8.8`; host loopback resolvers such as `127.0.0.53` are not reachable as guest loopback addresses. |
| Requested guest IP is rejected | Confirm it is inside the configured subnet and not used, gateway, network, or broadcast. |
| Create says TAP already exists | Inspect the interface; do not delete it unless you know it is stale UVM state. |
| Existing bridge is rejected | Verify it is a software Linux bridge with STP disabled. |
| Firewall unavailable/unknown | Check privileges, `nft`, `conntrack`, kernel support, and bridge/TAP settings. |
| Policy missing/drift after external reload | Resolve the other manager's conflict, then explicitly run `ports sync VM`; there is no automatic repair. |
| SSH cannot connect | Confirm boot/sshd, guest user and password/key, chosen SSH port, allowlist, and guest firewall. |
| Closed SSH session hangs rather than disconnects | Close drops subsequent traffic in both directions; a TCP RST is not guaranteed. |

External firewall integration (including UFW, firewalld, Docker, and libvirt)
is deferred. A UVM policy readback does not prove that external rules permit
traffic, or that they will not later remove UVM's rules.

### SSH Startup and Guest Entropy

`running` checks the Firecracker process, and `enforced` checks the private
firewall policy. Neither probes the guest's SSH listener. If ping works but
TCP/22 returns `Connection refused`, inspect the guest boot log:

```sh
sudo ./uvm.py logs VM --tail 200
sudo ./uvm.py inspect VM --json
```

An SSH service stuck at `Starting OpenBSD Secure Shell server...`, together
with an unfinished `Load/Save Random Seed` service and no `random: crng init done`,
strongly suggests entropy starvation. A login prompt can still appear while
SSH is waiting for its cryptographic random generator. On the old ARM demo
kernel, `random: fast init done` is not full cryptographic RNG initialization.

Earlier UVM versions downloaded a Linux 4.14.174 ARM quickstart kernel without
the built-in virtio RNG driver and did not attach an entropy device. Recovery
requires **both** the updated UVM boot code and an RNG-capable kernel. A restart
with the old cached kernel is insufficient.

From the updated checkout, stop the affected VM before refreshing assets.
Stop any other VMs using the same installed Firecracker binary before the
forced reinstall as well. Use the Firecracker archive checksum for the configured
release and a new trusted checksum for the selected architecture's Linux 6.1.155
kernel; the old kernel checksum will not match:

```sh
sudo ./uvm.py stop VM
sudo env -u UVM_KERNEL_URL \
  UVM_FIRECRACKER_SHA256='<trusted-firecracker-archive-sha256>' \
  UVM_KERNEL_SHA256='<trusted-new-kernel-sha256>' \
  ./uvm.py install --force
sudo ./uvm.py start VM
sudo ./uvm.py logs VM --tail 100
sudo ./uvm.py ssh VM
```

This uses the default kernel URL and boots the retained private disk with the
updated device configuration. The VM's identity, files, credentials, and SSH
host keys are retained. `install --force` also refreshes the shared rootfs
template for future creates. If using a custom kernel, supply its URL and
matching checksum with the required drivers instead. For an explicitly
unverified development installation, retain the existing
`UVM_ALLOW_UNVERIFIED_DOWNLOADS=1` setting as described in the installation section.

Restart a running Nest server to load the updated code before using dashboard
start/restart actions. Look for `Started OpenBSD Secure Shell server` in the new
boot log and confirm an actual SSH connection. `start` on an already-running
VM only refreshes networking; use a stopped VM or `restart` to attach the device.

## 16. Important Limitations

Keep these constraints in mind when deciding whether UVM fits a scenario:

- Firecracker is launched directly; the downloaded Jailer is not yet used.
- No quotas or cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU requests are advisory only.
- No autostart, background lifecycle/firewall reconciliation, or guest-coordinated stop.
- No UDP/IPv6 service policies, public inbound connectivity setup, host-port mapping,
  DNS publication, guest reverse proxy/TLS setup, DHCP, load balancing,
  snapshots, multi-network tenancy, or multi-host scheduling.
- The API uses one shared bearer token, not individual users, roles, or tenants.
- Execution probes guest-agent readiness; VM list/status still describes the
  Firecracker process rather than guest application health.
- UVM is not a hardened multi-tenant platform.

## 17. Safe Operating Checklist

Before relying on a VM for useful work:

1. Use strict checksums for installed artifacts.
2. Confirm `/dev/kvm` and nested virtualization are available.
3. Confirm UVM generated unique guest SSH host keys and set a non-default password.
4. Test guest egress and SSH on the actual host network.
5. Quiesce guest workloads before stopping/restarting and back up data before deletion.
6. Keep the API loopback-only unless you have a strong remote-access need.
7. For remote API access, use a long random token, valid TLS, and firewall
   restrictions.
8. Review `/var/lib/uvm/vms/<vm-id>/firecracker.log` after failed starts.

## 18. Verification

From the project root, normal unit tests need no root, KVM, or host network
changes. Install the test extra in your Python environment for API tests:

```sh
python3 -m pip install '.[test]'
python3 -B -m unittest discover -s tests -v
```

The opt-in packet test runs **only in disposable isolated user/network
namespaces**, which may require privileges. It needs `unshare`, `ip`,
`bridge`, `nft`, `iptables`, `conntrack`, and `sysctl`:

```sh
UVM_FIREWALL_INTEGRATION=1 python3 -B -m unittest discover -s tests/integration -v
```

Namespace creation was denied in this workspace (`Operation not permitted`),
so the packet test skipped. That is a permission limitation, not real-packet
or real-host validation. The harness uses veths as TAP stand-ins and simulated
upstreams to check NAT, offline startup, reconnects and uplink changes. It does
not boot Firecracker or validate the host's real TAP setup. Run actual
VM/SSH/egress checks on an isolated Linux host before relying on the deployment.

The execution smoke tests target an already-created **disposable VM on an
isolated host**, with a non-root login, an installed/running agent, and an empty
TCP allowlist. They check both privilege modes, output and exit status, bounded
output, and cgroup cleanup of a descendant that starts its own session. They do
not create, stop, or delete VMs:

```sh
UVM_EXEC_INTEGRATION=1 UVM_EXEC_VM_ID='<disposable-vm-id>' \
  python3 -B -m unittest discover -s tests/integration -p test_guest_exec.py -v
```

Set `UVM_API_URL` and `UVM_API_TOKEN` as in the API examples. For TLS, the Python
client uses the system trust store (or `SSL_CERT_FILE`). Without the explicit
integration flag these tests skip and make no requests. The ordinary unit tests
exercise real child processes with fake cgroup scopes, local Unix-socket protocol
fixtures, and temporary ext4 upgrades; they do not prove the real vsock/cgroup
path inside a VM.

The tools smoke test installs the full catalog in a prepared **disposable Ubuntu
22.04 VM with a non-root login and no open TCP ports**. It downloads packages,
checks all runtimes, compiles Java, creates Python venvs, and creates/activates
temporary Conda and micromamba environments as the guest user. Run it separately
on x86_64 and ARM64 hosts:

```sh
UVM_TOOLS_INTEGRATION=1 UVM_TOOLS_VM_ID='<disposable-vm-id>' \
  python3 -B -m unittest discover -s tests/integration -p test_guest_tools.py -v
```

Set `UVM_API_URL` and `UVM_API_TOKEN` as above. The test is opt-in and was not run
against a real microVM during this implementation. Automated API and browser
checks use simulated guest execution and real job persistence/workers.

## 19. Get Help

```sh
./uvm.py --help
./uvm.py create --help
./uvm.py ports set --help
./uvm.py ssh --help
```

When reporting a problem, collect:

- The command or HTTP request, with passwords and tokens redacted.
- The full error message.
- Password-free output from `list --json`, `inspect VM --json`, and `doctor`.
- The VM Firecracker log, reviewed for guest/application secrets.
- The relevant `UVM_*` variables, with secrets such as `UVM_API_TOKEN` removed.

Never include `/var/lib/uvm/state.json` because it contains guest passwords.