nest/USAGE.md

# UVM Usage Guide

This guide explains how to install, operate, and troubleshoot UVM as an
operator. It covers the command-line tool, the optional FastAPI server, common
scenarios, and expected limitations.

For the project architecture and contributor guidance, see
[DEVELOPER.md](DEVELOPER.md). For a concise overview, see [README.md](README.md).

## 1. What UVM Does

UVM creates small Linux microVMs through Firecracker and KVM on one Linux host.
Each VM receives:

- A unique VM ID.
- A private writable root disk copied from a shared template.
- A private IP address and MAC address.
- A TAP interface connected to a host bridge.
- A dedicated Firecracker process, API socket, log, and config file.

By default, guest networking uses:

```text
Guest network: 10.42.0.0/24
Gateway:       10.42.0.1
Bridge:        uvm0
Guest pool:    10.42.0.2 through 10.42.0.254
```

The default locations on the host are under `/var/lib/uvm`.

## 2. Before You Start

### Host Requirements

You need:

- Linux, preferably Ubuntu or a compatible distribution.
- Python 3.12 or newer.
- KVM enabled and readable/writable at `/dev/kvm`.
- Root access for installation and VM lifecycle operations.
- An x86_64 host for the default guest kernel/rootfs downloads.
- Internet access if downloading Firecracker and guest assets.

Check virtualization before installing:

```sh
ls -l /dev/kvm
lscpu | grep -i virtualization
```

### Source Checkout Setup

From the parent directory of this project:

```sh
cd uvm
./uvm.py --help
python3 -m uvm --help
```

To install the optional HTTP API dependencies and the console command:

```sh
cd uvm
python3 -m pip install .
uvm --help
```

The rest of this guide uses `./uvm.py` so it works from a source checkout. If
you installed the package, replace `./uvm.py` with `uvm`.

### Privilege Rules

| Operation | Root required? |
|---|---|
| `install` | Yes |
| `create` | Yes |
| `stop` | Yes |
| `destroy` | Yes |
| `list` | Yes with the default root-owned registry |
| `ssh` | Yes when resolving a VM through the registry |
| API server that manages VMs | Yes |

The registry contains plaintext guest credentials and is mode `0600`. Direct
SSH to a known guest IP does not require root.

## 3. Command Overview

```sh
./uvm.py --help

./uvm.py install [--force]
./uvm.py create [--cpu CPU] [--ram RAM] [--host-ip GUEST_IP] \
  [--username USER] [--password PASSWORD]
sudo ./uvm.py list
sudo ./uvm.py ssh VM_OR_IP [--user USER] [--key PATH] [--insecure-host-key]
./uvm.py stop VM_ID
./uvm.py destroy VM_ID

./uvm.py --serve [--host HOST] [--port PORT]
```

`--host-ip` is a legacy CLI name. It means the **guest IP address**, not the
host's public or uplink address.

## 4. Install UVM Assets

UVM installs host packages, Firecracker, Jailer, a guest kernel, and a guest
rootfs template. It requires SHA-256 checksums by default because it downloads
artifacts that will be used by a privileged process.

### Scenario: Strict First Installation

Obtain trusted checksums independently, then run:

```sh
cd uvm

sudo env \
  UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
  UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
  UVM_ROOTFS_SHA256='<sha256-of-rootfs>' \
  ./uvm.py install
```

On success, UVM prints paths similar to:

```text
Firecracker: /var/lib/uvm/bin/firecracker
Kernel:      /var/lib/uvm/images/vmlinux
Rootfs:      /var/lib/uvm/images/ubuntu.ext4
```

The checksums and whether the installation was verified are recorded in:

```text
/var/lib/uvm/integrity.json
```

### Scenario: Refresh Firecracker or Guest Assets

Use `--force` after changing a URL, replacing an image, or rotating checksums:

```sh
sudo env \
  UVM_FIRECRACKER_SHA256='<new-firecracker-archive-sha256>' \
  UVM_KERNEL_SHA256='<new-kernel-sha256>' \
  UVM_ROOTFS_SHA256='<new-rootfs-sha256>' \
  ./uvm.py install --force
```

`--force` downloads to temporary files and only replaces the existing artifact
after checksum verification succeeds.

### Scenario: Local Experimentation Without Checksums

This mode is unsafe for production. It is useful only when you deliberately
accept the source artifacts without checksum verification:

```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py install
```

An unverified install is marked as unverified. It cannot later become a strict
trusted installation merely by setting checksum variables. Reinstall with the
strict command above when you are ready to trust the host assets.

For later local-development VM creation, keep the opt-out on the command too:

```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py create
```

The same rule applies to an API server that creates VMs from an unverified
installation: start that server with `UVM_ALLOW_UNVERIFIED_DOWNLOADS=1`, or
perform a strict reinstall first.

### Installation Failures

| Message | Meaning and action |
|---|---|
| `/dev/kvm does not exist` | Enable hardware/nested virtualization first. |
| `checksum is required` | Provide the requested `UVM_*_SHA256` variable. |
| `SHA-256 mismatch` | Stop and obtain the correct digest or artifact URL. |
| `unsupported host architecture` | Use x86_64 defaults or provide compatible assets. |
| `default guest assets support x86_64 only` | Configure both custom kernel and rootfs URLs for ARM. |

## 5. Create VMs

### Scenario: Create a Default VM

The default is one CPU, 512 MiB RAM, username `root`, and password `root`:

```sh
sudo ./uvm.py create
```

Typical output includes the VM ID, guest IP, TAP name, and SSH target:

```text
VM created: vm-...
  IP:       10.42.0.2
  RAM:      512 MiB
  CPU:      1.0
  TAP:      uvm-...
  Username: root
  SSH:      ssh root@10.42.0.2

WARNING: the guest is using the default password 'root'. Change it promptly.
```

Set a different password during creation:

```sh
sudo ./uvm.py create --username root --password '<guest-password>'
```

`--username` must name an account already present in the guest image. The
password is visible in the process arguments and may be recorded in shell
history, so avoid reusing a sensitive host or service password.

### Scenario: Choose CPU and RAM

```sh
sudo ./uvm.py create --cpu 2 --ram 2G
sudo ./uvm.py create --cpu 1 --ram 1024M
sudo ./uvm.py create --cpu 1 --ram 768
```

RAM is in MiB by default. Supported suffixes include `B`, `K`, `M`, `G`,
`KiB`, `MiB`, and `GiB`.

UVM enforces a minimum of 128 MiB. Firecracker uses whole vCPUs. A fractional
request such as `--cpu 0.5` is accepted but rounds the VM up to one Firecracker
vCPU; it does not currently apply a host CPU quota.

```sh
sudo ./uvm.py create --cpu 0.5 --ram 512
```

### Scenario: Request a Specific Guest IP

```sh
sudo ./uvm.py create --host-ip 10.42.0.10
```

The address must be inside the configured guest network, must not be the
network address, broadcast address, or gateway, and must not already belong to
another persisted VM.

### What Happens During Create

UVM performs these operations in order:

1. Validates root access, KVM, assets, and integrity state.
2. Reserves a VM ID, IP, MAC, and state record.
3. Creates `/var/lib/uvm/vms/<vm-id>/`.
4. Copies the rootfs template to a private `rootfs.ext4` disk.
5. Sets the requested account password and enables SSH password login in the copy.
6. Ensures the bridge/NAT/forwarding rules exist.
7. Creates a TAP device.
8. Starts and configures Firecracker.
9. Marks the VM as `running`.

If a step fails, UVM attempts to terminate the new VMM, remove the TAP, remove
the private runtime directory, and release the reservation.

## 6. Inspect VMs

```sh
sudo ./uvm.py list
```

Example:

```text
ID                                    IP               USER             CPU     RAM      STATUS     PID
vm-0123456789abcdef...                10.42.0.2       root             1.0     512      running    12345
```

Status meanings:

| Status | Meaning |
|---|---|
| `starting` | VM creation is in progress. |
| `running` | Firecracker started successfully. |
| `stopping` | Stop is in progress. |
| `stopped` | VMM and TAP are gone; disk remains. |
| `terminating` | Destruction is in progress. |
| `failed` | A lifecycle operation or cleanup failed. |
| `dead` | Stored state claims the VM is active but its VMM process is gone. |

If a VM is `failed` or `dead`, inspect its log before deciding whether to stop
or destroy it:

```sh
sudo ls -la /var/lib/uvm/vms/<vm-id>
sudo cat /var/lib/uvm/vms/<vm-id>/firecracker.log
```

## 7. Connect with SSH

UVM provisions a password for an existing account. The default credentials are
`root` / `root`. Your guest image must include:

- An SSH server.
- The requested login user plus standard passwd and shadow files.

UVM replaces conventional RSA, ECDSA, and Ed25519 SSH host keys in the private
disk during creation and removes Firecracker's publicly known demo login key.

This is applied only while creating a new VM. Existing VM disks and legacy
registry entries are not changed retroactively.

### Scenario: SSH as Root

```sh
sudo ./uvm.py ssh vm-<id>
ssh root@10.42.0.2
```

### Scenario: Use a Different User and Private Key

```sh
sudo ./uvm.py ssh vm-<id> --user ubuntu --key ~/.ssh/id_ed25519
```

By default, UVM uses SSH `StrictHostKeyChecking=accept-new` and a stable
host-key alias derived from the VM ID. This lets a guest IP be reused later
without confusing it with the prior VM's known-hosts entry.

### Scenario: Temporarily Bypass Host-Key Verification

Use only for troubleshooting a guest you trust:

```sh
sudo ./uvm.py ssh vm-<id> --insecure-host-key
```

This disables normal host-key checking for that connection.

## 8. Stop and Destroy VMs

### Scenario: Stop a VM and Keep Its Disk

```sh
sudo ./uvm.py stop vm-<id>
```

Stopping terminates Firecracker and deletes the TAP device. The private root
disk, config, and log remain under the VM runtime directory.

UVM does not currently offer a `start` command, so a stopped disk cannot be
rebooted through the public CLI yet. Keep it only if you need to inspect it or
expect a future start feature.

### Scenario: Permanently Remove a VM

```sh
sudo ./uvm.py destroy vm-<id>
```

Destroying a VM:

1. Terminates the VMM if it is alive.
2. Deletes the TAP device.
3. Deletes the VM runtime directory and private root disk.
4. Removes the VM record from `state.json`.
5. Releases its IP address for later allocation.

This is destructive. Back up guest data before running it.

## 9. Use the FastAPI Server

The API is an alternative management surface for the same local lifecycle
services. It does not make UVM multi-host or safe to expose directly to the
public internet.

### Scenario: Start a Local API Server

Install dependencies first:

```sh
cd uvm
python3 -m pip install .
```

Then start the server as root so lifecycle endpoints can manipulate KVM,
networking, and host files:

```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'

sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  ./uvm.py --serve --host 127.0.0.1 --port 8000
```

Open these local URLs after it starts:

```text
Health:  http://127.0.0.1:8000/health
OpenAPI: http://127.0.0.1:8000/docs
```

The API token is required even for a loopback server. All VM and installation
routes require it. FastAPI's informational `/docs`, `/redoc`, and
`/openapi.json` endpoints are also available without a token; they do not
perform host mutations.

### Scenario: Start a Remote TLS API Server

Non-loopback binds require a valid TLS certificate and key. The files must be
readable by the server process:

```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'

sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
  UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
  ./uvm.py --serve --host 0.0.0.0 --port 8443
```

For real remote use, use a certificate trusted by clients and restrict network
access with a firewall or reverse proxy. Do not use a self-signed development
certificate for an untrusted network.

### API Authentication

Set one reusable shell variable for API examples:

```sh
export UVM_API_TOKEN='replace-with-the-server-token'
export UVM_API_URL='http://127.0.0.1:8000'
```

Every management request needs:

```http
X-UVM-Token: <token>
```

For a TLS server, set `UVM_API_URL` to an `https://` URL and use your CA:

```sh
export UVM_API_URL='https://host.example:8443'
curl --cacert /path/to/ca.pem \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms"
```

## 10. API Endpoint Examples

### Health Check

No token is required:

```sh
curl "$UVM_API_URL/health"
```

Response:

```json
{
  "status": "ok"
}
```

### List VMs

```sh
curl \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms"
```

Example response:

```json
[
  {
    "id": "vm-0123456789abcdef",
    "cpu": 1.0,
    "ram_mib": 512,
    "guest_ip": "10.42.0.2",
    "gateway": "10.42.0.1",
    "mac": "02:fc:00:00:00:01",
    "username": "root",
    "status": "running",
    "observed_status": "running",
    "pid": 12345,
    "created_at": 1700000000,
    "updated_at": 1700000000,
    "last_error": null
  }
]
```

### Create a VM

Create a default-sized VM:

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"cpu": 1, "ram": "512", "username": "root", "password": "root"}' \
  "$UVM_API_URL/vms"
```

Create a VM with a chosen guest address:

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"cpu": 2, "ram": "2G", "guest_ip": "10.42.0.10", "username": "root", "password": "replace-me"}' \
  "$UVM_API_URL/vms"
```

The response is the VM record with its username, but never its password.
Omitting credentials uses `root` / `root`. Request fields are strict:
misspelled or extra fields return HTTP `422` instead of being silently ignored.

### Look Up One VM

By ID:

```sh
curl \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/vm-0123456789abcdef"
```

By guest IP:

```sh
curl \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/10.42.0.2"
```

### Stop a VM

```sh
curl -X POST \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/vm-0123456789abcdef/stop"
```

The response is the VM record with `status: "stopped"`.

### Destroy a VM

```sh
curl -X DELETE \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  "$UVM_API_URL/vms/vm-0123456789abcdef"
```

Response:

```json
{
  "id": "vm-0123456789abcdef",
  "status": "terminated"
}
```

### Install or Refresh Through the API

The server process must itself have checksum variables configured for strict
installation. Start the server with those environment variables before using
this endpoint.

For a deliberately unverified local-development server, the server process
must instead include `UVM_ALLOW_UNVERIFIED_DOWNLOADS=1`. Do not use that mode
on a host that runs untrusted workloads.

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"force": false}' \
  "$UVM_API_URL/install"
```

Refresh all downloaded artifacts:

```sh
curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-UVM-Token: $UVM_API_TOKEN" \
  -d '{"force": true}' \
  "$UVM_API_URL/install"
```

## 11. API Errors

| Status | Typical meaning |
|---|---|
| `401` | Missing or incorrect `X-UVM-Token`. |
| `403` | UVM process lacks root privileges for a host mutation. |
| `404` | Requested VM was not found. |
| `409` | A lifecycle operation is already in progress. |
| `422` | Invalid JSON body, extra request fields, or invalid UVM input. |
| `500` | Host command, Firecracker, state, or unexpected operational failure. |

UVM operation errors use this envelope:

```json
{
  "error": {
    "message": "human-readable explanation"
  }
}
```

FastAPI request-schema errors use FastAPI's normal `detail` response format.

## 12. Configuration Scenarios

### Scenario: Use a Different Storage Directory

```sh
sudo env \
  UVM_BASE='/srv/uvm' \
  UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 \
  ./uvm.py install
```

All UVM binaries, images, VM runtime directories, locks, and state use the
configured base path.

### Scenario: Use a Different Guest Network

```sh
sudo env \
  UVM_NETWORK='10.50.0.0/24' \
  UVM_GATEWAY='10.50.0.1' \
  UVM_BRIDGE='uvm50' \
  UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 \
  ./uvm.py install
```

Use the same environment values for later `create`, `list`, `stop`, `destroy`,
and server commands. Changing the network variables after VMs exist can make
their persisted state inconsistent with host networking.

### Environment Variables

| Variable | Use |
|---|---|
| `UVM_BASE` | Base data directory; default `/var/lib/uvm` |
| `UVM_NETWORK` | Guest subnet; default `10.42.0.0/24` |
| `UVM_GATEWAY` | Guest gateway / bridge IP; default `10.42.0.1` |
| `UVM_BRIDGE` | Linux bridge name; default `uvm0` |
| `UVM_KERNEL_URL` | Custom kernel download URL |
| `UVM_ROOTFS_URL` | Custom rootfs download URL |
| `UVM_FIRECRACKER_SHA256` | Strict Firecracker archive checksum |
| `UVM_KERNEL_SHA256` | Strict kernel checksum |
| `UVM_ROOTFS_SHA256` | Strict rootfs checksum |
| `UVM_FIRECRACKER_BINARY_SHA256` | Optional local binary checksum |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | Set to `1` only for local development |
| `UVM_API_TOKEN` | Required API token |
| `UVM_API_TLS_CERT` | TLS certificate for remote server binds |
| `UVM_API_TLS_KEY` | TLS key for remote server binds |

## 13. Files, Logs, and State

The default host layout is:

```text
/var/lib/uvm/
├── bin/                 # Firecracker and Jailer binaries
├── images/              # Shared kernel and rootfs template
├── vms/<vm-id>/         # Private rootfs, config, socket, and log
├── integrity.json       # Artifact checksums and verification status
└── state.json           # Mode-0600 VM inventory and plaintext guest credentials
```

Useful inspection commands:

```sh
sudo cat /var/lib/uvm/state.json
sudo ls -la /var/lib/uvm/vms
sudo cat /var/lib/uvm/vms/<vm-id>/firecracker.log
sudo cat /var/lib/uvm/vms/<vm-id>/config.json
```

Avoid editing `state.json`, VM config files, runtime directories, TAPs, or
iptables rules by hand while UVM is running. Use UVM commands or the API so
locks and cleanup rules remain correct.

`state.json` contains plaintext guest passwords. Keep it root-only, never copy
it into logs or bug reports, and use unique non-default passwords outside local
throwaway environments.

## 14. Networking Troubleshooting

Check the bridge and TAP state:

```sh
ip link show uvm0
ip addr show dev uvm0
ip route show default
```

Check NAT and forwarding rules:

```sh
sudo iptables -t nat -S POSTROUTING
sudo iptables -S FORWARD
```

Common issues:

| Symptom | First checks |
|---|---|
| Guest cannot reach the internet | Bridge is up, TAP is up, host has default route, NAT and FORWARD rules exist. |
| Requested guest IP is rejected | Confirm it is inside the configured subnet and not used, gateway, network, or broadcast. |
| Create says TAP already exists | Inspect the interface; do not delete it unless you know it is stale UVM state. |
| Existing bridge is rejected | `UVM_BRIDGE` points to an interface that is not a Linux bridge. |
| SSH cannot connect | Confirm guest booted, the requested user exists, the image runs OpenSSH, and the configured password is being used. |

## 15. Important Limitations

Keep these constraints in mind when deciding whether UVM fits a scenario:

- Firecracker is launched directly; the downloaded Jailer is not yet used.
- No cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU requests are advisory only.
- There is no public `start` or `restart` command after a VM is stopped.
- There is no DHCP, inbound port mapping, load balancing, multi-network
  isolation, or multi-host scheduling.
- The API uses one shared bearer token, not individual users, roles, or tenants.
- There is no guest readiness probe beyond Firecracker process state.
- UVM is not a hardened multi-tenant platform.

## 16. Safe Operating Checklist

Before relying on a VM for useful work:

1. Use strict checksums for installed artifacts.
2. Confirm `/dev/kvm` and nested virtualization are available.
3. Confirm UVM generated unique guest SSH host keys and set a non-default password.
4. Test guest egress and SSH on the actual host network.
5. Back up guest data before destroy operations.
6. Keep the API loopback-only unless you have a strong remote-access need.
7. For remote API access, use a long random token, valid TLS, and firewall
   restrictions.
8. Review `/var/lib/uvm/vms/<vm-id>/firecracker.log` after failed starts.

## 17. Get Help

```sh
./uvm.py --help
./uvm.py install --help
./uvm.py --serve --help
```

When reporting a problem, collect:

- The exact command or HTTP request.
- The full error message.
- Output from `sudo ./uvm.py list`.
- The VM Firecracker log, if a VM was created.
- The relevant `UVM_*` variables, with secrets such as `UVM_API_TOKEN` removed.

Never include `/var/lib/uvm/state.json` because it contains guest passwords.