nest/DEVELOPER.md

# UVM Developer Guide

This document is the developer handoff for the entire UVM codebase. Read it
before changing lifecycle behavior, host networking, Firecracker launch
semantics, persistent state, or the HTTP API.

`uvm` is a local, privileged Firecracker microVM manager. It is an
application with a CLI and optional FastAPI management server. It is not a
general-purpose Python library or a multi-host control plane.

## 1. Goals and Boundaries

UVM currently provides a small host-local workflow:

1. Install Firecracker, a guest kernel, and a rootfs template.
2. Allocate an ID, private IPv4 address, MAC address, TAP device, and VM
   runtime directory.
3. Copy the rootfs template to a private writable VM disk and provision the
   requested existing guest account password.
4. Configure and launch one Firecracker process.
5. List, stop, SSH to, and destroy the VM.
6. Optionally expose those operations through an authenticated FastAPI API.

The design deliberately does not provide these capabilities yet:

- Multi-host scheduling or remote state coordination.
- User tenancy, quotas, or a public-facing control plane.
- A `start` or `restart` command for stopped VMs.
- Guest snapshots, extra disks, port forwarding, or DHCP.
- Cgroup CPU enforcement for fractional CPU requests.
- Jailer-based Firecracker execution, namespaces, or seccomp hardening.
- Production-grade image building or guest SSH-key injection.

Treat UVM as a privileged local systems tool. It manipulates `/dev/kvm`, TAP
interfaces, bridge addresses, iptables rules, host filesystems, and processes.

## 2. First Run

All UVM-specific files live in this directory. Start here:

```sh
cd uvm
python3 -m pip install -e '.[test]'
PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest discover -s tests -v
./uvm.py --help
```

The `uvm` console script is registered by `pyproject.toml`. During source
development, `./uvm.py` and `python3 -m uvm` are equivalent entry points.

Requirements for real host operations:

- Python 3.12+
- Linux with KVM enabled and `/dev/kvm` accessible
- Root privileges for installation and lifecycle mutations
- `iproute2`, `iptables`, `e2fsprogs`, `util-linux`, `openssl`, and OpenSSH
  client tools
- An x86_64 host for the default kernel/rootfs URLs

Do not run real integration operations on a developer workstation unless the
bridge, subnet, iptables changes, and guest workload are acceptable there.

## 3. Project Layout

```text
uvm/
├── DEVELOPER.md              # This document
├── README.md                 # Operator-facing usage guide
├── pyproject.toml            # Package metadata and dependencies
├── uvm.py                    # Executable source-tree launcher
├── uvm/
│   ├── __main__.py           # `python -m uvm`
│   ├── cli.py                # argparse, CLI rendering, serve dispatch
│   ├── app.py                # Composition root
│   ├── config.py             # Settings and environment loading
│   ├── domain.py             # VmSpec, VmRecord, State
│   ├── errors.py             # Expected error hierarchy
│   ├── validation.py         # CPU and RAM parsing
│   ├── system.py             # CommandRunner, root/KVM/directory checks
│   ├── state.py              # JSON state, atomic writes, file locks
│   ├── integrity.py          # SHA-256 and integrity manifest logic
│   ├── install.py            # Apt, Firecracker, kernel, rootfs installation
│   ├── images.py             # Template lookup, private disks, and guest credentials
│   ├── network.py            # IP/MAC/TAP/bridge/NAT/forward rules
│   ├── lifecycle.py          # Create/list/stop/destroy orchestration
│   ├── api_models.py         # Pydantic HTTP request/response models
│   ├── server.py             # FastAPI factory and Uvicorn launcher
│   ├── routers/
│   │   ├── dependencies.py   # API application/token dependencies
│   │   ├── health.py         # Liveness endpoint
│   │   ├── installation.py   # Installation endpoint
│   │   └── vms.py            # VM lifecycle endpoints
│   └── firecracker/
│       ├── config.py         # Firecracker JSON and boot arguments
│       ├── api.py            # Unix-socket HTTP client
│       └── process.py        # Process launch, liveness, termination
└── tests/                    # Unit tests and conditional API tests
```

Keep feature code in the most specific existing module. Do not add a new
abstraction layer merely to wrap one function. The current module boundaries
are intentionally small and operationally focused.

## 4. Architecture Overview

```mermaid
flowchart LR
  CLI[CLI: uvm] --> APP[app.py composition root]
  HTTP[FastAPI server] --> APP
  APP --> INSTALL[Installer]
  APP --> LIFE[LifecycleService]
  APP --> STATE[StateStore]
  LIFE --> IMAGES[ImageStore]
  LIFE --> NET[NetworkManager]
  LIFE --> FC[Firecracker adapters]
  INSTALL --> RUNNER[CommandRunner]
  NET --> RUNNER
  FC --> HOST[Linux host primitives]
  RUNNER --> HOST
  STATE --> DISK[(state.json)]
  IMAGES --> DISKSTORE[/var/lib/uvm/images and vms]
  FC --> KVM[/dev/kvm]
```

The public entry points are thin adapters. The lifecycle layer is the only
place that coordinates multiple host resources for a VM. Lower-level modules
must not import `cli.py`, `server.py`, or `lifecycle.py`.

### Dependency Direction

```text
cli.py / server.py / routers/
              |
              v
            app.py
              |
      +-------+--------+
      |                |
  install.py      lifecycle.py
                     |
        +------------+------------+
        |            |            |
     state.py    network.py   images.py
                                  |
                         firecracker/*
        |            |            |
        +------ system.py --------+
                     |
                  Linux host
```

Leaf modules are `config.py`, `domain.py`, `errors.py`, and `validation.py`.
They must remain free of subprocess, network, CLI, and FastAPI side effects.

## 5. Entrypoints and Request Paths

### CLI Path

```mermaid
sequenceDiagram
  participant U as User
  participant C as cli.py
  participant A as app.py
  participant L as LifecycleService
  participant H as Linux host

  U->>C: uvm create --cpu 1 --ram 512
  C->>A: build_application()
  A->>L: composed service graph
  C->>L: create(VmSpec)
  L->>H: allocate files, TAP, process, Firecracker API
  L-->>C: VmRecord
  C-->>U: formatted VM details
```

`cli.py` owns parsing, terminal output, exit handling, SSH `execvp`, and the
`--serve` switch. It must not manipulate state files, interfaces, or
Firecracker sockets directly.

### HTTP Path

```mermaid
sequenceDiagram
  participant Client
  participant Uvicorn
  participant API as server.py / routers
  participant L as LifecycleService
  participant H as Linux host

  Client->>Uvicorn: HTTP(S) request + X-UVM-Token
  Uvicorn->>API: route dispatch
  API->>API: token dependency and Pydantic validation
  API->>L: lifecycle operation
  L->>H: host mutation
  L-->>API: VmRecord or UvmError
  API-->>Client: JSON response or JSON error
```

Use only `uvm --serve` or `./uvm.py --serve` to launch the API. The
`create_api()` factory requires an explicit host specifically to prevent it
from being used as an insecure generic Uvicorn factory.

## 6. Application Composition

[`app.py`](uvm/app.py) constructs one concrete object graph per CLI or server
process:

| Component | Constructor | Responsibility |
|---|---|---|
| Settings | `Settings.from_environment()` | Immutable runtime configuration |
| CommandRunner | `CommandRunner` | Logged/testable subprocess execution |
| StateStore | `StateStore` | Inventory, locks, JSON persistence |
| ImageStore | `ImageStore` | Template verification, VM disks, and offline credentials |
| NetworkManager | `NetworkManager` | Guest allocation and Linux networking |
| FirecrackerProcessManager | `FirecrackerProcessManager` | VMM process control |
| LifecycleService | `LifecycleService` | Ordered VM resource changes and rollback |
| Installer | `Installer` | Host packages and artifacts |

The FastAPI factory calls `build_application(emit=None)` so API-driven host
commands do not print shell-style command traces to the server output. The CLI
uses the default emitter and prints commands as `+ ...` for transparency.

## 7. Persistent Data and Models

### Host Data Layout

Default runtime storage is `/var/lib/uvm`:

```text
/var/lib/uvm/
├── bin/
│   ├── firecracker
│   └── jailer
├── images/
│   ├── vmlinux
│   └── ubuntu.ext4
├── vms/
│   └── vm-<uuid>/
│       ├── rootfs.ext4
│       ├── config.json
│       ├── firecracker.log
│       └── firecracker.sock
├── state.json
├── state.lock
├── operations.lock
└── integrity.json
```

Per-VM runtime directories are mode `0700`; copied VM disks are mode `0600`.
`state.json` is mode `0600` because VM records contain plaintext guest
passwords. Registry-backed commands therefore need access to the root-owned
state file.

### State Schema

[`domain.py`](uvm/domain.py) is the schema authority. The current state
version is `1`.

```json
{
  "schema_version": 1,
  "next_mac_index": 2,
  "vms": {
    "vm-<uuid>": {
      "id": "vm-<uuid>",
      "cpu": 1.0,
      "ram_mib": 512,
      "guest_ip": "10.42.0.2",
      "gateway": "10.42.0.1",
      "tap": "uvm-...",
      "mac": "02:fc:00:00:00:01",
      "socket": "/var/lib/uvm/vms/.../firecracker.sock",
      "config": "/var/lib/uvm/vms/.../config.json",
      "log": "/var/lib/uvm/vms/.../firecracker.log",
      "disk": "/var/lib/uvm/vms/.../rootfs.ext4",
      "username": "root",
      "password": "replace-me",
      "status": "running",
      "pid": 1234,
      "process_start_time": "...",
      "created_at": 0,
      "updated_at": 0,
      "last_error": null
    }
  }
}
```

`VmSpec` is the validated create request. `VmRecord` is the persisted local
record. Do not make API models or CLI namespaces the persistence format.

### State Guarantees

- `state.json` writes use a sibling temporary file, `fsync`, atomic replace,
  and parent-directory `fsync`.
- `state.json` is mode `0600`; it must be treated as a credential registry and
  excluded from logs and bug reports.
- `state.lock` serializes short JSON read-modify-write transactions.
- `operations.lock` serializes long lifecycle and installation operations.
- State reads are lock-free because atomic replacement gives readers either a
  complete old file or a complete new file.
- Legacy state without a schema version is accepted as version `0`; missing
  `disk` and `process_start_time` fields have safe defaults.
- Records created before credential provisioning default to username `root`
  and password `null`; UVM does not apply a password retroactively.
- `next_mac_index` is reconstructed from older locally administered MACs when
  absent.

When adding a persisted field, update `VmRecord.to_dict`, `VmRecord.from_dict`,
the state schema tests, and this document. Never silently discard existing
state fields without a migration plan.

## 8. VM Lifecycle

### Stored and Observed Status

The persisted status is a string, not an enum, to keep legacy JSON readable.

| Status | Meaning |
|---|---|
| `starting` | Identity reserved; host resources are being created |
| `running` | Firecracker configuration and `InstanceStart` succeeded |
| `stopping` | Stop mutation is in progress |
| `stopped` | Process/TAP removed; private disk remains |
| `terminating` | Destruction is in progress |
| `failed` | A lifecycle step or cleanup failed; `last_error` explains why |
| `dead` | Observed-only list status when a supposedly active process is gone |

`dead` is not persisted. `destroy` removes the record rather than retaining a tombstone.

### Create Order

[`LifecycleService.create`](uvm/lifecycle.py) follows this sequence:

```mermaid
flowchart TD
  A[Validate root, KVM, binary, integrity] --> B[Acquire operations.lock]
  B --> C[Reserve ID, IP, MAC, and starting state]
  C --> D[Create 0700 runtime directory]
  D --> E[Copy template to private rootfs.ext4]
  E --> F[Provision guest password and SSH policy]
  F --> G[Ensure bridge, NAT, and forwarding rules]
  G --> H[Create TAP]
  H --> I[Write Firecracker config atomically]
  I --> J[Spawn Firecracker and wait for API socket]
  J --> K[Persist PID and process start token]
  K --> L[Configure Firecracker through Unix HTTP]
  L --> M[Persist running state]
```

The record is persisted as `starting` before host side effects. The PID and
Linux process-start token are persisted before Firecracker API configuration.
This makes cleanup and later diagnosis possible if API configuration fails.

### Rollback Rules

Any `Exception`, `KeyboardInterrupt`, or `SystemExit` during create triggers
compensation:

1. Terminate a started Firecracker process when its identity is known.
2. Remove the TAP when it was created.
3. Remove the VM runtime directory and private disk.
4. Remove the reserved state record if cleanup succeeds.
5. Preserve a `failed` record with `last_error` if cleanup itself fails.

Do not reorder these operations without considering leaked TAPs, orphaned VMM
processes, and allocation reuse.

### Stop and Destroy

```text
stop:
  running/failed/stopped -> stopping -> terminate process -> delete TAP -> stopped
  private disk and config remain

destroy:
  any non-transition state -> terminating -> terminate process -> delete TAP
  -> remove VM runtime directory -> remove state record
```

`stop` is intentionally disk-preserving. There is no public `start` command
yet, so do not add start behavior by modifying stop semantics.

## 9. Firecracker Integration

The `firecracker/` package has three separate responsibilities.

| File | Responsibility |
|---|---|
| `config.py` | Build boot args and Firecracker JSON; atomically write config |
| `api.py` | Typed HTTP-over-Unix-socket requests |
| `process.py` | Spawn, socket readiness, liveness, signal safety, cleanup |

### Configuration Order

`FirecrackerClient.configure_and_start()` must keep this ordering:

1. `PUT /machine-config`
2. `PUT /boot-source`
3. `PUT /drives/rootfs`
4. `PUT /network-interfaces/eth0`
5. `PUT /actions` with `InstanceStart`

The guest receives a static IP via the kernel `ip=` boot argument. The netmask
comes from `Settings.network`, not a hard-coded `/24` value.

### Process Safety

Firecracker starts detached in a new session, with stdout/stderr redirected to
the per-VM log. The manager records `/proc/<pid>/stat` start time and only
signals a process when its persisted token still matches. This prevents a stale
state record from killing an unrelated process after PID reuse.

Legacy records with a PID but no start token are only cleaned up automatically
when that PID is already gone. If it remains alive, UVM fails closed and asks
the operator to inspect it.

## 10. Guest Images and Integrity

### Template Contract

`images/ubuntu.ext4` is a template only. Each create operation copies it into
the VM runtime directory before UVM modifies the requested existing account's
shadow entry and OpenSSH password policy with `debugfs`. Firecracker only sees
the resulting private copy. Never change that to a shared writable rootfs.

Guest images must provide `/etc/passwd`, `/etc/shadow`, OpenSSH's
`/etc/ssh/sshd_config`, and the requested account. UVM does not create missing
users. It replaces conventional RSA, ECDSA, and Ed25519 host keys in each
private disk and removes the known public login key from Firecracker's bionic
image. A VM must never rely on reusable SSH private keys from a shared template.

### Integrity Model

[`integrity.py`](uvm/integrity.py) has two concepts:

- A configured trusted SHA-256 from the environment.
- A local `integrity.json` manifest recording artifact hashes and whether the
  installation was verified.

Strict installation requires all of these values:

```text
UVM_FIRECRACKER_SHA256
UVM_KERNEL_SHA256
UVM_ROOTFS_SHA256
```

`UVM_ALLOW_UNVERIFIED_DOWNLOADS=1` is a local-development escape hatch. It
writes a manifest marked `verified: false`; a later strict run will not treat
that manifest as publisher trust. Use `install --force` with trusted digests to
refresh an existing artifact without manually deleting managed files.

The optional `UVM_FIRECRACKER_BINARY_SHA256` validates a local Firecracker
binary when no verified manifest is available.

## 11. Networking

### Topology

```text
guest eth0
    |
virtio-net / Firecracker
    |
per-VM TAP: uvm-<suffix>
    |
Linux bridge: uvm0 (10.42.0.1/24 by default)
    |
iptables forwarding + MASQUERADE
    |
host default uplink
    |
internet
```

[`network.py`](uvm/network.py) owns all networking details:

- IP allocation from `Settings.network`, excluding network, broadcast, and
  gateway addresses.
- Locally administered MAC generation: `02:fc:00:xx:xx:xx`.
- Interface names limited to Linux `IFNAMSIZ` (15 characters).
- Bridge creation or validation that an existing interface is actually a
  Linux bridge.
- Gateway address setup and IPv4 forwarding.
- One egress NAT rule and forward/return rules for the default uplink.
- TAP creation, cleanup, and error reporting.

Network actions are idempotent where practical. Do not adopt an existing TAP:
it may belong to another VM. A name collision is an error.

## 12. Installer and Host Commands

`Installer` is the only owner of apt installation, Firecracker artifact
handling, downloads, and integrity-manifest publication.

`system.py` centralizes host command execution:

- `CommandRunner.run()` converts subprocess failures into `CommandError`.
- `require_root()` is called by host-mutating operations.
- `check_kvm()` validates the KVM device.
- `ensure_data_directories()` creates the UVM storage layout.

Use `CommandRunner` rather than direct `subprocess.run` in installation or
network code. This keeps command construction observable in the CLI and
mockable in tests.

Downloads use temporary sibling files, optional checksum verification, and
atomic replacement. Archive extraction rejects path traversal before using
Python's safe tar extraction filter.

## 13. HTTP API

### Server Startup

```sh
cd uvm
python3 -m pip install .

export UVM_API_TOKEN='long-random-visible-ascii-token'
sudo env UVM_API_TOKEN="$UVM_API_TOKEN" \
  ./uvm.py --serve --host 127.0.0.1 --port 8000
```

For non-loopback binds, both TLS variables are required and loaded into an
`ssl.SSLContext` before Uvicorn starts:

```sh
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'
sudo env \
  UVM_API_TOKEN="$UVM_API_TOKEN" \
  UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
  UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
  ./uvm.py --serve --host 0.0.0.0 --port 8443
```

The token must contain visible ASCII characters without whitespace. It is
never accepted as a UVM command-line argument, so it does not appear in UVM
process arguments. If you type an `export` command interactively, your shell
may record it in history; use a protected environment file, service manager,
or secret manager when that matters.

### API Contract

| Method | Route | Router | Notes |
|---|---|---|---|
| `GET` | `/health` | `health.py` | Unauthenticated liveness response |
| `POST` | `/install` | `installation.py` | `{ "force": false }` body |
| `GET` | `/vms` | `vms.py` | List persisted VM records and observed status |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, optional `guest_ip`, `username`, and `password` |
| `GET` | `/vms/{id-or-ip}` | `vms.py` | Lookup by ID or guest IP |
| `POST` | `/vms/{id}/stop` | `vms.py` | Stop while retaining private disk |
| `DELETE` | `/vms/{id}` | `vms.py` | Remove runtime files and state |

All management routes use `get_authorized_application()` and require:

```http
X-UVM-Token: <UVM_API_TOKEN>
```

Request schemas reject unexpected fields. `UvmError` instances are rendered as
JSON errors by `server.py`; Pydantic request validation uses FastAPI's normal
422 response format. VM responses include the username but never the password.

The API intentionally does not expose interactive SSH, raw Firecracker calls,
arbitrary command execution, or host path access.

## 14. Configuration Reference

| Variable | Default | Used by |
|---|---|---|
| `UVM_BASE` | `/var/lib/uvm` | Storage layout |
| `UVM_NETWORK` | `10.42.0.0/24` | IP allocation and bridge subnet |
| `UVM_GATEWAY` | `10.42.0.1` | Bridge and guest boot args |
| `UVM_BRIDGE` | `uvm0` | Linux bridge name |
| `UVM_KERNEL_URL` | Firecracker quickstart URL | Guest kernel download |
| `UVM_ROOTFS_URL` | Firecracker quickstart URL | Guest rootfs download |
| `UVM_FIRECRACKER_SHA256` | unset | Trusted release archive checksum |
| `UVM_KERNEL_SHA256` | unset | Trusted kernel checksum |
| `UVM_ROOTFS_SHA256` | unset | Trusted rootfs checksum |
| `UVM_FIRECRACKER_BINARY_SHA256` | unset | Optional direct binary checksum |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | `false` | Development-only integrity bypass |
| `UVM_API_TOKEN` | unset | Required API bearer token |
| `UVM_API_TLS_CERT` | unset | TLS certificate for remote API bind |
| `UVM_API_TLS_KEY` | unset | TLS key for remote API bind |

`Settings` validates the gateway, bridge-name length, API token format, and
environment boolean values at process startup. Add configuration fields there,
not as module-level global constants in feature code.

## 15. Error Handling Rules

Expected failures inherit from `UvmError`:

| Error | Use |
|---|---|
| `ConfigurationError` | Invalid local settings or environment |
| `ValidationError` | Invalid CLI/API resource input |
| `StateError` | Unsafe or malformed persisted state |
| `CommandError` | Required host command failure |
| `FirecrackerError` | VMM process or Unix API failure |
| `TapCreationError` | TAP setup failed after interface creation |

CLI code catches `UvmError` and prints `uvm: error: ...` without a traceback.
The FastAPI server maps the same class to an error envelope. New expected
operational failures should use this hierarchy rather than leaking raw
`OSError`, `CalledProcessError`, or implementation-specific exceptions.

## 16. Concurrency and Ownership Invariants

These invariants are critical:

1. Only `LifecycleService` may coordinate VM state, disks, TAPs, and
   Firecracker as one operation.
2. Only `StateStore` may read/write `state.json`.
3. All long host mutations acquire `operations.lock`.
4. All short inventory mutations acquire `state.lock`.
5. The installer shares `operations.lock` with lifecycle actions, preventing a
   Firecracker binary replacement between verification and launch.
6. A VM rootfs is private before it becomes writable.
7. A PID is not signaled unless its persisted start token still matches
   `/proc/<pid>/stat`.
8. A new VM never adopts a pre-existing TAP.
9. A server must have an API token; a remote server must also have usable TLS.
10. Routers call application services; they do not run shell commands or touch
    state files directly.
11. Plaintext guest passwords remain in the mode-`0600` registry and never
    enter Firecracker config, helper command arguments/traces, or API responses.

When a change would violate one of these rules, redesign the change rather
than adding a special-case bypass.

## 17. Testing Strategy

### Standard Unit Suite

```sh
cd uvm
python3 -m pip install -e '.[test]'
PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest discover -s tests -v
```

The suite covers:

- CPU/RAM parsing and validation.
- State serialization, legacy migration, locking, and permissions.
- Checksum verification and integrity-manifest provenance.
- Private rootfs behavior.
- Guest credential persistence, validation, and offline rootfs provisioning.
- IP, MAC, bridge, TAP, NAT, and forwarding command construction.
- Firecracker configuration and API request ordering.
- PID safety and startup interrupt cleanup.
- Lifecycle success, rollback, stop, and destroy behavior.
- CLI SSH safety and `--serve` parsing.
- Server token/TLS validation and conditional API route tests.

FastAPI route tests are conditionally enabled when both `fastapi` and `httpx`
are installed. They should be enabled in CI using the `test` extra.

### Real Integration Tests

No current test automatically boots a real microVM. Before a release, run an
isolated host test that covers:

1. Strict checked installation.
2. `create` and successful guest boot.
3. Guest IP reachability and egress.
4. SSH with unique guest host keys.
5. Stop and retained disk inspection.
6. Destroy and IP/TAP release.
7. API launch with token and TLS on a non-loopback test address.
8. Firecracker crash and interrupted create recovery.

## 18. Common Development Tasks

### Add a CLI Command

1. Put business logic in an existing service or a cohesive new internal
   module, not `cli.py`.
2. Add parser arguments in `build_parser()`.
3. Add display/dispatch behavior in `_run_command()`.
4. Use `UvmError` subclasses for expected failures.
5. Add unit tests for parsing and service behavior.
6. Update `README.md` and this document if the architecture changes.

### Add an HTTP Endpoint

1. Add or extend a Pydantic request/response model in `api_models.py`.
2. Put the endpoint in the appropriate `routers/` module.
3. Use `get_authorized_application()` for every privileged endpoint.
4. Keep `/health` as the only intentionally unauthenticated management route.
   FastAPI's default documentation endpoints are informational and public.
5. Add the router to `create_api()`.
6. Add `httpx`/FastAPI tests and update the API table in both docs.

### Add a Host Primitive

1. Put subprocess construction in `system.py`, `network.py`, `install.py`, or
   a specific Firecracker adapter.
2. Use `CommandRunner` when invoking external commands.
3. Make repeated application safe where possible.
4. Define compensation behavior before calling it from lifecycle code.
5. Add fake-runner tests; do not require root in ordinary unit tests.

### Change State

1. Increment `STATE_SCHEMA_VERSION` only when the serialized interpretation
   changes incompatibly.
2. Keep older state readable or provide an explicit migration path.
3. Update `VmRecord`, `State`, state tests, and runtime cleanup behavior.
4. Consider lock ownership and crash/retry behavior before changing fields.

## 19. Debugging Guide

| Symptom | First places to inspect |
|---|---|
| `guest assets missing` | `/var/lib/uvm/images`, integrity variables, `integrity.json` |
| Firecracker exits early | VM `firecracker.log`, VM config, `/dev/kvm` access |
| VM is `dead` in `list` | PID/start token in state, `/proc/<pid>`, VM log |
| Cannot stop legacy VM | Missing process start token; inspect PID manually before cleanup |
| Guest has no egress | `ip route show default`, bridge address, TAP state, iptables rules |
| API server refuses to start | `UVM_API_TOKEN`, requested host, TLS cert/key files |
| API returns 401 | `X-UVM-Token` header mismatch |
| API returns 422 | Request schema or UVM input validation failure |

Useful host commands during isolated debugging:

```sh
ip link show uvm0
ip addr show dev uvm0
ip route show default
iptables -t nat -S POSTROUTING
iptables -S FORWARD
ls -la /var/lib/uvm/vms
sudo cat /var/lib/uvm/state.json  # Contains plaintext guest passwords; do not share.
```

Do not use these commands to mutate UVM resources manually while a lifecycle
operation is active. Prefer fixing the service behavior and retrying through
the CLI or API.

## 20. Current Production Gaps

The code is intentionally honest about current limits:

- Firecracker is launched directly, even though the Jailer binary is
  downloaded.
- No cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU is represented by rounding vCPUs up and printing an advisory.
- Guest health is inferred from VMM process state, not application readiness.
- There is no DHCP, port mapping, ingress firewall policy, or network tenancy.
- `iptables` rules are incrementally managed rather than rendered from a full
  declarative firewall policy.
- The JSON inventory is suitable for one host, not distributed coordination.
- Guest passwords are stored in plaintext in the root-only local registry.
- API authentication is a single bearer token, not a user/tenant model.

Do not market UVM as a hardened multi-tenant platform until these boundaries
are deliberately addressed with architecture, tests, and operational review.

## 21. Contributor Checklist

Before submitting a change:

1. Run the complete unit suite.
2. Run `git diff --check`.
3. Confirm a new feature stays inside the `uvm/` project directory.
4. Verify command/API errors use `UvmError` rather than raw tracebacks.
5. Verify host mutations have a lock owner and cleanup path.
6. Verify persistent state remains readable across upgrades.
7. Update `README.md` for operator-facing behavior.
8. Update this document for architectural, trust-boundary, or lifecycle
   changes.