kstyagi/nest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor VM lifecycle management and enhance validation

Browse Source 
- Introduced a new method `installed_kernel` in `ImageStore` to handle kernel verification and checksum validation.
- Updated `Installer` to include additional packages: `nftables` and `conntrack`.
- Refined `LifecycleService` to improve error handling and state management during VM creation, starting, stopping, and deletion.
- Enhanced port management with new methods for setting and syncing ports, including validation for port rules.
- Improved error handling in `StateStore` to raise specific `NotFoundError` and `ConflictError` exceptions.
- Updated network management to ensure proper TAP device handling and firewall integration.
- Enhanced API responses in `vms.py` to include policy status for better client-side handling.
- Added validation for VM names and ports in `validation.py` to enforce stricter rules and prevent conflicts.
This commit is contained in:
kstyagi
2026-09-09 17:27:50 +05:30
parent 4ce14c3378
commit d8a25209b9
31 changed files with 9576 additions and 1331 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEVELOPER.md
+389 -291
View File
@@ -10,60 +10,53 @@ general-purpose Python library or a multi-host control plane.
## 1. Goals and Boundaries
UVM currently provides a small host-local workflow:
UVM installs host/guest assets, provisions private disks, and manages named
VMs and private IPv4 TCP allowlists on one host. The CLI offers retained-disk
start/stop/restart, deletion, fixed-target bulk operations, inspection, SSH,
logs, doctor, and explicit port edits/sync. The optional API exposes only the
existing health/install/create/list/inspect/stop/delete operations.
1. Install Firecracker, a guest kernel, and a rootfs template.
2. Allocate an ID, private IPv4 address, MAC address, TAP device, and VM
runtime directory.
3. Copy the rootfs template to a private writable VM disk and provision the
requested existing guest account password.
4. Configure and launch one Firecracker process.
5. List, stop, SSH to, and destroy the VM.
6. Optionally expose those operations through an authenticated FastAPI API.
The design deliberately does not provide these capabilities yet:
- Multi-host scheduling or remote state coordination.
- User tenancy, quotas, or a public-facing control plane.
- A `start` or `restart` command for stopped VMs.
- Guest snapshots, extra disks, port forwarding, or DHCP.
- Cgroup CPU enforcement for fractional CPU requests.
- Jailer-based Firecracker execution, namespaces, or seccomp hardening.
- Production-grade image building or guest SSH-key injection.
There is no autostart or background reconciler, multi-host control plane,
tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port
filtering is not public-port publishing or a hardened multi-tenant boundary;
see Current Production Gaps for the remaining scope limits.
Treat UVM as a privileged local systems tool. It manipulates `/dev/kvm`, TAP
interfaces, bridge addresses, iptables rules, host filesystems, and processes.
interfaces, bridge addresses, nftables/conntrack and existing iptables egress
rules, host filesystems, and processes. For the complete operator command and
flag reference, see [USAGE.md](USAGE.md).
## 2. First Run
All UVM-specific files live in this directory. Start here:
```sh
cd uvm
python3 -m pip install -e '.[test]'
PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest discover -s tests -v
python3 -B -m unittest discover -s tests -v
./uvm.py --help
```
The `uvm` console script is registered by `pyproject.toml`. During source
development, `./uvm.py` and `python3 -m uvm` are equivalent entry points.
development, `./uvm.py` and `python3 -B -m uvm` are equivalent entry points.
Requirements for real host operations:
- Python 3.12+
- Linux with KVM enabled and `/dev/kvm` accessible
- Linux >= 5.3 with native bridge conntrack and safe pidfd support
- Root privileges for installation and lifecycle mutations
- `iproute2`, `iptables`, `e2fsprogs`, `util-linux`, `openssl`, and OpenSSH
client tools
- `iproute2`, `iptables`, `nftables`, `conntrack`, `e2fsprogs`, `util-linux`,
`openssl`, and OpenSSH client tools
- An x86_64 host for the default kernel/rootfs URLs
Do not run real integration operations on a developer workstation unless the
bridge, subnet, iptables changes, and guest workload are acceptable there.
bridge, subnet, firewall changes, and guest workload are acceptable there.
Packet integration tests must remain inside disposable isolated namespaces.
## 3. Project Layout
```text
uvm/
./
├── DEVELOPER.md # This document
├── README.md # Operator-facing usage guide
├── pyproject.toml # Package metadata and dependencies
@@ -73,16 +66,17 @@ uvm/
│ ├── cli.py # argparse, CLI rendering, serve dispatch
│ ├── app.py # Composition root
│ ├── config.py # Settings and environment loading
│ ├── domain.py # VmSpec, VmRecord, State
│ ├── domain.py # PortRule, VmSpec, VmRecord, State
│ ├── errors.py # Expected error hierarchy
│ ├── validation.py # CPU and RAM parsing
│ ├── validation.py # Resources, names, credentials, ports, timeouts
│ ├── system.py # CommandRunner, root/KVM/directory checks
│ ├── state.py # JSON state, atomic writes, file locks
│ ├── integrity.py # SHA-256 and integrity manifest logic
│ ├── install.py # Apt, Firecracker, kernel, rootfs installation
│ ├── images.py # Template lookup, private disks, and guest credentials
│ ├── network.py # IP/MAC/TAP/bridge/NAT/forward rules
│ ├── lifecycle.py # Create/list/stop/destroy orchestration
│ ├── firewall.py # Owned nft bridge tables and offline conntrack cleanup
│ ├── lifecycle.py # Lifecycle, bulk operations, recovery, port intent
│ ├── api_models.py # Pydantic HTTP request/response models
│ ├── server.py # FastAPI factory and Uvicorn launcher
│ ├── routers/
@@ -94,7 +88,7 @@ uvm/
│ ├── config.py # Firecracker JSON and boot arguments
│ ├── api.py # Unix-socket HTTP client
│ └── process.py # Process launch, liveness, termination
└── tests/ # Unit tests and conditional API tests
└── tests/ # Unit/API tests; integration/ has opt-in namespace packets
```
Keep feature code in the most specific existing module. Do not add a new
@@ -112,9 +106,11 @@ flowchart LR
APP --> STATE[StateStore]
LIFE --> IMAGES[ImageStore]
LIFE --> NET[NetworkManager]
NET --> FW[FirewallManager]
LIFE --> FC[Firecracker adapters]
INSTALL --> RUNNER[CommandRunner]
NET --> RUNNER
FW --> RUNNER
FC --> HOST[Linux host primitives]
RUNNER --> HOST
STATE --> DISK[(state.json)]
@@ -126,29 +122,6 @@ The public entry points are thin adapters. The lifecycle layer is the only
place that coordinates multiple host resources for a VM. Lower-level modules
must not import `cli.py`, `server.py`, or `lifecycle.py`.
### Dependency Direction
```text
cli.py / server.py / routers/
|
v
app.py
|
+-------+--------+
| |
install.py lifecycle.py
|
+------------+------------+
| | |
state.py network.py images.py
|
firecracker/*
| | |
+------ system.py --------+
|
Linux host
```
Leaf modules are `config.py`, `domain.py`, `errors.py`, and `validation.py`.
They must remain free of subprocess, network, CLI, and FastAPI side effects.
@@ -156,45 +129,37 @@ They must remain free of subprocess, network, CLI, and FastAPI side effects.
### CLI Path
```mermaid
sequenceDiagram
participant U as User
participant C as cli.py
participant A as app.py
participant L as LifecycleService
participant H as Linux host
`cli.py` builds the application, validates arguments, and calls lifecycle
services. It owns confirmation, terminal/JSON output, exit handling, SSH
`execvp`, local log streaming from a validated path, and `--serve` dispatch.
It must not coordinate state files, interfaces, or Firecracker sockets itself.
U->>C: uvm create --cpu 1 --ram 512
C->>A: build_application()
A->>L: composed service graph
C->>L: create(VmSpec)
L->>H: allocate files, TAP, process, Firecracker API
L-->>C: VmRecord
C-->>U: formatted VM details
```
The local surface is install/create/list/inspect/start/stop/restart/delete
(`destroy` alias)/ssh/logs/doctor and ports list/open/close/set/sync. Identifiers
resolve by exact ID, name, or guest IP, never an ID prefix. Generated-ID-shaped
inputs resolve only by ID; names cannot impersonate generated IDs. Ambiguous
non-ID aliases raise `ConflictError`.
`cli.py` owns parsing, terminal output, exit handling, SSH `execvp`, and the
`--serve` switch. It must not manipulate state files, interfaces, or
Firecracker sockets directly.
Create alone adds `22:TCP` unless `--no-default-ports` is supplied. Port syntax
is `PORT[:TCP|HTTP|HTTPS]`, comma-separated or repeated via `--ports`; all labels
use TCP. `--password-stdin` reads one line instead of using a password argument.
`_vm_output()` is an explicit output whitelist, not `VmRecord.to_dict()`.
JSON stays on stdout, while warnings, prompts, and host traces use stderr.
Bulk lifecycle actions and ports sync resolve/deduplicate IDs before prompting.
Delete and mutating `--all` selections require a TTY confirmation or `--yes`/`-y`.
Port set/close-all and removal of TCP/22 also prompt, except empty managed
no-ops. `--dry-run` previews without lifecycle/policy mutations or confirmation;
it is not a full preflight. Stop/restart/delete accept `--timeout`; start does
not. Logs follow does not hold the operation lock. Doctor runs readiness
checks, not per-VM repair or guest application probes.
### HTTP Path
```mermaid
sequenceDiagram
participant Client
participant Uvicorn
participant API as server.py / routers
participant L as LifecycleService
participant H as Linux host
Client->>Uvicorn: HTTP(S) request + X-UVM-Token
Uvicorn->>API: route dispatch
API->>API: token dependency and Pydantic validation
API->>L: lifecycle operation
L->>H: host mutation
L-->>API: VmRecord or UvmError
API-->>Client: JSON response or JSON error
```
Routers validate the token and Pydantic request, invoke application services,
and render whitelisted response models. The CLI expansion does not imply HTTP
endpoint parity: start/restart, port edits/sync, bulk operations, logs, doctor,
and interactive SSH have no HTTP endpoints.
Use only `uvm --serve` or `./uvm.py --serve` to launch the API. The
`create_api()` factory requires an explicit host specifically to prevent it
@@ -212,13 +177,15 @@ process:
| StateStore | `StateStore` | Inventory, locks, JSON persistence |
| ImageStore | `ImageStore` | Template verification, VM disks, and offline credentials |
| NetworkManager | `NetworkManager` | Guest allocation and Linux networking |
| FirewallManager | Constructed by `NetworkManager` | Owned native bridge policy and scoped offline conntrack cleanup |
| FirecrackerProcessManager | `FirecrackerProcessManager` | VMM process control |
| LifecycleService | `LifecycleService` | Ordered VM resource changes and rollback |
| Installer | `Installer` | Host packages and artifacts |
The FastAPI factory calls `build_application(emit=None)` so API-driven host
commands do not print shell-style command traces to the server output. The CLI
uses the default emitter and prints commands as `+ ...` for transparency.
Composition can disable command tracing with `emit=None`; `create_api()` does
so when it constructs its own application. The CLI supplies an emitter that
prints `+ ...` traces to stderr, including when passing its application to
the server launcher.
## 7. Persistent Data and Models
@@ -253,42 +220,38 @@ state file.
### State Schema
[`domain.py`](uvm/domain.py) is the schema authority. The current state
version is `1`.
[`domain.py`](uvm/domain.py) is the schema authority. The default registry is
`/var/lib/uvm/state.json`; its current schema version is **2**. An empty registry:
```json
{
"schema_version": 1,
"next_mac_index": 2,
"vms": {
"vm-<uuid>": {
"id": "vm-<uuid>",
"cpu": 1.0,
"ram_mib": 512,
"guest_ip": "10.42.0.2",
"gateway": "10.42.0.1",
"tap": "uvm-...",
"mac": "02:fc:00:00:00:01",
"socket": "/var/lib/uvm/vms/.../firecracker.sock",
"config": "/var/lib/uvm/vms/.../config.json",
"log": "/var/lib/uvm/vms/.../firecracker.log",
"disk": "/var/lib/uvm/vms/.../rootfs.ext4",
"username": "root",
"password": "replace-me",
"status": "running",
"pid": 1234,
"process_start_time": "...",
"created_at": 0,
"updated_at": 0,
"last_error": null
}
}
"schema_version": 2,
"next_mac_index": 1,
"vms": {}
}
```
`VmSpec` is the validated create request. `VmRecord` is the persisted local
record. Do not make API models or CLI namespaces the persistence format.
| Record Fields | Purpose |
|---|---|
| `id`, `name`, `cpu`, `ram_mib` | Stable identity, optional unique name, resource request |
| `guest_ip`, `gateway`, `network`, `bridge`, `tap`, `mac` | Retained network identity/configuration |
| `socket`, `config`, `log`, `disk` | Canonical per-VM runtime paths |
| `username`, `password` | Guest credentials; password stays out of public output |
| `ports` | Null for legacy/unmanaged, or a list of `{ "port": 80, "protocol": "HTTP" }` objects |
| `disk_ready`, `runtime_owned`, `tap_owned` | Completed provisioning and resource ownership evidence |
| `status`, `operation`, `spawn_pending`, `last_error` | Durable state, pending intent, ambiguous-spawn guard, failure detail |
| `pid`, `process_start_time`, `boot_id` | VMM identity, including host reboot detection |
| `created_at`, `updated_at` | Registry timestamps |
`PortRule` accepts TCP/HTTP/HTTPS labels and integer ports 1-65535. A port
number is unique regardless of label. `VmSpec.ports` defaults to TCP/22,
whereas `VmRecord.ports=None` deliberately preserves the legacy distinction.
An explicit empty tuple/list is a managed empty TCP allowlist, not unmanaged
mode. There is no persisted boolean asserting that a policy is enforced.
### State Guarantees
- `state.json` writes use a sibling temporary file, `fsync`, atomic replace,
@@ -296,11 +259,18 @@ record. Do not make API models or CLI namespaces the persistence format.
- `state.json` is mode `0600`; it must be treated as a credential registry and
excluded from logs and bug reports.
- `state.lock` serializes short JSON read-modify-write transactions.
- `operations.lock` serializes long lifecycle and installation operations.
- `operations.lock` serializes long lifecycle, bulk, policy, and installation operations.
- State reads are lock-free because atomic replacement gives readers either a
complete old file or a complete new file.
- Legacy state without a schema version is accepted as version `0`; missing
`disk` and `process_start_time` fields have safe defaults.
- Legacy versions 0 (including unversioned state) and 1 are loaded into schema
2 in memory and written as schema 2 on the next transaction. Newer schemas
are rejected. Reads do not reconcile VMs or adopt process identities.
- Missing `name`, `ports`, and `boot_id` remain null. No policy or default SSH
rule is silently imposed on a legacy VM. Use stop, explicit `ports set`,
then start; first policy adoption requires a stopped, proven-gone VM.
- Missing `disk` and process start time remain absent. Legacy `disk_ready` is
inferred only for running/stopped records with a recorded private disk;
start still validates the path, file, and provisioning state.
- Records created before credential provisioning default to username `root`
and password `null`; UVM does not apply a password retroactively.
- `next_mac_index` is reconstructed from older locally administered MACs when
@@ -324,62 +294,108 @@ The persisted status is a string, not an enum, to keep legacy JSON readable.
| `stopped` | Process/TAP removed; private disk remains |
| `terminating` | Destruction is in progress |
| `failed` | A lifecycle step or cleanup failed; `last_error` explains why |
| `dead` | Observed-only list status when a supposedly active process is gone |
| `dead` | Observed-only status when a stored running process is proven gone |
| `unknown` | Observed process identity cannot be established safely |
`dead` is not persisted. `destroy` removes the record rather than retaining a tombstone.
Transition statuses can survive interruption. `dead`/observed `unknown` do
not rewrite inventory. Delete removes the record rather than keeping a
tombstone. Pending deletion remains authoritative even if a subsequent stop
finishes runtime cleanup; start and policy edits must not resurrect it.
`ListedVm` separates the record, observed process status, and observed policy
status. Policy can be `enforced`, `missing`, `drift`, `quarantined`, `unknown`,
legacy `unmanaged`, or `inactive` for a cleanly stopped managed VM. Readback is
a ruleset/attachment snapshot, not application health or a reachability test.
### Create Order
[`LifecycleService.create`](uvm/lifecycle.py) follows this sequence:
```mermaid
flowchart TD
A[Validate root, KVM, binary, integrity] --> B[Acquire operations.lock]
B --> C[Reserve ID, IP, MAC, and starting state]
C --> D[Create 0700 runtime directory]
D --> E[Copy template to private rootfs.ext4]
E --> F[Provision guest password and SSH policy]
F --> G[Ensure bridge, NAT, and forwarding rules]
G --> H[Create TAP]
H --> I[Write Firecracker config atomically]
I --> J[Spawn Firecracker and wait for API socket]
J --> K[Persist PID and process start token]
K --> L[Configure Firecracker through Unix HTTP]
L --> M[Persist running state]
```
1. Validate the request/root access and acquire `operations.lock`.
2. Check KVM, verified Firecracker, firewall capability, and installed templates.
3. Reserve ID/name/IP/MAC/ports with `status=starting`, `operation=preparing`.
4. Create the owned mode-0700 runtime directory, copy a mode-0600 private disk,
provision credentials/host keys, and persist `disk_ready=True`.
5. Ensure bridge/egress networking and prepare an owned, attached TAP **down**.
6. Apply/read back the nftables bridge policy, then activate the TAP.
7. Write Firecracker config; persist `operation=spawning`, `spawn_pending=True`,
and the host boot ID before `Popen`.
8. Publish PID/start time/boot ID through `on_spawn` before socket readiness
or API configuration; clear the pending-spawn guard.
9. Configure Firecracker and `InstanceStart`, verify policy enforcement again,
then save `running` with no pending operation.
The record is persisted as `starting` before host side effects. The PID and
Linux process-start token are persisted before Firecracker API configuration.
This makes cleanup and later diagnosis possible if API configuration fails.
This records both intent and acquired ownership across crash windows. Never
make a managed TAP usable before enforcement or mark a partial disk ready.
### Rollback Rules
Any `Exception`, `KeyboardInterrupt`, or `SystemExit` during create triggers
compensation:
1. Terminate a started Firecracker process when its identity is known.
2. Remove the TAP when it was created.
3. Remove the VM runtime directory and private disk.
4. Remove the reserved state record if cleanup succeeds.
5. Preserve a `failed` record with `last_error` if cleanup itself fails.
1. Attempt quarantine/TAP deactivation and confirm safe process exit.
2. Remove the owned TAP, clean up scoped conntrack entries, and remove its policy.
3. Remove the owned runtime directory/private disk only after confirmed exit.
4. Remove the reservation only if cleanup succeeds.
5. Otherwise retain a `failed` record with identity/intent and `last_error`;
do not release its disk or IP for reuse.
Do not reorder these operations without considering leaked TAPs, orphaned VMM
processes, and allocation reuse.
### Stop and Destroy
### Start and Restart
Start requires managed ports, a proven-gone process (or an already identified
running VM), a complete safe private disk, and compatible retained gateway,
network, and bridge settings. It verifies the VMM and installed **kernel**;
the original rootfs template is not needed. It never recopies the rootfs,
reprovisions credentials, or regenerates guest SSH host keys. Name/IP/MAC,
ports, and guest disk changes survive stop/start; guest RAM does not.
Start on a healthy running VM synchronizes policy without spawning another
VMM. An ambiguous process or interrupted live boot/stop is not adopted as a
successful start. Failed resume retains the private disk for retry. Restart
performs stop and start under one operation lock, not two independent public
calls with a race between them.
### Stop and Delete
```text
stop:
running/failed/stopped -> stopping -> terminate process -> delete TAP -> stopped
private disk and config remain
persist stop intent -> quarantine/deactivate TAP -> confirm VMM exit
-> delete owned TAP -> scoped conntrack cleanup -> remove owned nft table
-> stopped (retain disk, credentials, name/IP and desired ports)
destroy:
any non-transition state -> terminating -> terminate process -> delete TAP
-> remove VM runtime directory -> remove state record
delete (destroy alias):
persist delete intent -> same runtime cleanup -> remove owned runtime directory
-> remove state record -> release name/IP
```
`stop` is intentionally disk-preserving. There is no public `start` command
yet, so do not add start behavior by modifying stop semantics.
Stop signals the **VMM**, not a guest shutdown agent or ACPI-coordinated
shutdown. SIGTERM is followed by SIGKILL when needed. `--timeout` is the
per-VM TERM wait, default 5 seconds, finite and non-negative; it is not a total
deadline. Even zero still requires confirmed kernel exit before cleanup.
Application consistency requires separately quiescing the guest.
Runtime cleanup retains reservations when process exit, TAP ownership, or
connection cleanup is uncertain. A pending spawn cannot be declared gone
merely because its PID was never persisted or its socket has not appeared.
### Policy and Bulk Operations
`set_ports()` persists the new allowlist with `operation=policy` before
applying it. Live updates apply atomically and retain conntrack; cleanly
stopped edits are metadata-only. On apply/readback failure, preserve the new
intent, attempt link-down/quarantine, and leave a failed pending record. Never
restore an older, more permissive allowlist after a failed close. An explicit
`ports sync` retries the desired policy and can reactivate a contained live VM.
`run_many()` holds one outer operation lock while processing the fixed ID
snapshot produced before CLI confirmation. It reloads by **exact ID**, not a
reusable name/IP. Results are `succeeded`, `failed`, `skipped` (already absent),
or `interrupted`; ordinary failures do not undo other successes or stop later
targets. Interruptions skip the remaining targets. Read-only views and API
startup never reconcile; there is no autostart or background repair loop.
## 9. Firecracker Integration
@@ -406,14 +422,30 @@ comes from `Settings.network`, not a hard-coded `/24` value.
### Process Safety
Firecracker starts detached in a new session, with stdout/stderr redirected to
the per-VM log. The manager records `/proc/<pid>/stat` start time and only
signals a process when its persisted token still matches. This prevents a stale
state record from killing an unrelated process after PID reuse.
Firecracker starts detached in a new session, appending stdout/stderr to the
per-VM log. Identity consists of PID, `/proc/<pid>/stat` start time, and the
host boot ID. `observe()` distinguishes `alive`, `gone`, and `unknown`;
unreadable evidence is not proof of exit. A changed boot ID or start token can
prove the old process gone without signaling a reused PID.
Legacy records with a PID but no start token are only cleaned up automatically
when that PID is already gone. If it remains alive, UVM fails closed and asks
the operator to inspect it.
Start probes safe pidfd support before spawning. Termination opens a pidfd,
rechecks identity, signals through that handle (never a bare PID), and confirms
whole-process exit before unlinking the socket. Python bindings or libc
bindings may supply pidfd support; there is no unsafe PID-signal fallback.
A zombie leader alone does not prove its worker threads have exited.
Legacy adoption is deliberately narrow. During a locked mutation,
`adopt_legacy_identity()` may fill a missing boot ID only when all evidence
matches: a saved numeric start token, a live pidfd-pinned process, the configured
Firecracker executable path **and inode**, the exact `--api-sock` command line,
and root-owned/protected binary, socket, and ancestor directories. It checks
that the process has not exited before returning, and the lifecycle layer
persists the adopted boot ID. Listing/inspection never adopts it.
A live legacy PID without a saved start token, or with mismatched/inaccessible
binary/socket evidence, remains unknown and fails closed. UVM does not guess
an identity from the current occupant of a PID or delete an in-use disk to
make recovery convenient. Proven-gone legacy processes can be cleaned up.
## 10. Guest Images and Integrity
@@ -451,6 +483,10 @@ writes a manifest marked `verified: false`; a later strict run will not treat
that manifest as publisher trust. Use `install --force` with trusted digests to
refresh an existing artifact without manually deleting managed files.
Retain the explicit opt-out on create/start/restart (or the API server) when
using an unverified development installation. It is not appropriate for
untrusted workloads. A bare strict install requires the trusted digests.
The optional `UVM_FIRECRACKER_BINARY_SHA256` validates a local Firecracker
binary when no verified manifest is available.
@@ -474,7 +510,8 @@ host default uplink
internet
```
[`network.py`](uvm/network.py) owns all networking details:
[`network.py`](uvm/network.py) owns address allocation and host attachments;
[`firewall.py`](uvm/firewall.py) owns native bridge filtering:
- IP allocation from `Settings.network`, excluding network, broadcast, and
gateway addresses.
@@ -484,16 +521,89 @@ internet
Linux bridge.
- Gateway address setup and IPv4 forwarding.
- One egress NAT rule and forward/return rules for the default uplink.
- TAP creation, cleanup, and error reporting.
- TAP ownership aliases (`uvm:<id>`), preparation while down, activation,
deactivation, cleanup, and error reporting.
Network actions are idempotent where practical. Do not adopt an existing TAP:
it may belong to another VM. A name collision is an error.
it may belong to another VM. A create-time name collision is an error. Cleanup
checks the TAP kind and ownership alias, with a narrow unaliased-TAP allowance
for legacy records; it does not treat an arbitrary interface name as ownership.
### Native Bridge Policy
The supported path is a **software Linux bridge** with STP disabled. Each
managed TAP's bridge-slave `group_fwd_mask` must include `0x4000` (the network
layer sets 16384). This sends LLDP through the early filter instead of a
pre-hook delivery/learning path. Firewall apply/status checks the bridge and
port settings when the TAP exists. Do not use hardware/flow offloads or other
paths that bypass these hooks. This is not an offload-isolation guarantee.
Native bridge conntrack requires Linux >= 5.3. Capability checks run
`conntrack --version`, inspect the kernel release, and ask `nft --json --check`
to validate a full policy without committing it. The nft CLI must support
the rendered JSON. This layer needs no `br_netfilter` sysctl changes.
Each VM owns one `bridge` table named `uvm_fw_<digest>`, derived from the
resolved `UVM_BASE` and exact VM ID, with a full ownership comment. Apply uses
an atomic nft JSON transaction, delete-by-handle for the old owned table, and
exclusive create for its replacement. Foreign ownership is an error, not an
adoption opportunity. Readback normalizes presentation differences but compares
all effective objects, ordered rules, hooks, identity, and the full port set;
rule counts or an owner comment alone are not proof of enforcement.
The policy has prerouting/input/forward/output hooks. Early prerouting identity
checks run at priority -310, before conntrack and bridge MAC learning; input
paths recheck identity as well. Managed traffic is bound to the recorded MAC
and private IPv4 address. Spoofed identity, IPv6, VLAN encapsulation, and other
unsupported EtherTypes are dropped. Host-to-guest and peer-to-guest paths are
covered, not just routed IP forwarding.
The current TCP allowlist is checked against conntrack's **original service
destination address/port in both directions**, including established service
flows. There is no blanket established/related accept before it. Consequently
close blocks subsequent packets on existing service connections in both
directions after commit, without promising a TCP RST or immediate application
disconnect. Live close does not flush conntrack, which would risk
reclassifying midstream service traffic as guest egress.
Guest-initiated IPv4 TCP/UDP and replies remain allowed, including outbound DNS.
ARP, ICMP echo, and necessary related ICMP errors are supported. Other inbound
UDP services are not; HTTP/HTTPS are only TCP labels. Empty managed ports
means no inbound TCP service allowance, not quarantine or loss of all egress.
Quarantine drops every scoped TAP path, including ARP.
Offline cleanup requires the TAP to be absent. It deletes and verifies absence
of conntrack entries scoped to the guest IP in all four original/reply source
and destination fields, including NAT reply tuples. Only then may teardown
release the address. Rule removal also requires an absent TAP and verified
table ownership; it never flushes global firewall or conntrack state.
### Coexistence and Recovery
Existing `iptables` egress MASQUERADE/forwarding remains unchanged. The new
policy adds no DNAT, public listener, or host-port mapping and does not replace
the host firewall manager. Public connectivity, DNS, guest reverse proxies/TLS,
and integration with UFW/firewalld/Docker/libvirt remain out of scope. An
administrator's external routing/NAT can still affect reachability; do not
infer source-CIDR restrictions or complete public-exposure protection from the
use of private addresses. Operators must choose a non-conflicting topology.
If an external actor loses or changes UVM rules, recovery is explicit
`ports sync`, not a promised daemon repair. An `enforced` snapshot cannot
prevent later root-level changes or prove that other rules allow traffic.
[FIREWALL.md](FIREWALL.md) retains the original proposal/backend evaluation;
do not import its unimplemented topology checks into current documentation.
## 12. Installer and Host Commands
`Installer` is the only owner of apt installation, Firecracker artifact
handling, downloads, and integrity-manifest publication.
The host package list includes `nftables` and `conntrack` as well as the
existing iproute2/iptables, ext4, OpenSSL, OpenSSH, download, and util-linux
tools. UVM adds no reconciliation daemon or tested external firewall-manager
integration profile.
`system.py` centralizes host command execution:
- `CommandRunner.run()` converts subprocess failures into `CommandError`.
@@ -513,27 +623,12 @@ Python's safe tar extraction filter.
### Server Startup
```sh
cd uvm
python3 -m pip install .
export UVM_API_TOKEN='long-random-visible-ascii-token'
sudo env UVM_API_TOKEN="$UVM_API_TOKEN" \
./uvm.py --serve --host 127.0.0.1 --port 8000
```
For non-loopback binds, both TLS variables are required and loaded into an
`ssl.SSLContext` before Uvicorn starts:
```sh
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'
sudo env \
UVM_API_TOKEN="$UVM_API_TOKEN" \
UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
./uvm.py --serve --host 0.0.0.0 --port 8443
```
See [USAGE.md](USAGE.md) for dependency installation, loopback deployment,
TLS deployment, and authenticated curl examples. Launch through `uvm --serve`
using the Python environment with the installed API dependencies. Every server
requires `UVM_API_TOKEN`, even at `127.0.0.1`. Non-loopback binds additionally
require `UVM_API_TLS_CERT` and `UVM_API_TLS_KEY`; paired certificate/key files
are loaded into an `ssl.SSLContext` before Uvicorn starts.
The token must contain visible ASCII characters without whitespace. It is
never accepted as a UVM command-line argument, so it does not appear in UVM
@@ -548,10 +643,10 @@ or secret manager when that matters.
| `GET` | `/health` | `health.py` | Unauthenticated liveness response |
| `POST` | `/install` | `installation.py` | `{ "force": false }` body |
| `GET` | `/vms` | `vms.py` | List persisted VM records and observed status |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, optional `guest_ip`, `username`, and `password` |
| `GET` | `/vms/{id-or-ip}` | `vms.py` | Lookup by ID or guest IP |
| `POST` | `/vms/{id}/stop` | `vms.py` | Stop while retaining private disk |
| `DELETE` | `/vms/{id}` | `vms.py` | Remove runtime files and state |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports` |
| `GET` | `/vms/{identifier}` | `vms.py` | Inspect by exact ID, name, or guest IP |
| `POST` | `/vms/{identifier}/stop` | `vms.py` | Stop while retaining private disk |
| `DELETE` | `/vms/{identifier}` | `vms.py` | Remove runtime files and state |
All management routes use `get_authorized_application()` and require:
@@ -561,29 +656,31 @@ X-UVM-Token: <UVM_API_TOKEN>
Request schemas reject unexpected fields. `UvmError` instances are rendered as
JSON errors by `server.py`; Pydantic request validation uses FastAPI's normal
422 response format. VM responses include the username but never the password.
422 response format. Create accepts `name: str | null`, `ports: list | null`,
and strict boolean `no_default_ports` (default false). Port objects contain a
strict integer `port` in 1-65535 and uppercase `protocol` TCP/HTTP/HTTPS (TCP
default). The router normalizes/deduplicates ports and adds TCP/22 unless
default ports are disabled, even for null/empty input. It never creates an
unmanaged VM from an omitted policy. Names share CLI validation.
The API intentionally does not expose interactive SSH, raw Firecracker calls,
arbitrary command execution, or host path access.
VM responses include `name`, `ports`, `policy_status`, and `operation` alongside
resource/status fields and username. The schema excludes password and
disk/socket/config/log path fields; error messages may still contain paths.
`ports: null` in a legacy response still means unmanaged. The existing
stop/delete routes use service defaults; no timeout/dry-run/bulk HTTP schemas
were added. Typed not-found/conflict errors map to 404/409.
There are **no new start/restart, ports, or bulk HTTP endpoints**. The API also
does not expose logs, doctor, interactive SSH, raw Firecracker calls, arbitrary
command execution, or host path access. Public `/docs`, `/redoc`, and
`/openapi.json` are informational, not privileged management routes.
## 14. Configuration Reference
| Variable | Default | Used by |
|---|---|---|
| `UVM_BASE` | `/var/lib/uvm` | Storage layout |
| `UVM_NETWORK` | `10.42.0.0/24` | IP allocation and bridge subnet |
| `UVM_GATEWAY` | `10.42.0.1` | Bridge and guest boot args |
| `UVM_BRIDGE` | `uvm0` | Linux bridge name |
| `UVM_KERNEL_URL` | Firecracker quickstart URL | Guest kernel download |
| `UVM_ROOTFS_URL` | Firecracker quickstart URL | Guest rootfs download |
| `UVM_FIRECRACKER_SHA256` | unset | Trusted release archive checksum |
| `UVM_KERNEL_SHA256` | unset | Trusted kernel checksum |
| `UVM_ROOTFS_SHA256` | unset | Trusted rootfs checksum |
| `UVM_FIRECRACKER_BINARY_SHA256` | unset | Optional direct binary checksum |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | `false` | Development-only integrity bypass |
| `UVM_API_TOKEN` | unset | Required API bearer token |
| `UVM_API_TLS_CERT` | unset | TLS certificate for remote API bind |
| `UVM_API_TLS_KEY` | unset | TLS key for remote API bind |
The environment-variable table is in [README.md](README.md#configuration);
[`config.py`](uvm/config.py) is the implementation authority. Keep the same
base/network/gateway/bridge settings across CLI and server operations. The
network identity is retained per VM and checked before resume.
`Settings` validates the gateway, bridge-name length, API token format, and
environment boolean values at process startup. Add configuration fields there,
@@ -598,11 +695,17 @@ Expected failures inherit from `UvmError`:
| `ConfigurationError` | Invalid local settings or environment |
| `ValidationError` | Invalid CLI/API resource input |
| `StateError` | Unsafe or malformed persisted state |
| `NotFoundError` | Missing registry entry |
| `ConflictError` | Ambiguous identifier, allocation conflict, or unsafe/pending operation |
| `CommandError` | Required host command failure |
| `FirecrackerError` | VMM process or Unix API failure |
| `TapCreationError` | TAP setup failed after interface creation |
CLI code catches `UvmError` and prints `uvm: error: ...` without a traceback.
Invalid CLI input returns 2, ordinary operational failure 1, and interruption
130. Batch JSON retains per-target outcomes rather than pretending a partial
success was atomic.
The FastAPI server maps the same class to an error envelope. New expected
operational failures should use this hierarchy rather than leaking raw
`OSError`, `CalledProcessError`, or implementation-specific exceptions.
@@ -611,22 +714,26 @@ operational failures should use this hierarchy rather than leaking raw
These invariants are critical:
1. Only `LifecycleService` may coordinate VM state, disks, TAPs, and
Firecracker as one operation.
1. Only `LifecycleService` may coordinate VM state, disks, TAPs, firewall,
and Firecracker as one operation.
2. Only `StateStore` may read/write `state.json`.
3. All long host mutations acquire `operations.lock`.
4. All short inventory mutations acquire `state.lock`.
5. The installer shares `operations.lock` with lifecycle actions, preventing a
Firecracker binary replacement between verification and launch.
6. A VM rootfs is private before it becomes writable.
7. A PID is not signaled unless its persisted start token still matches
`/proc/<pid>/stat`.
8. A new VM never adopts a pre-existing TAP.
7. Process signaling requires a safe pidfd and matching persisted start/boot
identity; unknown is not gone. Legacy adoption must prove binary/socket identity.
8. A new VM never adopts a pre-existing TAP or activates it before verified policy.
9. A server must have an API token; a remote server must also have usable TLS.
10. Routers call application services; they do not run shell commands or touch
state files directly.
11. Plaintext guest passwords remain in the mode-`0600` registry and never
enter Firecracker config, helper command arguments/traces, or API responses.
enter Firecracker config, helper command arguments/traces, or CLI/API VM output.
12. Retained-disk start never overwrites guest data, credentials, or SSH host keys.
13. No disk/IP release before confirmed exit and scoped network cleanup; no
old-policy rollback after a failed close, and no resurrection of pending deletion.
14. Policy state is observed from the actual owned rules, not a cached applied flag.
When a change would violate one of these rules, redesign the change rather
than adding a special-case bypass.
@@ -636,80 +743,75 @@ than adding a special-case bypass.
### Standard Unit Suite
```sh
cd uvm
python3 -m pip install -e '.[test]'
PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest discover -s tests -v
python3 -B -m unittest discover -s tests -v
```
The suite covers:
- CPU/RAM parsing and validation.
- State serialization, legacy migration, locking, and permissions.
- Checksum verification and integrity-manifest provenance.
- Private rootfs behavior.
- Guest credential persistence, validation, and offline rootfs provisioning.
- IP, MAC, bridge, TAP, NAT, and forwarding command construction.
- Firecracker configuration and API request ordering.
- PID safety and startup interrupt cleanup.
- Lifecycle success, rollback, stop, and destroy behavior.
- CLI SSH safety and `--serve` parsing.
- Server token/TLS validation and conditional API route tests.
Ordinary tests require no root, KVM, Firecracker, or host network mutations.
They cover validation, schema migration/locking/permissions, artifact trust,
private disks/credentials, network/firewall rendering and readback, conntrack
semantics, pidfd/legacy identity, lifecycle recovery/resume/bulk targeting,
CLI parsing/output, and server security. Mock/fixture policy checks do not
prove the kernel's actual packet behavior.
FastAPI route tests are conditionally enabled when both `fastapi` and `httpx`
are installed. They should be enabled in CI using the `test` extra.
### Real Integration Tests
### Isolated Packet Tests
No current test automatically boots a real microVM. Before a release, run an
isolated host test that covers:
The opt-in packet harness must run **only in disposable isolated user/network
namespaces**. It verifies namespace separation before creating links or
changing rules, and never falls back to the parent host namespace. It may
require privileges (or enabled unprivileged user namespaces) plus `unshare`,
`ip`, `bridge`, `nft`, `conntrack`, and `sysctl`:
```sh
UVM_FIREWALL_INTEGRATION=1 python3 -B -m unittest discover -s tests/integration -v
```
The harness uses bridge-connected veths in place of guest TAPs and routed
egress without NAT. It exercises host/peer service traffic, both-direction
close of existing streams while preserving conntrack/egress, IPv6/spoof
blocking, quarantine, and offline cleanup. It does not boot a microVM or
validate real TAP/NAT deployment.
The packet test skipped in this workspace because namespace creation was
denied (`Operation not permitted`). Document that permission limitation, not
an assertion that real packets or a real host were validated. Unit tests and
nft JSON parser checks are not substitutes for kernel packet-path execution.
### Isolated Host Validation
No current test automatically boots a real microVM. Before a release, validate
on an isolated Linux host:
1. Strict checked installation.
2. `create` and successful guest boot.
3. Guest IP reachability and egress.
4. SSH with unique guest host keys.
5. Stop and retained disk inspection.
6. Destroy and IP/TAP release.
7. API launch with token and TLS on a non-loopback test address.
8. Firecracker crash and interrupted create recovery.
5. Stop/start/restart with retained guest data, SSH keys, and credentials.
6. Live private-port close in both directions, empty policy, and explicit sync
after deliberately lost rules inside the isolated setup.
7. Delete and IP/TAP/policy/conntrack cleanup.
8. API launch with token and TLS on a non-loopback test address.
9. Firecracker crash, interrupted create/start/stop/delete, and legacy recovery.
## 18. Common Development Tasks
### Add a CLI Command
1. Put business logic in an existing service or a cohesive new internal
module, not `cli.py`.
2. Add parser arguments in `build_parser()`.
3. Add display/dispatch behavior in `_run_command()`.
4. Use `UvmError` subclasses for expected failures.
5. Add unit tests for parsing and service behavior.
6. Update `README.md` and this document if the architecture changes.
### Add an HTTP Endpoint
1. Add or extend a Pydantic request/response model in `api_models.py`.
2. Put the endpoint in the appropriate `routers/` module.
3. Use `get_authorized_application()` for every privileged endpoint.
4. Keep `/health` as the only intentionally unauthenticated management route.
FastAPI's default documentation endpoints are informational and public.
5. Add the router to `create_api()`.
6. Add `httpx`/FastAPI tests and update the API table in both docs.
### Add a Host Primitive
1. Put subprocess construction in `system.py`, `network.py`, `install.py`, or
a specific Firecracker adapter.
2. Use `CommandRunner` when invoking external commands.
3. Make repeated application safe where possible.
4. Define compensation behavior before calling it from lifecycle code.
5. Add fake-runner tests; do not require root in ordinary unit tests.
### Change State
1. Increment `STATE_SCHEMA_VERSION` only when the serialized interpretation
changes incompatibly.
2. Keep older state readable or provide an explicit migration path.
3. Update `VmRecord`, `State`, state tests, and runtime cleanup behavior.
4. Consider lock ownership and crash/retry behavior before changing fields.
- CLI: keep business logic in services; wire arguments in `build_parser()` and
dispatch/output in `_run_command()`. Test validation, prompts, dry runs,
JSON, and service behavior; use `UvmError` for expected failures.
- HTTP: define strict models in `api_models.py`, use the appropriate router and
`get_authorized_application()`, and register new routers in `create_api()`.
Add API tests and update all route tables. New CLI commands do not implicitly
authorize new HTTP endpoints; `/health` remains the only unauthenticated
operational route.
- Host primitives: use the specific network/firewall/install/process module
and `CommandRunner` for external commands. Define ownership, idempotence,
and compensation before invoking a primitive from lifecycle code. Use fake
runners rather than requiring privileges in unit tests.
- State: update `VmRecord`/`State`, schema tests, cleanup, and documentation.
Version incompatible interpretations and preserve older data or provide an
explicit migration. Consider lock order and crash/retry behavior first.
## 19. Debugging Guide
@@ -718,27 +820,19 @@ isolated host test that covers:
| `guest assets missing` | `/var/lib/uvm/images`, integrity variables, `integrity.json` |
| Firecracker exits early | VM `firecracker.log`, VM config, `/dev/kvm` access |
| VM is `dead` in `list` | PID/start token in state, `/proc/<pid>`, VM log |
| Cannot stop legacy VM | Missing process start token; inspect PID manually before cleanup |
| Cannot stop legacy VM | Missing start/boot identity or failed protected binary/socket match; never guess a PID |
| Guest has no egress | `ip route show default`, bridge address, TAP state, iptables rules |
| Policy missing/drift/unknown | Actual nft JSON and ownership, STP/group mask, permissions; explicit sync after fixing cause |
| Retained-disk start fails | `disk_ready`, safe private path, kernel/VMM integrity, process identity, retained network, pending operation |
| API server refuses to start | `UVM_API_TOKEN`, requested host, TLS cert/key files |
| API returns 401 | `X-UVM-Token` header mismatch |
| API returns 422 | Request schema or UVM input validation failure |
Useful host commands during isolated debugging:
```sh
ip link show uvm0
ip addr show dev uvm0
ip route show default
iptables -t nat -S POSTROUTING
iptables -S FORWARD
ls -la /var/lib/uvm/vms
sudo cat /var/lib/uvm/state.json # Contains plaintext guest passwords; do not share.
```
Do not use these commands to mutate UVM resources manually while a lifecycle
operation is active. Prefer fixing the service behavior and retrying through
the CLI or API.
Use `inspect VM --json`, `logs VM`, `ports list --all`, and `doctor` before
examining host state. [USAGE.md](USAGE.md) lists read-only bridge/nft/iptables
debugging commands. Never share the credential registry or unredacted guest
logs. Do not manually mutate UVM resources during a lifecycle operation;
fix the service/host failure and retry through supported commands.
## 20. Current Production Gaps
@@ -749,9 +843,13 @@ The code is intentionally honest about current limits:
- No cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU is represented by rounding vCPUs up and printing an advisory.
- Guest health is inferred from VMM process state, not application readiness.
- There is no DHCP, port mapping, ingress firewall policy, or network tenancy.
- `iptables` rules are incrementally managed rather than rendered from a full
declarative firewall policy.
- No quotas, DHCP, host-port mapping, UDP/IPv6 services, or network tenancy.
- No guest snapshots, extra disks, production image builder, or SSH-key injection.
- Per-VM private TCP ingress policy exists, but public connectivity, DNS,
guest reverse proxy/TLS, and external firewall integration remain out of scope.
- Existing egress `iptables` rules are incremental and unchanged; only the
private bridge policy is atomically rendered per VM.
- No autostart, background reconciliation, or guest-coordinated shutdown.
- The JSON inventory is suitable for one host, not distributed coordination.
- Guest passwords are stored in plaintext in the root-only local registry.
- API authentication is a single bearer token, not a user/tenant model.