kstyagi/nest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

__init__

Browse Source
This commit is contained in:
its.kstyagi@gmail.com
2026-09-04 20:56:18 +00:00
commit 1022f24c34
43 changed files with 6160 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
USAGE.md
+774
View File
@@ -0,0 +1,774 @@
# UVM Usage Guide
This guide explains how to install, operate, and troubleshoot UVM as an
operator. It covers the command-line tool, the optional FastAPI server, common
scenarios, and expected limitations.
For the project architecture and contributor guidance, see
[DEVELOPER.md](DEVELOPER.md). For a concise overview, see [README.md](README.md).
## 1. What UVM Does
UVM creates small Linux microVMs through Firecracker and KVM on one Linux host.
Each VM receives:
- A unique VM ID.
- A private writable root disk copied from a shared template.
- A private IP address and MAC address.
- A TAP interface connected to a host bridge.
- A dedicated Firecracker process, API socket, log, and config file.
By default, guest networking uses:
```text
Guest network: 10.42.0.0/24
Gateway: 10.42.0.1
Bridge: uvm0
Guest pool: 10.42.0.2 through 10.42.0.254
```
The default locations on the host are under `/var/lib/uvm`.
## 2. Before You Start
### Host Requirements
You need:
- Linux, preferably Ubuntu or a compatible distribution.
- Python 3.12 or newer.
- KVM enabled and readable/writable at `/dev/kvm`.
- Root access for installation and VM lifecycle operations.
- An x86_64 host for the default guest kernel/rootfs downloads.
- Internet access if downloading Firecracker and guest assets.
Check virtualization before installing:
```sh
ls -l /dev/kvm
lscpu | grep -i virtualization
```
### Source Checkout Setup
From the parent directory of this project:
```sh
cd uvm
./uvm.py --help
python3 -m uvm --help
```
To install the optional HTTP API dependencies and the console command:
```sh
cd uvm
python3 -m pip install .
uvm --help
```
The rest of this guide uses `./uvm.py` so it works from a source checkout. If
you installed the package, replace `./uvm.py` with `uvm`.
### Privilege Rules
| Operation | Root required? |
|---|---|
| `install` | Yes |
| `create` | Yes |
| `stop` | Yes |
| `destroy` | Yes |
| `list` | Yes with the default root-owned registry |
| `ssh` | Yes when resolving a VM through the registry |
| API server that manages VMs | Yes |
The registry contains plaintext guest credentials and is mode `0600`. Direct
SSH to a known guest IP does not require root.
## 3. Command Overview
```sh
./uvm.py --help
./uvm.py install [--force]
./uvm.py create [--cpu CPU] [--ram RAM] [--host-ip GUEST_IP] \
[--username USER] [--password PASSWORD]
sudo ./uvm.py list
sudo ./uvm.py ssh VM_OR_IP [--user USER] [--key PATH] [--insecure-host-key]
./uvm.py stop VM_ID
./uvm.py destroy VM_ID
./uvm.py --serve [--host HOST] [--port PORT]
```
`--host-ip` is a legacy CLI name. It means the **guest IP address**, not the
host's public or uplink address.
## 4. Install UVM Assets
UVM installs host packages, Firecracker, Jailer, a guest kernel, and a guest
rootfs template. It requires SHA-256 checksums by default because it downloads
artifacts that will be used by a privileged process.
### Scenario: Strict First Installation
Obtain trusted checksums independently, then run:
```sh
cd uvm
sudo env \
UVM_FIRECRACKER_SHA256='<sha256-of-firecracker-archive>' \
UVM_KERNEL_SHA256='<sha256-of-vmlinux>' \
UVM_ROOTFS_SHA256='<sha256-of-rootfs>' \
./uvm.py install
```
On success, UVM prints paths similar to:
```text
Firecracker: /var/lib/uvm/bin/firecracker
Kernel: /var/lib/uvm/images/vmlinux
Rootfs: /var/lib/uvm/images/ubuntu.ext4
```
The checksums and whether the installation was verified are recorded in:
```text
/var/lib/uvm/integrity.json
```
### Scenario: Refresh Firecracker or Guest Assets
Use `--force` after changing a URL, replacing an image, or rotating checksums:
```sh
sudo env \
UVM_FIRECRACKER_SHA256='<new-firecracker-archive-sha256>' \
UVM_KERNEL_SHA256='<new-kernel-sha256>' \
UVM_ROOTFS_SHA256='<new-rootfs-sha256>' \
./uvm.py install --force
```
`--force` downloads to temporary files and only replaces the existing artifact
after checksum verification succeeds.
### Scenario: Local Experimentation Without Checksums
This mode is unsafe for production. It is useful only when you deliberately
accept the source artifacts without checksum verification:
```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py install
```
An unverified install is marked as unverified. It cannot later become a strict
trusted installation merely by setting checksum variables. Reinstall with the
strict command above when you are ready to trust the host assets.
For later local-development VM creation, keep the opt-out on the command too:
```sh
sudo env UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 ./uvm.py create
```
The same rule applies to an API server that creates VMs from an unverified
installation: start that server with `UVM_ALLOW_UNVERIFIED_DOWNLOADS=1`, or
perform a strict reinstall first.
### Installation Failures
| Message | Meaning and action |
|---|---|
| `/dev/kvm does not exist` | Enable hardware/nested virtualization first. |
| `checksum is required` | Provide the requested `UVM_*_SHA256` variable. |
| `SHA-256 mismatch` | Stop and obtain the correct digest or artifact URL. |
| `unsupported host architecture` | Use x86_64 defaults or provide compatible assets. |
| `default guest assets support x86_64 only` | Configure both custom kernel and rootfs URLs for ARM. |
## 5. Create VMs
### Scenario: Create a Default VM
The default is one CPU, 512 MiB RAM, username `root`, and password `root`:
```sh
sudo ./uvm.py create
```
Typical output includes the VM ID, guest IP, TAP name, and SSH target:
```text
VM created: vm-...
IP: 10.42.0.2
RAM: 512 MiB
CPU: 1.0
TAP: uvm-...
Username: root
SSH: ssh root@10.42.0.2
WARNING: the guest is using the default password 'root'. Change it promptly.
```
Set a different password during creation:
```sh
sudo ./uvm.py create --username root --password '<guest-password>'
```
`--username` must name an account already present in the guest image. The
password is visible in the process arguments and may be recorded in shell
history, so avoid reusing a sensitive host or service password.
### Scenario: Choose CPU and RAM
```sh
sudo ./uvm.py create --cpu 2 --ram 2G
sudo ./uvm.py create --cpu 1 --ram 1024M
sudo ./uvm.py create --cpu 1 --ram 768
```
RAM is in MiB by default. Supported suffixes include `B`, `K`, `M`, `G`,
`KiB`, `MiB`, and `GiB`.
UVM enforces a minimum of 128 MiB. Firecracker uses whole vCPUs. A fractional
request such as `--cpu 0.5` is accepted but rounds the VM up to one Firecracker
vCPU; it does not currently apply a host CPU quota.
```sh
sudo ./uvm.py create --cpu 0.5 --ram 512
```
### Scenario: Request a Specific Guest IP
```sh
sudo ./uvm.py create --host-ip 10.42.0.10
```
The address must be inside the configured guest network, must not be the
network address, broadcast address, or gateway, and must not already belong to
another persisted VM.
### What Happens During Create
UVM performs these operations in order:
1. Validates root access, KVM, assets, and integrity state.
2. Reserves a VM ID, IP, MAC, and state record.
3. Creates `/var/lib/uvm/vms/<vm-id>/`.
4. Copies the rootfs template to a private `rootfs.ext4` disk.
5. Sets the requested account password and enables SSH password login in the copy.
6. Ensures the bridge/NAT/forwarding rules exist.
7. Creates a TAP device.
8. Starts and configures Firecracker.
9. Marks the VM as `running`.
If a step fails, UVM attempts to terminate the new VMM, remove the TAP, remove
the private runtime directory, and release the reservation.
## 6. Inspect VMs
```sh
sudo ./uvm.py list
```
Example:
```text
ID IP USER CPU RAM STATUS PID
vm-0123456789abcdef... 10.42.0.2 root 1.0 512 running 12345
```
Status meanings:
| Status | Meaning |
|---|---|
| `starting` | VM creation is in progress. |
| `running` | Firecracker started successfully. |
| `stopping` | Stop is in progress. |
| `stopped` | VMM and TAP are gone; disk remains. |
| `terminating` | Destruction is in progress. |
| `failed` | A lifecycle operation or cleanup failed. |
| `dead` | Stored state claims the VM is active but its VMM process is gone. |
If a VM is `failed` or `dead`, inspect its log before deciding whether to stop
or destroy it:
```sh
sudo ls -la /var/lib/uvm/vms/<vm-id>
sudo cat /var/lib/uvm/vms/<vm-id>/firecracker.log
```
## 7. Connect with SSH
UVM provisions a password for an existing account. The default credentials are
`root` / `root`. Your guest image must include:
- An SSH server.
- The requested login user plus standard passwd and shadow files.
- Unique SSH host keys generated at first boot.
This is applied only while creating a new VM. Existing VM disks and legacy
registry entries are not changed retroactively.
### Scenario: SSH as Root
```sh
sudo ./uvm.py ssh vm-<id>
ssh root@10.42.0.2
```
### Scenario: Use a Different User and Private Key
```sh
sudo ./uvm.py ssh vm-<id> --user ubuntu --key ~/.ssh/id_ed25519
```
By default, UVM uses SSH `StrictHostKeyChecking=accept-new` and a stable
host-key alias derived from the VM ID. This lets a guest IP be reused later
without confusing it with the prior VM's known-hosts entry.
### Scenario: Temporarily Bypass Host-Key Verification
Use only for troubleshooting a guest you trust:
```sh
sudo ./uvm.py ssh vm-<id> --insecure-host-key
```
This disables normal host-key checking for that connection.
## 8. Stop and Destroy VMs
### Scenario: Stop a VM and Keep Its Disk
```sh
sudo ./uvm.py stop vm-<id>
```
Stopping terminates Firecracker and deletes the TAP device. The private root
disk, config, and log remain under the VM runtime directory.
UVM does not currently offer a `start` command, so a stopped disk cannot be
rebooted through the public CLI yet. Keep it only if you need to inspect it or
expect a future start feature.
### Scenario: Permanently Remove a VM
```sh
sudo ./uvm.py destroy vm-<id>
```
Destroying a VM:
1. Terminates the VMM if it is alive.
2. Deletes the TAP device.
3. Deletes the VM runtime directory and private root disk.
4. Removes the VM record from `state.json`.
5. Releases its IP address for later allocation.
This is destructive. Back up guest data before running it.
## 9. Use the FastAPI Server
The API is an alternative management surface for the same local lifecycle
services. It does not make UVM multi-host or safe to expose directly to the
public internet.
### Scenario: Start a Local API Server
Install dependencies first:
```sh
cd uvm
python3 -m pip install .
```
Then start the server as root so lifecycle endpoints can manipulate KVM,
networking, and host files:
```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'
sudo env \
UVM_API_TOKEN="$UVM_API_TOKEN" \
./uvm.py --serve --host 127.0.0.1 --port 8000
```
Open these local URLs after it starts:
```text
Health: http://127.0.0.1:8000/health
OpenAPI: http://127.0.0.1:8000/docs
```
The API token is required even for a loopback server. All VM and installation
routes require it. FastAPI's informational `/docs`, `/redoc`, and
`/openapi.json` endpoints are also available without a token; they do not
perform host mutations.
### Scenario: Start a Remote TLS API Server
Non-loopback binds require a valid TLS certificate and key. The files must be
readable by the server process:
```sh
export UVM_API_TOKEN='replace-with-a-long-random-visible-ascii-token'
export UVM_API_TLS_CERT='/etc/uvm/api-cert.pem'
export UVM_API_TLS_KEY='/etc/uvm/api-key.pem'
sudo env \
UVM_API_TOKEN="$UVM_API_TOKEN" \
UVM_API_TLS_CERT="$UVM_API_TLS_CERT" \
UVM_API_TLS_KEY="$UVM_API_TLS_KEY" \
./uvm.py --serve --host 0.0.0.0 --port 8443
```
For real remote use, use a certificate trusted by clients and restrict network
access with a firewall or reverse proxy. Do not use a self-signed development
certificate for an untrusted network.
### API Authentication
Set one reusable shell variable for API examples:
```sh
export UVM_API_TOKEN='replace-with-the-server-token'
export UVM_API_URL='http://127.0.0.1:8000'
```
Every management request needs:
```http
X-UVM-Token: <token>
```
For a TLS server, set `UVM_API_URL` to an `https://` URL and use your CA:
```sh
export UVM_API_URL='https://host.example:8443'
curl --cacert /path/to/ca.pem \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms"
```
## 10. API Endpoint Examples
### Health Check
No token is required:
```sh
curl "$UVM_API_URL/health"
```
Response:
```json
{
"status": "ok"
}
```
### List VMs
```sh
curl \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms"
```
Example response:
```json
[
{
"id": "vm-0123456789abcdef",
"cpu": 1.0,
"ram_mib": 512,
"guest_ip": "10.42.0.2",
"gateway": "10.42.0.1",
"mac": "02:fc:00:00:00:01",
"username": "root",
"status": "running",
"observed_status": "running",
"pid": 12345,
"created_at": 1700000000,
"updated_at": 1700000000,
"last_error": null
}
]
```
### Create a VM
Create a default-sized VM:
```sh
curl -X POST \
-H "Content-Type: application/json" \
-H "X-UVM-Token: $UVM_API_TOKEN" \
-d '{"cpu": 1, "ram": "512", "username": "root", "password": "root"}' \
"$UVM_API_URL/vms"
```
Create a VM with a chosen guest address:
```sh
curl -X POST \
-H "Content-Type: application/json" \
-H "X-UVM-Token: $UVM_API_TOKEN" \
-d '{"cpu": 2, "ram": "2G", "guest_ip": "10.42.0.10", "username": "root", "password": "replace-me"}' \
"$UVM_API_URL/vms"
```
The response is the VM record with its username, but never its password.
Omitting credentials uses `root` / `root`. Request fields are strict:
misspelled or extra fields return HTTP `422` instead of being silently ignored.
### Look Up One VM
By ID:
```sh
curl \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms/vm-0123456789abcdef"
```
By guest IP:
```sh
curl \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms/10.42.0.2"
```
### Stop a VM
```sh
curl -X POST \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms/vm-0123456789abcdef/stop"
```
The response is the VM record with `status: "stopped"`.
### Destroy a VM
```sh
curl -X DELETE \
-H "X-UVM-Token: $UVM_API_TOKEN" \
"$UVM_API_URL/vms/vm-0123456789abcdef"
```
Response:
```json
{
"id": "vm-0123456789abcdef",
"status": "terminated"
}
```
### Install or Refresh Through the API
The server process must itself have checksum variables configured for strict
installation. Start the server with those environment variables before using
this endpoint.
For a deliberately unverified local-development server, the server process
must instead include `UVM_ALLOW_UNVERIFIED_DOWNLOADS=1`. Do not use that mode
on a host that runs untrusted workloads.
```sh
curl -X POST \
-H "Content-Type: application/json" \
-H "X-UVM-Token: $UVM_API_TOKEN" \
-d '{"force": false}' \
"$UVM_API_URL/install"
```
Refresh all downloaded artifacts:
```sh
curl -X POST \
-H "Content-Type: application/json" \
-H "X-UVM-Token: $UVM_API_TOKEN" \
-d '{"force": true}' \
"$UVM_API_URL/install"
```
## 11. API Errors
| Status | Typical meaning |
|---|---|
| `401` | Missing or incorrect `X-UVM-Token`. |
| `403` | UVM process lacks root privileges for a host mutation. |
| `404` | Requested VM was not found. |
| `409` | A lifecycle operation is already in progress. |
| `422` | Invalid JSON body, extra request fields, or invalid UVM input. |
| `500` | Host command, Firecracker, state, or unexpected operational failure. |
UVM operation errors use this envelope:
```json
{
"error": {
"message": "human-readable explanation"
}
}
```
FastAPI request-schema errors use FastAPI's normal `detail` response format.
## 12. Configuration Scenarios
### Scenario: Use a Different Storage Directory
```sh
sudo env \
UVM_BASE='/srv/uvm' \
UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 \
./uvm.py install
```
All UVM binaries, images, VM runtime directories, locks, and state use the
configured base path.
### Scenario: Use a Different Guest Network
```sh
sudo env \
UVM_NETWORK='10.50.0.0/24' \
UVM_GATEWAY='10.50.0.1' \
UVM_BRIDGE='uvm50' \
UVM_ALLOW_UNVERIFIED_DOWNLOADS=1 \
./uvm.py install
```
Use the same environment values for later `create`, `list`, `stop`, `destroy`,
and server commands. Changing the network variables after VMs exist can make
their persisted state inconsistent with host networking.
### Environment Variables
| Variable | Use |
|---|---|
| `UVM_BASE` | Base data directory; default `/var/lib/uvm` |
| `UVM_NETWORK` | Guest subnet; default `10.42.0.0/24` |
| `UVM_GATEWAY` | Guest gateway / bridge IP; default `10.42.0.1` |
| `UVM_BRIDGE` | Linux bridge name; default `uvm0` |
| `UVM_KERNEL_URL` | Custom kernel download URL |
| `UVM_ROOTFS_URL` | Custom rootfs download URL |
| `UVM_FIRECRACKER_SHA256` | Strict Firecracker archive checksum |
| `UVM_KERNEL_SHA256` | Strict kernel checksum |
| `UVM_ROOTFS_SHA256` | Strict rootfs checksum |
| `UVM_FIRECRACKER_BINARY_SHA256` | Optional local binary checksum |
| `UVM_ALLOW_UNVERIFIED_DOWNLOADS` | Set to `1` only for local development |
| `UVM_API_TOKEN` | Required API token |
| `UVM_API_TLS_CERT` | TLS certificate for remote server binds |
| `UVM_API_TLS_KEY` | TLS key for remote server binds |
## 13. Files, Logs, and State
The default host layout is:
```text
/var/lib/uvm/
├── bin/ # Firecracker and Jailer binaries
├── images/ # Shared kernel and rootfs template
├── vms/<vm-id>/ # Private rootfs, config, socket, and log
├── integrity.json # Artifact checksums and verification status
└── state.json # Mode-0600 VM inventory and plaintext guest credentials
```
Useful inspection commands:
```sh
sudo cat /var/lib/uvm/state.json
sudo ls -la /var/lib/uvm/vms
sudo cat /var/lib/uvm/vms/<vm-id>/firecracker.log
sudo cat /var/lib/uvm/vms/<vm-id>/config.json
```
Avoid editing `state.json`, VM config files, runtime directories, TAPs, or
iptables rules by hand while UVM is running. Use UVM commands or the API so
locks and cleanup rules remain correct.
`state.json` contains plaintext guest passwords. Keep it root-only, never copy
it into logs or bug reports, and use unique non-default passwords outside local
throwaway environments.
## 14. Networking Troubleshooting
Check the bridge and TAP state:
```sh
ip link show uvm0
ip addr show dev uvm0
ip route show default
```
Check NAT and forwarding rules:
```sh
sudo iptables -t nat -S POSTROUTING
sudo iptables -S FORWARD
```
Common issues:
| Symptom | First checks |
|---|---|
| Guest cannot reach the internet | Bridge is up, TAP is up, host has default route, NAT and FORWARD rules exist. |
| Requested guest IP is rejected | Confirm it is inside the configured subnet and not used, gateway, network, or broadcast. |
| Create says TAP already exists | Inspect the interface; do not delete it unless you know it is stale UVM state. |
| Existing bridge is rejected | `UVM_BRIDGE` points to an interface that is not a Linux bridge. |
| SSH cannot connect | Confirm guest booted, the requested user exists, the image runs OpenSSH, and the configured password is being used. |
## 15. Important Limitations
Keep these constraints in mind when deciding whether UVM fits a scenario:
- Firecracker is launched directly; the downloaded Jailer is not yet used.
- No cgroup CPU, memory, I/O, or process limits are applied.
- Fractional CPU requests are advisory only.
- There is no public `start` or `restart` command after a VM is stopped.
- There is no DHCP, inbound port mapping, load balancing, multi-network
isolation, or multi-host scheduling.
- The API uses one shared bearer token, not individual users, roles, or tenants.
- There is no guest readiness probe beyond Firecracker process state.
- UVM is not a hardened multi-tenant platform.
## 16. Safe Operating Checklist
Before relying on a VM for useful work:
1. Use strict checksums for installed artifacts.
2. Confirm `/dev/kvm` and nested virtualization are available.
3. Use a guest image with unique first-boot SSH host keys and set a non-default password.
4. Test guest egress and SSH on the actual host network.
5. Back up guest data before destroy operations.
6. Keep the API loopback-only unless you have a strong remote-access need.
7. For remote API access, use a long random token, valid TLS, and firewall
restrictions.
8. Review `/var/lib/uvm/vms/<vm-id>/firecracker.log` after failed starts.
## 17. Get Help
```sh
./uvm.py --help
./uvm.py install --help
./uvm.py --serve --help
```
When reporting a problem, collect:
- The exact command or HTTP request.
- The full error message.
- Output from `sudo ./uvm.py list`.
- The VM Firecracker log, if a VM was created.
- The relevant `UVM_*` variables, with secrets such as `UVM_API_TOKEN` removed.
Never include `/var/lib/uvm/state.json` because it contains guest passwords.