feat: Add guest execution agent and vsock support
- Introduced a standalone guest execution agent to facilitate command execution within the guest VM over Firecracker's vsock. - Enhanced FirecrackerClient to support vsock configuration, validating guest CID and UDS path. - Updated configuration to include vsock settings by default. - Refactored FirecrackerProcessManager to manage runtime sockets more effectively. - Implemented guest agent installation and provisioning in the image store. - Added API endpoints for executing commands in the VM and installing the guest agent. - Improved error handling for guest execution failures and agent availability.
This commit is contained in:
@@ -873,6 +873,8 @@ curl --cacert /path/to/ca.pem \
|
||||
| `DELETE` | `/vms/{identifier}` | Delete VM and private disk; accepts `timeout` query parameter |
|
||||
| `POST` | `/vms/{vm_id}/ports` | Set, open, or close private ports for an exact VM ID |
|
||||
| `POST` | `/vms/{vm_id}/ports/sync` | Reapply the policy for an exact VM ID |
|
||||
| `POST` | `/vms/{vm_id}/exec` | Execute a guest command/script over vsock, with optional root privileges |
|
||||
| `POST` | `/vms/{vm_id}/agent/install` | Install/upgrade the agent on a stopped VM's disk |
|
||||
| `GET` | `/vms/{identifier}/logs` | Read a bounded, non-following log tail |
|
||||
|
||||
Individual lifecycle requests do not expose a CLI dry-run flag and do not prompt
|
||||
@@ -1046,6 +1048,137 @@ the other policy operations. `open`, `close`, and `sync` reject a legacy policy.
|
||||
Mutating a live policy is subject to the same containment and confirmation rules
|
||||
as the CLI.
|
||||
|
||||
### Execute Commands Without SSH
|
||||
|
||||
`POST /vms/{vm_id}/exec` accepts an **exact VM ID** and a JSON body:
|
||||
|
||||
| Field | Contract |
|
||||
|---|---|
|
||||
| `command` | Required, non-empty command or multiline script; at most 65,536 UTF-8 bytes; no NUL characters |
|
||||
| `sudo` | Strict boolean, default `false`; select guest root instead of the VM's configured login user |
|
||||
| `timeout_seconds` | Finite JSON number greater than 0 and at most 3600; default 60 |
|
||||
|
||||
```sh
|
||||
export VM_ID='<exact-vm-id-from-create-or-list>'
|
||||
curl --fail-with-body -X POST \
|
||||
-H "X-UVM-Token: $UVM_API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"command":"whoami","sudo":true,"timeout_seconds":60}' \
|
||||
"$UVM_API_URL/vms/$VM_ID/exec"
|
||||
```
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
{"exit_code":0,"stdout":"root\n","stderr":"","timed_out":false,"truncated":false}
|
||||
```
|
||||
|
||||
The agent runs `/bin/sh -c COMMAND` in the selected user's home directory, with
|
||||
that user's UID and groups and a fresh `HOME`/`USER`/`PATH` environment. Stdin is
|
||||
closed and no TTY is allocated. The entire script, including pipes and
|
||||
redirections, uses the selected identity. For a Bash script already in the VM,
|
||||
use a command such as `bash /opt/jobs/process.sh`. The flag selects an execution
|
||||
identity directly; it does not invoke an interactive `sudo` password prompt.
|
||||
For VMs whose configured username is `root`, both flag values run as root.
|
||||
|
||||
Submit a local script's contents using `jq` to handle JSON quoting:
|
||||
|
||||
```sh
|
||||
jq -n --rawfile command ./script.sh \
|
||||
'{command:$command,sudo:false,timeout_seconds:300}' |
|
||||
curl --fail-with-body -X POST \
|
||||
-H "X-UVM-Token: $UVM_API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data-binary @- "$UVM_API_URL/vms/$VM_ID/exec"
|
||||
```
|
||||
|
||||
Or call it from Python using the standard library:
|
||||
|
||||
```python
|
||||
import json
|
||||
import os
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
request = Request(
|
||||
f"{os.environ['UVM_API_URL']}/vms/{os.environ['VM_ID']}/exec",
|
||||
data=json.dumps({"command": "id && uname -a", "sudo": False}).encode(),
|
||||
headers={"X-UVM-Token": os.environ["UVM_API_TOKEN"],
|
||||
"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urlopen(request, timeout=75) as response:
|
||||
result = json.load(response)
|
||||
print(result["stdout"])
|
||||
print(result["exit_code"])
|
||||
```
|
||||
|
||||
**Behavior and limits:**
|
||||
|
||||
- The VM must be running with a known process identity and no pending lifecycle
|
||||
operation. Guest readiness is checked separately by connecting to its agent.
|
||||
- The call waits for completion. Nonzero command exits still return HTTP 200;
|
||||
inspect `exit_code`. Negative codes indicate a terminating signal.
|
||||
- The deadline is enforced inside the guest. On timeout the agent terminates
|
||||
the job, including descendants in its cgroup, and returns partial output with
|
||||
`timed_out: true` and `exit_code: null`. Connection and cleanup time are additional
|
||||
to `timeout_seconds`; give the HTTP client about 15 extra seconds.
|
||||
- The agent captures up to **1 MiB combined stdout/stderr**, then drains and
|
||||
discards excess data. `truncated` reports this limit; invalid UTF-8 is replaced.
|
||||
- Up to eight connections/jobs per VM are admitted. Excess connections fail
|
||||
before submission. Execution does not hold the global lifecycle lock while
|
||||
waiting, so another request can stop or restart the VM.
|
||||
- A normal script exit also cleans up background descendants in the job's
|
||||
cgroup. Use a guest service manager for workloads that should outlive the call.
|
||||
- A lost connection after submission returns HTTP 502 with
|
||||
`error.outcome_unknown: true`. The command may already have produced effects;
|
||||
it is never automatically resubmitted. An HTTP client disconnect does not
|
||||
itself cancel the guest job; the guest deadline still applies.
|
||||
- Results are returned in the response, not stored as durable jobs. Commands
|
||||
and their captured output are not copied to the VM registry or host traces.
|
||||
|
||||
#### Prepare New or Existing VMs
|
||||
|
||||
The default Ubuntu template recipe v4 includes Python 3 and `uvm-agent.service`.
|
||||
Rebuild it with `uvm install --force` using the checksum settings from the
|
||||
installation section, then create new VMs. Cached older templates and custom
|
||||
images require the explicit agent installation below. Restart a running Nest or
|
||||
UVM API server to load the new code before making these requests.
|
||||
|
||||
For an existing VM, run these requests in order, checking each succeeds:
|
||||
|
||||
```sh
|
||||
curl --fail-with-body -X POST -H "X-UVM-Token: $UVM_API_TOKEN" \
|
||||
"$UVM_API_URL/vms/$VM_ID/stop"
|
||||
curl --fail-with-body -X POST -H "X-UVM-Token: $UVM_API_TOKEN" \
|
||||
"$UVM_API_URL/vms/$VM_ID/agent/install"
|
||||
curl --fail-with-body -X POST -H "X-UVM-Token: $UVM_API_TOKEN" \
|
||||
"$UVM_API_URL/vms/$VM_ID/start"
|
||||
```
|
||||
|
||||
Installation returns `{"id":"...","status":"installed","restart_required":true}`.
|
||||
It requires a cleanly stopped, proven-offline private disk and does not start the
|
||||
VM itself. Updates are checked on a sparse copy before publication. Guest files,
|
||||
credentials, and SSH keys are retained; customized agent files are preserved
|
||||
rather than overwritten. Repeating installation also upgrades unedited managed
|
||||
agent files using their recorded checksums.
|
||||
|
||||
The guest image must provide systemd and apt (Ubuntu 22.04 or newer is the
|
||||
supported baseline). If Python 3.10+ is missing, a bootstrap service installs
|
||||
`python3` inside the VM on its next boot. This one-time install needs guest DNS,
|
||||
internet access, and working apt repositories. The service retries failed startup;
|
||||
execution returns 503 until it can connect. New v4 templates already include
|
||||
the runtime and do not need this download.
|
||||
|
||||
The guest kernel needs built-in `CONFIG_VSOCKETS=y`, `CONFIG_VIRTIO_VSOCKETS=y`,
|
||||
and `CONFIG_CGROUPS=y`, plus cgroup v2 and `cgroup.kill` (Linux 5.14+). UVM's
|
||||
default Linux 6.1 kernel supports these. Refresh an older installed kernel as
|
||||
described in the installation/entropy recovery sections before booting.
|
||||
|
||||
All transport is via a per-VM Unix socket and Firecracker vsock port 1070. This
|
||||
management channel is independent of the IPv4 port policy, including TCP/22;
|
||||
the guest agent accepts connections from the host CID only. VM execution does
|
||||
not need SSH credentials, SSH host-key caches, or an open guest TCP port.
|
||||
|
||||
### Diagnostics and Log Tails
|
||||
|
||||
The dashboard's System view calls the protected diagnostics endpoint. It runs
|
||||
@@ -1095,6 +1228,8 @@ Use `{"force": true}` to refresh all downloaded artifacts.
|
||||
| `409` | Name/identifier conflict, incompatible retained state, or an unsafe/pending operation. |
|
||||
| `422` | Invalid JSON body, extra request fields, or invalid UVM input. |
|
||||
| `500` | Host command, Firecracker, state, or unexpected operational failure. |
|
||||
| `502` | Guest execution failure or lost/invalid execution response; inspect `error.outcome_unknown`. |
|
||||
| `503` | Guest agent unavailable, not ready, or at its connection limit before command submission. |
|
||||
|
||||
UVM operation errors use this envelope:
|
||||
|
||||
@@ -1103,6 +1238,8 @@ UVM operation errors use this envelope:
|
||||
```
|
||||
|
||||
FastAPI request-schema errors use FastAPI's normal `detail` response format.
|
||||
Guest execution errors additionally include a boolean `outcome_unknown` in the
|
||||
`error` object; `true` means the caller must not assume the command did not run.
|
||||
|
||||
## 13. Configuration Scenarios
|
||||
|
||||
@@ -1347,7 +1484,8 @@ Keep these constraints in mind when deciding whether UVM fits a scenario:
|
||||
DNS publication, guest reverse proxy/TLS setup, DHCP, load balancing,
|
||||
snapshots, multi-network tenancy, or multi-host scheduling.
|
||||
- The API uses one shared bearer token, not individual users, roles, or tenants.
|
||||
- There is no guest readiness probe beyond Firecracker process state.
|
||||
- Execution probes guest-agent readiness; VM list/status still describes the
|
||||
Firecracker process rather than guest application health.
|
||||
- UVM is not a hardened multi-tenant platform.
|
||||
|
||||
## 17. Safe Operating Checklist
|
||||
@@ -1389,6 +1527,24 @@ upstreams to check NAT, offline startup, reconnects and uplink changes. It does
|
||||
not boot Firecracker or validate the host's real TAP setup. Run actual
|
||||
VM/SSH/egress checks on an isolated Linux host before relying on the deployment.
|
||||
|
||||
The execution smoke tests target an already-created **disposable VM on an
|
||||
isolated host**, with a non-root login, an installed/running agent, and an empty
|
||||
TCP allowlist. They check both privilege modes, output and exit status, bounded
|
||||
output, and cgroup cleanup of a descendant that starts its own session. They do
|
||||
not create, stop, or delete VMs:
|
||||
|
||||
```sh
|
||||
UVM_EXEC_INTEGRATION=1 UVM_EXEC_VM_ID='<disposable-vm-id>' \
|
||||
python3 -B -m unittest discover -s tests/integration -p test_guest_exec.py -v
|
||||
```
|
||||
|
||||
Set `UVM_API_URL` and `UVM_API_TOKEN` as in the API examples. For TLS, the Python
|
||||
client uses the system trust store (or `SSL_CERT_FILE`). Without the explicit
|
||||
integration flag these tests skip and make no requests. The ordinary unit tests
|
||||
exercise real child processes with fake cgroup scopes, local Unix-socket protocol
|
||||
fixtures, and temporary ext4 upgrades; they do not prove the real vsock/cgroup
|
||||
path inside a VM.
|
||||
|
||||
## 19. Get Help
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user