feat: Add guest execution agent and vsock support
- Introduced a standalone guest execution agent to facilitate command execution within the guest VM over Firecracker's vsock. - Enhanced FirecrackerClient to support vsock configuration, validating guest CID and UDS path. - Updated configuration to include vsock settings by default. - Refactored FirecrackerProcessManager to manage runtime sockets more effectively. - Implemented guest agent installation and provisioning in the image store. - Added API endpoints for executing commands in the VM and installing the guest agent. - Improved error handling for guest execution failures and agent availability.
This commit is contained in:
+77
-7
@@ -17,7 +17,7 @@ start/stop/restart, CPU/RAM/storage resizing, deletion, fixed-target bulk operat
|
||||
logs, doctor, and explicit port edits/sync. The UVM API and Nest dashboard
|
||||
expose health, installation, VM lifecycle controls, and managed port-policy
|
||||
controls, host diagnostics, bounded VM-log tails, and fixed-target bulk
|
||||
operations. Interactive SSH remains CLI-only.
|
||||
operations, and bounded guest command execution over vsock. Interactive SSH remains CLI-only.
|
||||
|
||||
There is no autostart or background reconciler, multi-host control plane,
|
||||
tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port
|
||||
@@ -89,6 +89,8 @@ Packet integration tests must remain inside disposable isolated namespaces.
|
||||
│ ├── install.py # Apt, Firecracker, kernel, rootfs installation
|
||||
│ ├── images.py # Template lookup, private disks, and guest credentials
|
||||
│ ├── known_hosts.py # Deletion-only cleanup of local SSH host-key caches
|
||||
│ ├── guest_agent.py # Standalone guest daemon, job scopes, and bounded wire protocol
|
||||
│ ├── guest_exec.py # Host-side guest sessions and execution results
|
||||
│ ├── network.py # IP/MAC/TAP/bridge/NAT/forward rules
|
||||
│ ├── firewall.py # Owned nft bridge tables and offline conntrack cleanup
|
||||
│ ├── lifecycle.py # Lifecycle, bulk operations, recovery, port intent
|
||||
@@ -238,7 +240,8 @@ Default runtime storage is `/var/lib/uvm`:
|
||||
│ ├── rootfs.ext4
|
||||
│ ├── config.json
|
||||
│ ├── firecracker.log
|
||||
│ └── firecracker.sock
|
||||
│ ├── firecracker.sock
|
||||
│ └── vsock.sock
|
||||
├── state.json
|
||||
├── state.lock
|
||||
├── operations.lock
|
||||
@@ -518,8 +521,9 @@ The `firecracker/` package has three separate responsibilities.
|
||||
2. `PUT /boot-source`
|
||||
3. `PUT /drives/rootfs`
|
||||
4. `PUT /network-interfaces/eth0`
|
||||
5. `PUT /entropy` with `{}` (a host-backed virtio RNG device)
|
||||
6. `PUT /actions` with `InstanceStart`
|
||||
5. `PUT /vsock` with a guest CID and a per-VM Unix socket path
|
||||
6. `PUT /entropy` with `{}` (a host-backed virtio RNG device)
|
||||
7. `PUT /actions` with `InstanceStart`
|
||||
|
||||
Every generated config includes `entropy`, including retained-disk boots.
|
||||
An empty object enables the device with default settings; do not skip it with
|
||||
@@ -607,7 +611,7 @@ fail locally with `Network is unreachable`.
|
||||
|
||||
`ImageStore.configure_guest_network()` fills in the assigned address/prefix and
|
||||
gateway during creation and stopped-disk resume. It only specializes the exact
|
||||
shipped profile from recognized UVM Jammy v1/v2/v3 images. Missing, customized,
|
||||
shipped profile from recognized UVM Jammy v1/v2/v3/v4 images. Missing, customized,
|
||||
symlinked, hard-linked, or unexpectedly owned/mode-set profiles are left intact.
|
||||
An already-specialized profile is a no-op. Custom images retain responsibility
|
||||
for their own guest networking.
|
||||
@@ -617,7 +621,8 @@ For migration, the image layer sparse-copies the private disk and runs
|
||||
committed guest edits cannot be overwritten based on stale offline reads.
|
||||
The updated profile is read back and the filesystem checked before fsync and
|
||||
atomic disk replacement. Failed staging keeps the original disk available for
|
||||
retry. The built template's recipe marker is v3; cached older templates are
|
||||
retry. The built template's recipe marker is v4 (v3 introduced the route fix);
|
||||
cached older templates are
|
||||
also handled by per-VM provisioning without requiring an installation refresh.
|
||||
|
||||
### Integrity Model
|
||||
@@ -894,6 +899,8 @@ points expose the remaining management routes below.
|
||||
| `DELETE` | `/vms/{identifier}` | `vms.py` | Remove runtime files/state; optional `timeout` query parameter |
|
||||
| `POST` | `/vms/{vm_id}/ports` | `vms.py` | Exact ID only; body has `mode` and `ports` |
|
||||
| `POST` | `/vms/{vm_id}/ports/sync` | `vms.py` | Exact ID only; reapply managed policy |
|
||||
| `POST` | `/vms/{vm_id}/exec` | `vms.py` | Exact ID; `command`, strict `sudo`, bounded `timeout_seconds` |
|
||||
| `POST` | `/vms/{vm_id}/agent/install` | `vms.py` | Exact ID; install/upgrade the agent on a stopped disk |
|
||||
| `GET` | `/vms/{identifier}/logs` | `vms.py` | Bounded non-following log tail |
|
||||
|
||||
All management routes use `get_authorized_application()` and require:
|
||||
@@ -940,10 +947,62 @@ marks a response truncated when an oversized line or that bound prevents a full
|
||||
tail. Bulk bodies use an action plus either explicit identifiers or `all_vms`;
|
||||
the router resolves them before `run_many()` executes the fixed IDs under one
|
||||
operation lock. The API does not expose interactive SSH, raw Firecracker calls,
|
||||
arbitrary command execution, or host path access. Public `/`, `/dashboard`,
|
||||
arbitrary host command execution, or host path access. Public `/`, `/dashboard`,
|
||||
`/docs`, `/redoc`, and `/openapi.json` do not perform privileged management
|
||||
requests.
|
||||
|
||||
### Guest Execution
|
||||
|
||||
`LifecycleService.execute()` resolves an exact registry ID, validates canonical
|
||||
paths and an observed running VMM with no pending operation, and connects under
|
||||
`operations.lock`. `GuestSession` checks the Unix peer PID against the selected
|
||||
VMM, sends Firecracker's `CONNECT 1070\n`, consumes its `OK` acknowledgement, and
|
||||
checks the agent's protocol version, VM ID, and configured username. The lock is
|
||||
released before submission/waiting; the established socket pins the original
|
||||
VMM and is never reconnected across a concurrent restart or address reuse.
|
||||
|
||||
`guest_agent.py` is both a host-importable protocol module and a standalone
|
||||
stdlib-only Python 3.10+ guest daemon. Keep it free of imports from the rest of
|
||||
UVM. Version 1 uses length-prefixed JSON with bounds before allocation, absolute
|
||||
socket deadlines, a 64 KiB UTF-8 command limit, and 1 MiB combined raw output.
|
||||
The synchronous HTTP response exposes only exit code, stdout/stderr, timeout,
|
||||
and truncation. Captured text uses UTF-8 replacement. Nonzero process exits are
|
||||
HTTP 200 results; agent failures are 502, and failures before submission are 503.
|
||||
Lost/malformed responses after submission carry `outcome_unknown: true`. There
|
||||
is no command retry, durable job registry, streaming response, or reconnection.
|
||||
|
||||
The root systemd agent accepts only host-CID connections and limits concurrent
|
||||
connections/jobs to eight. Each request creates a delegated cgroup v2 job scope.
|
||||
A fresh child interpreter joins the scope before initializing the chosen user's
|
||||
supplementary groups, GID, UID, home, and environment, then execs `/bin/sh -c`.
|
||||
`sudo` selects guest root directly; otherwise the recorded username is used.
|
||||
Password authentication is not involved, and a recorded `root` login remains
|
||||
root for both flag values. Input is closed and there is no PTY.
|
||||
|
||||
Both output pipes are drained concurrently. The guest enforces the deadline,
|
||||
signals the process group, escalates via `cgroup.kill`, and confirms cleanup.
|
||||
The leader is observed with `waitid(WNOWAIT)` until signaling finishes, preventing
|
||||
PGID reuse before reap. Normal exits clean up remaining background descendants
|
||||
too. An unconfirmed cleanup is an unknown outcome. Stopping the agent service
|
||||
also kills its job scopes; dropping an HTTP connection does not cancel a job.
|
||||
|
||||
The v4 Ubuntu builder installs Python and the service. Creation provisions the
|
||||
VM ID/username in root-only `/etc/uvm-agent.json` before marking the disk ready.
|
||||
Older templates are left for an explicit stopped-disk `agent/install` request.
|
||||
Provisioning and upgrades stage a sparse copy, replay/check ext4, validate guest
|
||||
path ownership and recorded agent-file hashes, enable the service, verify writes,
|
||||
and publish atomically. A root-only guest manifest records managed file hashes
|
||||
so later upgrades preserve guest edits. An older guest missing Python installs
|
||||
it through apt inside the guest on its next boot; no guest binaries execute on
|
||||
the host. This bootstrap requires working guest egress until the runtime is ready.
|
||||
|
||||
The guest needs systemd, Python 3.10+, built-in virtio-vsock, and cgroup v2 with
|
||||
`cgroup.kill` (Linux 5.14+). Host socket paths are derived from Settings, so the
|
||||
registry schema stays at 3. Each Firecracker instance has an independent vsock
|
||||
device and uses CID 3. The process adapter retains both sockets while identity
|
||||
is uncertain and removes them only after confirmed VMM exit. IPv4 port policies
|
||||
do not govern this host-only management channel.
|
||||
|
||||
## 14. Configuration Reference
|
||||
|
||||
The environment-variable table is in [README.md](README.md#configuration);
|
||||
@@ -968,6 +1027,8 @@ Expected failures inherit from `UvmError`:
|
||||
| `ConflictError` | Ambiguous identifier, allocation conflict, or unsafe/pending operation |
|
||||
| `CommandError` | Required host command failure |
|
||||
| `FirecrackerError` | VMM process or Unix API failure |
|
||||
| `GuestAgentUnavailableError` | Agent connection/readiness failure before submission (HTTP 503) |
|
||||
| `GuestExecutionError` | Execution failure or unknown outcome after submission (HTTP 502) |
|
||||
| `TapCreationError` | TAP setup failed after interface creation |
|
||||
|
||||
CLI code catches `UvmError` and prints `uvm: error: ...` without a traceback.
|
||||
@@ -1050,6 +1111,15 @@ addresses/gateways, custom subnets, idempotent migration, journal-replay
|
||||
rechecks, and retention of guest data, credentials and keys on failures.
|
||||
Lifecycle tests require confirmed process exit before network-profile repair.
|
||||
|
||||
Guest execution tests use real child processes with fake cgroup scopes, local
|
||||
Unix sockets emulating Firecracker's vsock handshake, API validation fixtures,
|
||||
and real temporary ext4 upgrades. They cover credential selection, concurrent
|
||||
output draining, timeout cleanup, connection loss, VM targeting/locking, and
|
||||
upgrade rollback. `tests/integration/test_guest_exec.py` is an opt-in smoke test
|
||||
against a prepared disposable VM with a non-root login and all TCP ports closed;
|
||||
see [USAGE.md](USAGE.md#18-verification). Unit tests do not validate the real
|
||||
guest vsock device or cgroup kernel path.
|
||||
|
||||
### Isolated Packet Tests
|
||||
|
||||
The opt-in packet harness must run **only in disposable isolated user/network
|
||||
|
||||
Reference in New Issue
Block a user