kstyagi/nest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tools): Enhance guest tools installation and management

Browse Source 
- Added new API endpoints for tool catalog and installation management.
- Updated usage documentation to include new tool installation features.
- Implemented asynchronous tool installation with job tracking.
- Introduced integration tests for tool installation in disposable VMs.
- Enhanced dashboard UI to reflect tool installation status and usage instructions.
- Improved error handling and logging for tool installation processes.
- Added support for multiple Python versions and Conda environments.
- Refactored tool management logic to ensure better state handling and job persistence.
This commit is contained in:
kstyagi
2026-09-25 23:52:21 +05:30
parent a41aafce83
commit 3e17e3e72f
16 changed files with 729 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEVELOPER.md
+108 -6
View File
@@ -19,7 +19,7 @@ expose health, installation, VM lifecycle controls, and managed port-policy
controls, host diagnostics, bounded VM-log tails, and fixed-target bulk
operations, and bounded guest command execution over vsock. Interactive SSH remains CLI-only.
There is no autostart or background reconciler, multi-host control plane,
There is no general VM autostart or background reconciler, multi-host control plane,
tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port
filtering is not public-port publishing or a hardened multi-tenant boundary;
see Current Production Gaps for the remaining scope limits.
@@ -95,6 +95,10 @@ Packet integration tests must remain inside disposable isolated namespaces.
│ ├── guest_agent.py # Standalone guest daemon, job scopes, and bounded wire protocol
│ ├── guest_exec.py # Host-side guest sessions and execution results
│ ├── guest_terminal.py # Async PTY transport over a process-pinned guest connection
│ ├── tool_catalog.py # Allowlisted tools, Python selections, and pinned release recipes
│ ├── guest_tools.py # Standalone guest-only package/download installer and probes
│ ├── tool_store.py # Atomic per-VM job history and inventory observations
│ ├── tools.py # Bounded background tool installation workers
│ ├── network.py # IP/MAC/TAP/bridge/NAT/forward rules
│ ├── firewall.py # Owned nft bridge tables and offline conntrack cleanup
│ ├── lifecycle.py # Lifecycle, bulk operations, recovery, port intent
@@ -106,6 +110,7 @@ Packet integration tests must remain inside disposable isolated namespaces.
│ │ ├── exit.py # Authenticated server shutdown
│ │ ├── health.py # Liveness endpoint
│ │ ├── installation.py # Installation endpoint
│ │ ├── tools.py # Catalog, inventory, and installation job endpoints
│ │ └── vms.py # VM lifecycle endpoints
│ └── firecracker/
│ ├── config.py # Firecracker JSON and boot arguments
@@ -218,6 +223,8 @@ process:
| FirecrackerProcessManager | `FirecrackerProcessManager` | VMM process control |
| LifecycleService | `LifecycleService` | Ordered VM resource changes and rollback |
| Installer | `Installer` | Host packages and artifacts |
| ToolService | `ToolService` | Browser-independent guest installation queue and workers |
| ToolJobStore | Constructed by lifecycle/tool services | Per-VM installation intent, results, and cached inventory |
Composition can disable command tracing with `emit=None`; `create_api()` does
so when it constructs its own application. The CLI supplies an emitter that
@@ -245,10 +252,12 @@ Default runtime storage is `/var/lib/uvm`:
│ ├── config.json
│ ├── firecracker.log
│ ├── firecracker.sock
│ ├── tools.json
│ └── vsock.sock
├── state.json
├── state.lock
├── operations.lock
├── tool-worker.lock
└── integrity.json
```
@@ -272,6 +281,8 @@ state file.
`VmSpec` is the validated create request. `VmRecord` is the persisted local
record. Do not make API models or CLI namespaces the persistence format.
`VmSpec.tools` is a tuple of validated `ToolSelection` values. Tool job state
uses its own schema-1 `vms/<id>/tools.json`; the VM registry remains schema 3.
| Record Fields | Purpose |
|---|---|
@@ -363,7 +374,8 @@ a ruleset/attachment snapshot, not application health or a reachability test.
3. Reserve ID/name/IP/MAC/ports with `status=starting`, `operation=preparing`.
4. Create the owned mode-0700 runtime directory, sparse-copy a mode-0600 private
disk, resize ext4 to the requested storage capacity, provision credentials/host
keys and the managed guest network profile, then persist `disk_ready=True`.
keys, the managed guest network profile, and the agent, then persist `disk_ready=True`.
If tools were selected, persist their queued job in the owned runtime directory.
5. Ensure bridge/egress networking and prepare an owned, attached TAP **down**.
6. Apply/read back the nftables bridge policy, then activate the TAP.
7. Write Firecracker config; persist `operation=spawning`, `spawn_pending=True`,
@@ -371,10 +383,13 @@ a ruleset/attachment snapshot, not application health or a reachability test.
8. Publish PID/start time/boot ID through `on_spawn` before socket readiness
or API configuration; clear the pending-spawn guard.
9. Configure Firecracker and `InstanceStart`, verify policy enforcement again,
then save `running` with no pending operation.
then save `running` with no pending operation.
This records both intent and acquired ownership across crash windows. Never
make a managed TAP usable before enforcement or mark a partial disk ready.
Tool workers claim the queued intent after this operation releases its lock.
Tool installation errors are job results on the successfully created VM and
never enter create's resource rollback path.
### Rollback Rules
@@ -505,7 +520,8 @@ snapshot produced before CLI confirmation. It reloads by **exact ID**, not a
reusable name/IP. Results are `succeeded`, `failed`, `skipped` (already absent),
or `interrupted`; ordinary failures do not undo other successes or stop later
targets. Interruptions skip the remaining targets. Read-only views and API
startup never reconcile; there is no autostart or background repair loop.
startup never reconcile lifecycle or firewall intent. The tool worker resumes
explicitly queued installation requests, as described under Guest Tools below.
## 9. Firecracker Integration
@@ -792,7 +808,7 @@ do not import its unimplemented topology checks into current documentation.
## 12. Installer and Host Commands
`Installer` is the only owner of apt installation, Firecracker artifact
`Installer` is the owner of host apt installation, Firecracker artifact
handling, downloads, and integrity-manifest publication.
`Settings` detects and normalizes the host architecture at construction time
@@ -893,7 +909,7 @@ Nest. Both server entry points expose the UVM management routes below.
| `GET` | `/diagnostics` | `diagnostics.py` | Host readiness checks |
| `POST` | `/install` | `installation.py` | `{ "force": false }` body |
| `GET` | `/vms` | `vms.py` | List persisted VM records and observed status |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports` |
| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports`, `tools` |
| `POST` | `/vms/actions` | `vms.py` | Fixed-ID snapshot bulk lifecycle action |
| `GET` | `/vms/{identifier}` | `vms.py` | Inspect by exact ID, name, or guest IP |
| `POST` | `/vms/{identifier}/start` | `vms.py` | Boot a retained private disk |
@@ -905,6 +921,10 @@ Nest. Both server entry points expose the UVM management routes below.
| `POST` | `/vms/{vm_id}/ports/sync` | `vms.py` | Exact ID only; reapply managed policy |
| `POST` | `/vms/{vm_id}/exec` | `vms.py` | Exact ID; `command`, strict `sudo`, bounded `timeout_seconds` |
| `POST` | `/vms/{vm_id}/agent/install` | `vms.py` | Exact ID; install/upgrade the agent on a stopped disk |
| `GET` | `/tools` | `tools.py` | Authenticated catalog and supported Python versions |
| `GET` | `/vms/{vm_id}/tools` | `tools.py` | Cached inventory/history; `refresh=true` probes the running guest |
| `POST` | `/vms/{vm_id}/tools/install` | `tools.py` | Exact ID; `tools`, `start_if_stopped`, `restart`, optional `request_id`; HTTP 202 |
| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | `tools.py` | Exact VM-scoped installation job and bounded output |
| `GET` | `/vms/{identifier}/logs` | `vms.py` | Bounded non-following log tail |
| `GET` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Saved session list (Nest only) |
| `POST` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Launch a guest PTY; optional rows/cols in a JSON object (Nest only) |
@@ -1030,6 +1050,78 @@ device and uses CID 3. The process adapter retains both sockets while identity
is uncertain and removes them only after confirmed VMM exit. IPv4 port policies
do not govern this host-only management channel.
### Guest Tools
`tool_catalog.py` is the source of truth for catalog metadata, normalized
`ToolSelection` values, and installer recipes. The API accepts catalog IDs and
explicit Python minors; executable text, download URLs, paths, and other runtime
versions are not request fields. There are ten tools and twelve selectable
entries when Python 3.11/3.12/3.13 are expanded. Update publisher versions, URLs,
and SHA-256 values together when changing a downloaded default.
`ToolService` owns two bounded worker threads by default. The shared UVM server
lifespan starts/stops them, so both Nest and the standalone API process the same
queue. An exclusive, mode-0600 `$UVM_BASE/tool-worker.lock` permits one management
server to own these workers per registry. Construction alone starts no threads.
Queued requests are dispatched on server startup; prior in-flight jobs are
marked interrupted with unknown outcomes after ownership has been acquired.
`LifecycleService.queue_tools()` validates an exact VM ID and persists a request
under the operation lock. Only one queued/active batch is allowed per VM.
Identical active requests are deduplicated. Optional request IDs also identify
completed jobs until they age out of the ten-job history; conflicting reuse is
an error. `prepare_tool_install()` performs an explicitly requested start or
restart under the normal lifecycle lock and returns the selected process identity.
Creation writes its tool job after disk provisioning and before boot.
Workers wait up to `Settings.tool_ready_timeout_s` (120 seconds) for the agent.
Each tool has `Settings.tool_install_timeout_s` (1,200 seconds), bounded by the
existing guest-exec protocol maximum. Readiness connection failures can be
retried before submission; an uncertain installation result is never replayed.
Each step passes `expected_identity` to `LifecycleService.execute()`, so a restart
between steps cannot retarget the batch to a new VMM. The guest boot ID also
detects in-guest reboots. Guest downloads and package waits hold no host operation
lock. Shutdown closes tracked waiting sockets, preserves interruption results,
and keeps worker ownership until outstanding host work has finished.
`guest_tools.py` is a standalone stdlib-only Python 3.10+ payload. Its source and
server-generated JSON recipes are shell-quoted into the bounded guest exec
request. Importing it on the host performs no installation. In the guest it:
- Checks Ubuntu 22.04 and the expected x86_64/ARM64 architecture before mutation.
- Serializes installation through `/run/uvm-tools.lock`, including inherited
lock ownership during package commands. A previous disconnected job can finish
in the guest, and this lock prevents overlapping retries.
- Uses non-interactive apt with bounded dpkg-lock waits and authenticated sources.
A pending marker permits a retry to finish its interrupted dpkg configuration.
PHP's repository key is checked against a pinned fingerprint and scoped with
`signed-by`.
- Checks pinned hashes before executing/extracting downloads. Archive extraction
validates paths/links, file types, and size bounds. Versioned payloads live in
`/opt/uvm-tools`; executable links and login-shell profiles have an ownership
manifest that detects guest edits.
- Checks command versions and companions as the configured guest account, including
npm/npx, Python pip/venv imports, and both Java tools. Python has versioned
executables; Conda uses per-user environments/caches. Get-started commands use
explicit POSIX shell hooks for activation in Nest's non-login shells.
- Captures a bounded 16 KiB output tail and returns structured JSON when each
step finishes. An ordinary recipe failure allows later tools to be attempted.
`ToolJobStore` keeps cached inventory and the last ten jobs in a mode-0600,
schema-1 `tools.json` in the existing private VM runtime directory. Atomic
replacement/fsync and short operation-lock transactions prevent races with
deletion or stale lifecycle record saves. Store writes require a still-registered,
owned VM directory and never recreate a deleted VM. VM deletion removes this
history with the runtime. Full inventory refreshes update `checked_at`; step
results also update individual tool observations. GET without `refresh=true`
returns cached observations, including the `stale` flag.
The dashboard's `tools.js` owns create selections, the VM Tools tab, job polling,
history, retries, and output rendering. Active jobs poll every 1.5 seconds while
visible; idle views poll cached status every five seconds. New selections use the
shared catalog. Navigation/logout invalidate in-flight UI responses, and tool
output is escaped before rendering.
### Nest Terminal Sessions
Nest owns session HTTP/WebSocket routes and history. UVM owns the process-pinned
@@ -1198,6 +1290,16 @@ This checks launch/input, reload/reconnect, session switching, per-VM isolation,
closed-session replay, rename persistence, active/ended deletion, cancellation,
multi-tab updates, and the mobile layout. It does not boot a real microVM.
With the same browser dependencies, `node tests/browser_tools.cjs` checks creation
selections, persisted background jobs across reload, installed-tool detection,
partial failure/retry, escaped output, history, stopped-VM startup, per-VM isolation,
multi-select installation, and mobile layout. Its fixture uses the real API,
worker threads, and atomic job store with simulated guest execution.
`tests/test_tool_*.py` cover validation, architecture-specific payloads, checksum
failure, archive traversal, exact-ID targeting, idempotency, recovery, and shutdown.
The opt-in real-VM catalog/runtime smoke test is documented in
[USAGE.md](USAGE.md#18-verification); run it on disposable x86_64 and ARM64 VMs.
When `nft` and `conntrack` are installed, command-parser tests append an invalid
final command/option so parsing is checked without executing any mutation.
These checks supplement fixture-based readback and cleanup tests; they do not