feat(tools): Enhance guest tools installation and management

- Added new API endpoints for tool catalog and installation management.
- Updated usage documentation to include new tool installation features.
- Implemented asynchronous tool installation with job tracking.
- Introduced integration tests for tool installation in disposable VMs.
- Enhanced dashboard UI to reflect tool installation status and usage instructions.
- Improved error handling and logging for tool installation processes.
- Added support for multiple Python versions and Conda environments.
- Refactored tool management logic to ensure better state handling and job persistence.

DEVELOPER.md

@@ -19,7 +19,7 @@ expose health, installation, VM lifecycle controls, and managed port-policy
 controls, host diagnostics, bounded VM-log tails, and fixed-target bulk
 operations, and bounded guest command execution over vsock. Interactive SSH remains CLI-only.
 
-There is no autostart or background reconciler, multi-host control plane,
+There is no general VM autostart or background reconciler, multi-host control plane,
 tenancy/quotas, or UVM-provided Jailer/cgroup/namespace hardening. Private port
 filtering is not public-port publishing or a hardened multi-tenant boundary;
 see Current Production Gaps for the remaining scope limits.
@@ -95,6 +95,10 @@ Packet integration tests must remain inside disposable isolated namespaces.
 │   ├── guest_agent.py        # Standalone guest daemon, job scopes, and bounded wire protocol
 │   ├── guest_exec.py         # Host-side guest sessions and execution results
 │   ├── guest_terminal.py     # Async PTY transport over a process-pinned guest connection
+│   ├── tool_catalog.py       # Allowlisted tools, Python selections, and pinned release recipes
+│   ├── guest_tools.py        # Standalone guest-only package/download installer and probes
+│   ├── tool_store.py         # Atomic per-VM job history and inventory observations
+│   ├── tools.py              # Bounded background tool installation workers
 │   ├── network.py            # IP/MAC/TAP/bridge/NAT/forward rules
 │   ├── firewall.py           # Owned nft bridge tables and offline conntrack cleanup
 │   ├── lifecycle.py          # Lifecycle, bulk operations, recovery, port intent
@@ -106,6 +110,7 @@ Packet integration tests must remain inside disposable isolated namespaces.
 │   │   ├── exit.py           # Authenticated server shutdown
 │   │   ├── health.py         # Liveness endpoint
 │   │   ├── installation.py   # Installation endpoint
+│   │   ├── tools.py          # Catalog, inventory, and installation job endpoints
 │   │   └── vms.py            # VM lifecycle endpoints
 │   └── firecracker/
 │       ├── config.py         # Firecracker JSON and boot arguments
@@ -218,6 +223,8 @@ process:
 | FirecrackerProcessManager | `FirecrackerProcessManager` | VMM process control |
 | LifecycleService | `LifecycleService` | Ordered VM resource changes and rollback |
 | Installer | `Installer` | Host packages and artifacts |
+| ToolService | `ToolService` | Browser-independent guest installation queue and workers |
+| ToolJobStore | Constructed by lifecycle/tool services | Per-VM installation intent, results, and cached inventory |
 
 Composition can disable command tracing with `emit=None`; `create_api()` does
 so when it constructs its own application. The CLI supplies an emitter that
@@ -245,10 +252,12 @@ Default runtime storage is `/var/lib/uvm`:
 │       ├── config.json
 │       ├── firecracker.log
 │       ├── firecracker.sock
+│       ├── tools.json
 │       └── vsock.sock
 ├── state.json
 ├── state.lock
 ├── operations.lock
+├── tool-worker.lock
 └── integrity.json
 ```
 
@@ -272,6 +281,8 @@ state file.
 
 `VmSpec` is the validated create request. `VmRecord` is the persisted local
 record. Do not make API models or CLI namespaces the persistence format.
+`VmSpec.tools` is a tuple of validated `ToolSelection` values. Tool job state
+uses its own schema-1 `vms/<id>/tools.json`; the VM registry remains schema 3.
 
 | Record Fields | Purpose |
 |---|---|
@@ -363,7 +374,8 @@ a ruleset/attachment snapshot, not application health or a reachability test.
 3. Reserve ID/name/IP/MAC/ports with `status=starting`, `operation=preparing`.
 4. Create the owned mode-0700 runtime directory, sparse-copy a mode-0600 private
    disk, resize ext4 to the requested storage capacity, provision credentials/host
-   keys and the managed guest network profile, then persist `disk_ready=True`.
+   keys, the managed guest network profile, and the agent, then persist `disk_ready=True`.
+   If tools were selected, persist their queued job in the owned runtime directory.
 5. Ensure bridge/egress networking and prepare an owned, attached TAP **down**.
 6. Apply/read back the nftables bridge policy, then activate the TAP.
 7. Write Firecracker config; persist `operation=spawning`, `spawn_pending=True`,
@@ -375,6 +387,9 @@ a ruleset/attachment snapshot, not application health or a reachability test.
 
 This records both intent and acquired ownership across crash windows. Never
 make a managed TAP usable before enforcement or mark a partial disk ready.
+Tool workers claim the queued intent after this operation releases its lock.
+Tool installation errors are job results on the successfully created VM and
+never enter create's resource rollback path.
 
 ### Rollback Rules
 
@@ -505,7 +520,8 @@ snapshot produced before CLI confirmation. It reloads by **exact ID**, not a
 reusable name/IP. Results are `succeeded`, `failed`, `skipped` (already absent),
 or `interrupted`; ordinary failures do not undo other successes or stop later
 targets. Interruptions skip the remaining targets. Read-only views and API
-startup never reconcile; there is no autostart or background repair loop.
+startup never reconcile lifecycle or firewall intent. The tool worker resumes
+explicitly queued installation requests, as described under Guest Tools below.
 
 ## 9. Firecracker Integration
 
@@ -792,7 +808,7 @@ do not import its unimplemented topology checks into current documentation.
 
 ## 12. Installer and Host Commands
 
-`Installer` is the only owner of apt installation, Firecracker artifact
+`Installer` is the owner of host apt installation, Firecracker artifact
 handling, downloads, and integrity-manifest publication.
 
 `Settings` detects and normalizes the host architecture at construction time
@@ -893,7 +909,7 @@ Nest. Both server entry points expose the UVM management routes below.
 | `GET` | `/diagnostics` | `diagnostics.py` | Host readiness checks |
 | `POST` | `/install` | `installation.py` | `{ "force": false }` body |
 | `GET` | `/vms` | `vms.py` | List persisted VM records and observed status |
-| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports` |
+| `POST` | `/vms` | `vms.py` | `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports`, `tools` |
 | `POST` | `/vms/actions` | `vms.py` | Fixed-ID snapshot bulk lifecycle action |
 | `GET` | `/vms/{identifier}` | `vms.py` | Inspect by exact ID, name, or guest IP |
 | `POST` | `/vms/{identifier}/start` | `vms.py` | Boot a retained private disk |
@@ -905,6 +921,10 @@ Nest. Both server entry points expose the UVM management routes below.
 | `POST` | `/vms/{vm_id}/ports/sync` | `vms.py` | Exact ID only; reapply managed policy |
 | `POST` | `/vms/{vm_id}/exec` | `vms.py` | Exact ID; `command`, strict `sudo`, bounded `timeout_seconds` |
 | `POST` | `/vms/{vm_id}/agent/install` | `vms.py` | Exact ID; install/upgrade the agent on a stopped disk |
+| `GET` | `/tools` | `tools.py` | Authenticated catalog and supported Python versions |
+| `GET` | `/vms/{vm_id}/tools` | `tools.py` | Cached inventory/history; `refresh=true` probes the running guest |
+| `POST` | `/vms/{vm_id}/tools/install` | `tools.py` | Exact ID; `tools`, `start_if_stopped`, `restart`, optional `request_id`; HTTP 202 |
+| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | `tools.py` | Exact VM-scoped installation job and bounded output |
 | `GET` | `/vms/{identifier}/logs` | `vms.py` | Bounded non-following log tail |
 | `GET` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Saved session list (Nest only) |
 | `POST` | `/vms/{vm_id}/sessions` | `nest/session_routes.py` | Launch a guest PTY; optional rows/cols in a JSON object (Nest only) |
@@ -1030,6 +1050,78 @@ device and uses CID 3. The process adapter retains both sockets while identity
 is uncertain and removes them only after confirmed VMM exit. IPv4 port policies
 do not govern this host-only management channel.
 
+### Guest Tools
+
+`tool_catalog.py` is the source of truth for catalog metadata, normalized
+`ToolSelection` values, and installer recipes. The API accepts catalog IDs and
+explicit Python minors; executable text, download URLs, paths, and other runtime
+versions are not request fields. There are ten tools and twelve selectable
+entries when Python 3.11/3.12/3.13 are expanded. Update publisher versions, URLs,
+and SHA-256 values together when changing a downloaded default.
+
+`ToolService` owns two bounded worker threads by default. The shared UVM server
+lifespan starts/stops them, so both Nest and the standalone API process the same
+queue. An exclusive, mode-0600 `$UVM_BASE/tool-worker.lock` permits one management
+server to own these workers per registry. Construction alone starts no threads.
+Queued requests are dispatched on server startup; prior in-flight jobs are
+marked interrupted with unknown outcomes after ownership has been acquired.
+
+`LifecycleService.queue_tools()` validates an exact VM ID and persists a request
+under the operation lock. Only one queued/active batch is allowed per VM.
+Identical active requests are deduplicated. Optional request IDs also identify
+completed jobs until they age out of the ten-job history; conflicting reuse is
+an error. `prepare_tool_install()` performs an explicitly requested start or
+restart under the normal lifecycle lock and returns the selected process identity.
+Creation writes its tool job after disk provisioning and before boot.
+
+Workers wait up to `Settings.tool_ready_timeout_s` (120 seconds) for the agent.
+Each tool has `Settings.tool_install_timeout_s` (1,200 seconds), bounded by the
+existing guest-exec protocol maximum. Readiness connection failures can be
+retried before submission; an uncertain installation result is never replayed.
+Each step passes `expected_identity` to `LifecycleService.execute()`, so a restart
+between steps cannot retarget the batch to a new VMM. The guest boot ID also
+detects in-guest reboots. Guest downloads and package waits hold no host operation
+lock. Shutdown closes tracked waiting sockets, preserves interruption results,
+and keeps worker ownership until outstanding host work has finished.
+
+`guest_tools.py` is a standalone stdlib-only Python 3.10+ payload. Its source and
+server-generated JSON recipes are shell-quoted into the bounded guest exec
+request. Importing it on the host performs no installation. In the guest it:
+
+- Checks Ubuntu 22.04 and the expected x86_64/ARM64 architecture before mutation.
+- Serializes installation through `/run/uvm-tools.lock`, including inherited
+  lock ownership during package commands. A previous disconnected job can finish
+  in the guest, and this lock prevents overlapping retries.
+- Uses non-interactive apt with bounded dpkg-lock waits and authenticated sources.
+  A pending marker permits a retry to finish its interrupted dpkg configuration.
+  PHP's repository key is checked against a pinned fingerprint and scoped with
+  `signed-by`.
+- Checks pinned hashes before executing/extracting downloads. Archive extraction
+  validates paths/links, file types, and size bounds. Versioned payloads live in
+  `/opt/uvm-tools`; executable links and login-shell profiles have an ownership
+  manifest that detects guest edits.
+- Checks command versions and companions as the configured guest account, including
+  npm/npx, Python pip/venv imports, and both Java tools. Python has versioned
+  executables; Conda uses per-user environments/caches. Get-started commands use
+  explicit POSIX shell hooks for activation in Nest's non-login shells.
+- Captures a bounded 16 KiB output tail and returns structured JSON when each
+  step finishes. An ordinary recipe failure allows later tools to be attempted.
+
+`ToolJobStore` keeps cached inventory and the last ten jobs in a mode-0600,
+schema-1 `tools.json` in the existing private VM runtime directory. Atomic
+replacement/fsync and short operation-lock transactions prevent races with
+deletion or stale lifecycle record saves. Store writes require a still-registered,
+owned VM directory and never recreate a deleted VM. VM deletion removes this
+history with the runtime. Full inventory refreshes update `checked_at`; step
+results also update individual tool observations. GET without `refresh=true`
+returns cached observations, including the `stale` flag.
+
+The dashboard's `tools.js` owns create selections, the VM Tools tab, job polling,
+history, retries, and output rendering. Active jobs poll every 1.5 seconds while
+visible; idle views poll cached status every five seconds. New selections use the
+shared catalog. Navigation/logout invalidate in-flight UI responses, and tool
+output is escaped before rendering.
+
 ### Nest Terminal Sessions
 
 Nest owns session HTTP/WebSocket routes and history. UVM owns the process-pinned
@@ -1198,6 +1290,16 @@ This checks launch/input, reload/reconnect, session switching, per-VM isolation,
 closed-session replay, rename persistence, active/ended deletion, cancellation,
 multi-tab updates, and the mobile layout. It does not boot a real microVM.
 
+With the same browser dependencies, `node tests/browser_tools.cjs` checks creation
+selections, persisted background jobs across reload, installed-tool detection,
+partial failure/retry, escaped output, history, stopped-VM startup, per-VM isolation,
+multi-select installation, and mobile layout. Its fixture uses the real API,
+worker threads, and atomic job store with simulated guest execution.
+`tests/test_tool_*.py` cover validation, architecture-specific payloads, checksum
+failure, archive traversal, exact-ID targeting, idempotency, recovery, and shutdown.
+The opt-in real-VM catalog/runtime smoke test is documented in
+[USAGE.md](USAGE.md#18-verification); run it on disposable x86_64 and ARM64 VMs.
+
 When `nft` and `conntrack` are installed, command-parser tests append an invalid
 final command/option so parsing is checked without executing any mutation.
 These checks supplement fixture-based readback and cleanup tests; they do not

README.md

@@ -310,7 +310,7 @@ and `/dashboard` routes belong to the Nest server.
 
 The Nest dashboard includes a fleet overview, searchable machine collection,
 private network and port-policy view, host diagnostics, and machine provisioning.
-Open a machine to manage its lifecycle, launch terminal sessions, edit or sync
+Open a machine to manage its lifecycle, install developer tools, launch terminal sessions, edit or sync
 its port rules, copy its SSH command, or read recent logs. Select machines in the
 collection for bulk operations.
 Use `Ctrl+K` / `⌘K` (or `/`) to find a machine or page. Fleet data refreshes every
@@ -330,7 +330,7 @@ The server exposes these routes:
 | `GET` | `/diagnostics` | Run host readiness checks for the dashboard/system view |
 | `POST` | `/install` | Install or refresh host assets; body: `{ "force": false }` |
 | `GET` | `/vms` | List VM records and observed status |
-| `POST` | `/vms` | Create; supports `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports` |
+| `POST` | `/vms` | Create; supports `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`, `ports`, `no_default_ports`, `tools` |
 | `POST` | `/vms/actions` | Run a fixed snapshot of selected or all VM lifecycle actions |
 | `GET` | `/vms/{identifier}` | Inspect by exact ID, name, or guest IP |
 | `POST` | `/vms/{identifier}/start` | Boot a retained private disk |
@@ -342,6 +342,10 @@ The server exposes these routes:
 | `POST` | `/vms/{vm_id}/ports/sync` | Reapply a managed private port policy for an exact VM ID |
 | `POST` | `/vms/{vm_id}/exec` | Execute a command/script inside a running VM over vsock, with optional root privileges |
 | `POST` | `/vms/{vm_id}/agent/install` | Install/upgrade the execution agent on a stopped VM's private disk |
+| `GET` | `/tools` | Guest tool catalog and supported Python versions |
+| `GET` | `/vms/{vm_id}/tools` | Tool inventory and installation history; `refresh=true` probes the running guest |
+| `POST` | `/vms/{vm_id}/tools/install` | Queue selected guest tools; HTTP 202 with a job ID |
+| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | Per-tool progress, results, and bounded installation output |
 | `GET` | `/vms/{identifier}/logs` | Read a bounded, non-following tail of a VM log |
 | `GET` | `/vms/{vm_id}/sessions` | List this VM's saved sessions (Nest) |
 | `POST` | `/vms/{vm_id}/sessions` | Launch a guest terminal; body: `{}` or `{"rows":24,"cols":80}` (Nest) |
@@ -365,7 +369,47 @@ Log reads use an optional `tail` query parameter from `0` to `1000`, default
 Bulk requests use an action (`start`, `stop`, `restart`, `delete`, or
 `ports-sync`) plus either `identifiers` or `all_vms: true`; they return a result
 for each fixed target and do not prompt for confirmation. Interactive SSH
-remains CLI-only. The API and dashboard are not background reconcilers.
+remains CLI-only. VM lifecycle changes are explicit; accepted tool installation
+jobs are processed in the background.
+
+### One-click Guest Tools
+
+Choose tools under **New machine → Pick your tools** to install them automatically
+after boot. For an existing VM, open **Machines → select a VM → Tools** and click
+**Install**, or select several tools and click **Install selected**. A stopped VM
+offers **Start & install**. Installation works over vsock with SSH disabled.
+
+The initial catalog targets Ubuntu 22.04 guests on x86_64 and ARM64:
+
+| Tool | Default |
+|---|---|
+| Git, wget, curl, htop | Ubuntu packages |
+| Node.js | 24 LTS (24.21.0), including npm and npx |
+| Python | 3.11, 3.12, and 3.13 with pip and venv; versions can coexist |
+| JDK | OpenJDK 21 LTS, including `java` and `javac` |
+| PHP | 8.5 CLI from the signed Ondřej Surý PHP PPA |
+| Conda | Miniforge 26.5.3-0, using conda-forge |
+| micromamba | 2.9.0 |
+
+Node, Python, Miniforge, and micromamba downloads use pinned release checksums.
+Python uses Astral's standalone distributions, exposed as `python3.11`,
+`python3.12`, and `python3.13`. The guest's OS Python continues serving the agent.
+Downloaded tools live under `/opt/uvm-tools`, with executable links in
+`/usr/local/bin`. Conda environments and caches are writable by the configured
+guest user. **Get started** shows commands, including shell initialization for
+Conda/micromamba activation in Nest's non-login terminals.
+
+Jobs run independently of the browser and show progress and captured output per
+tool. Already available tools are verified and skipped. A failed tool can be
+retried on the created VM. Guest internet access is required for downloads.
+Restart the management server to load the feature; older running VMs may need
+**Restart & install** to apply the current guest-agent setup.
+
+The most recent ten jobs and cached inventory are stored in the VM's private
+`tools.json` file. Queued requests resume when the server starts. An interrupted
+in-flight installation has an uncertain outcome and may finish inside the guest;
+use **Check installed tools** before retrying. See
+[API examples and recovery](USAGE.md#install-guest-tools) for details.
 
 ### MicroVM Terminal Sessions
 

USAGE.md

@@ -920,6 +920,10 @@ curl --cacert /path/to/ca.pem \
 | `POST` | `/vms/{vm_id}/ports/sync` | Reapply the policy for an exact VM ID |
 | `POST` | `/vms/{vm_id}/exec` | Execute a guest command/script over vsock, with optional root privileges |
 | `POST` | `/vms/{vm_id}/agent/install` | Install/upgrade the agent on a stopped VM's disk |
+| `GET` | `/tools` | Authenticated tool catalog and Python version options |
+| `GET` | `/vms/{vm_id}/tools` | Cached tool inventory/history; `refresh=true` checks the running guest |
+| `POST` | `/vms/{vm_id}/tools/install` | Queue tool installation; HTTP 202 with a job ID |
+| `GET` | `/vms/{vm_id}/tools/jobs/{job_id}` | Installation progress, results, and captured output |
 | `GET` | `/vms/{identifier}/logs` | Read a bounded, non-following log tail |
 | `GET` | `/vms/{vm_id}/sessions` | List saved sessions for an exact VM ID (Nest) |
 | `POST` | `/vms/{vm_id}/sessions` | Launch a terminal, with `{}` or optional `rows`/`cols` (Nest) |
@@ -973,7 +977,7 @@ curl -X POST \
 ```
 
 Request fields are `cpu`, `ram`, `storage`, `guest_ip`, `username`, `password`, `name`,
-`ports`, and `no_default_ports`. The name has the same validation as the CLI.
+`ports`, `no_default_ports`, and `tools`. The name has the same validation as the CLI.
 `ports` is an array of objects (or null/omitted), not a CLI colon string.
 Each object has a strict integer `port` from 1 to 65535 and an optional,
 case-sensitive `protocol` of `TCP` (default), `HTTP`, or `HTTPS`.
@@ -988,6 +992,106 @@ The response never includes the password. Omitting credentials still uses
 labels return HTTP `422`. Example passwords above are placeholders; real
 secrets in curl arguments can enter shell history and process listings.
 
+### Install Guest Tools
+
+The dashboard provides tool checkboxes during creation and a **Tools** tab on
+each VM. Choose **Install** for a single tool or **Install selected** for a batch.
+**Start & install** also boots a stopped VM. The catalog covers Ubuntu 22.04 on
+x86_64 and ARM64, with guest internet access for package downloads.
+
+Use the same API selections during creation:
+
+```sh
+curl -X POST \
+  -H "X-UVM-Token: $UVM_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"dev-tools","username":"developer","password":"replace-me","tools":[{"id":"git"},{"id":"nodejs"},{"id":"python","version":"3.12"}]}' \
+  "$UVM_API_URL/vms"
+```
+
+The VM response is HTTP 201 after creation. Selected tools install asynchronously;
+read the returned VM's `/tools` endpoint for its job. Omitted `tools` defaults to
+an empty list. An explicit null is rejected.
+
+For an existing VM, use its exact ID from create/list:
+
+```sh
+export VM_ID='<exact-vm-id>'
+curl -H "X-UVM-Token: $UVM_API_TOKEN" "$UVM_API_URL/tools"
+
+curl -X POST \
+  -H "X-UVM-Token: $UVM_API_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"tools":[{"id":"jdk"},{"id":"conda"},{"id":"micromamba"}],"start_if_stopped":true,"request_id":"dev-tools-1"}' \
+  "$UVM_API_URL/vms/$VM_ID/tools/install"
+
+# Use the job ID from the HTTP 202 response.
+export JOB_ID='<tools-job-id>'
+curl -H "X-UVM-Token: $UVM_API_TOKEN" \
+  "$UVM_API_URL/vms/$VM_ID/tools/jobs/$JOB_ID"
+curl -H "X-UVM-Token: $UVM_API_TOKEN" \
+  "$UVM_API_URL/vms/$VM_ID/tools?refresh=true"
+```
+
+Selections are objects with `id` and an optional `version`. IDs are `git`,
+`nodejs`, `python`, `jdk`, `wget`, `curl`, `htop`, `php`, `conda`, and
+`micromamba`. Python requires `version: "3.11"`, `"3.12"`, or `"3.13"`;
+the other tools use the curated catalog default and accept no version override.
+Up to twelve selections are accepted, with identical selections deduplicated.
+Multiple Python versions can be installed together.
+
+Install requests require a nonempty `tools` list. `start_if_stopped` and `restart`
+are strict booleans, both default false. Set `restart: true` to restart first and
+refresh guest-agent setup. In the dashboard, the **Restart & install** recovery
+button appears when agent readiness fails. Unknown tools/options return 422;
+unavailable VM states or a different active install return 409.
+
+`request_id` is an optional idempotency key of 1–80 letters, digits, underscores,
+or hyphens. Reusing it with the same request returns the retained job; different
+selections/options return 409. Use a new key for an intentional retry. Idempotency
+keys last as long as their job in the ten-job history. One batch per VM is active
+at a time, with two VM workers by default and sequential tools within a batch.
+
+Jobs move through `queued`, `starting`, `waiting`, and `installing`, then
+`succeeded`, `failed`, or `interrupted`. Each item reports `installed`,
+`already_installed`, `failed`, `interrupted`, or `skipped`, plus its detected
+version, error, bounded output, and truncation flag. Ordinary recipe failures
+allow subsequent tools to run; uncertain execution outcomes end that batch.
+Output is captured as a 16 KiB tail per tool and returned when the step finishes.
+Default limits are 120 seconds for agent readiness and 1,200 seconds per tool.
+
+`GET /vms/{vm_id}/tools` returns cached inventory, the latest job, and summaries
+of the last ten jobs. `refresh=true` probes all tools in a running VM and updates
+`checked_at`; it does not install software or start a stopped VM. `stale` marks
+old or stopped-VM observations. Installed binaries and their version checks run
+as the configured guest user. Matching installed runtimes are accepted, including
+tools installed outside Nest. The install operation is not a general updater.
+
+Tool jobs survive browser navigation, reload, and logout. The active management
+server owns the queue through `$UVM_BASE/tool-worker.lock`. After server restart,
+queued work resumes; in-flight jobs become interrupted with
+`outcome_unknown: true`. The last command may still finish inside the guest.
+Refresh inventory before retrying; a guest-side lock blocks overlapping installs.
+Stopping/restarting the VM also interrupts a running batch. Successful tools and
+the VM's private disk remain available. Deleting the VM removes its tool history.
+
+Use versioned Python commands, for example `python3.12 -m venv .venv`. For Conda
+or micromamba activation in a Nest terminal, initialize the current POSIX shell:
+
+```sh
+eval "$(conda shell.posix hook)"
+conda create -n myenv python=3.13
+conda activate myenv
+
+# In another shell, for micromamba:
+eval "$(micromamba shell hook --shell posix)"
+micromamba create -n myenv -c conda-forge python=3.13
+micromamba activate myenv
+```
+
+The installer also adds login-shell profiles under `/etc/profile.d`. See
+[the catalog](README.md#one-click-guest-tools) for default versions and sources.
+
 ### Start, Restart, Stop, and Delete
 
 ```sh
@@ -1597,6 +1701,21 @@ exercise real child processes with fake cgroup scopes, local Unix-socket protoco
 fixtures, and temporary ext4 upgrades; they do not prove the real vsock/cgroup
 path inside a VM.
 
+The tools smoke test installs the full catalog in a prepared **disposable Ubuntu
+22.04 VM with a non-root login and no open TCP ports**. It downloads packages,
+checks all runtimes, compiles Java, creates Python venvs, and creates/activates
+temporary Conda and micromamba environments as the guest user. Run it separately
+on x86_64 and ARM64 hosts:
+
+```sh
+UVM_TOOLS_INTEGRATION=1 UVM_TOOLS_VM_ID='<disposable-vm-id>' \
+  python3 -B -m unittest discover -s tests/integration -p test_guest_tools.py -v
+```
+
+Set `UVM_API_URL` and `UVM_API_TOKEN` as above. The test is opt-in and was not run
+against a real microVM during this implementation. Automated API and browser
+checks use simulated guest execution and real job persistence/workers.
+
 ## 19. Get Help
 
 ```sh

nest/dashboard/static/js/tools.js

@@ -212,7 +212,7 @@
     canInstall() {
       const status = this.vm?.observed_status || this.vm?.status;
       return ['running', 'stopped'].includes(status) && !this.vm.operation && !this.vm.pending_resize
-        && (status !== 'stopped' || this.vm.ports !== null);
+        && (status !== 'stopped' || this.vm.ports !== null) && this.data?.supported !== false;
     }
 
     render() {
@@ -239,7 +239,7 @@
           <p>${this.esc(tool.description)}</p><small class="tool-source">${this.esc(tool.source)}</small>
           <div class="tool-card-footer"><span class="tool-state ${installed ? 'tool-state-good' : failed ? 'tool-state-error' : installing ? 'tool-state-active' : ''}">${this.esc(state)}</span>
           <button class="button button-small" type="button" data-install-tool="${this.esc(tool.key)}" ${disabled || installed ? 'disabled' : ''}>${installed ? 'Installed' : this.esc(caption)}</button></div>
-          ${installed && found.usage ? `<details class="tool-usage"><summary>Get started</summary><code>${this.esc(found.usage)}</code><small>Open a new session for shell activation commands.</small></details>` : ''}</article>`;
+          ${installed && found.usage ? `<details class="tool-usage"><summary>Get started</summary><code>${this.esc(found.usage)}</code><small>Run these commands in this machine’s terminal.</small></details>` : ''}</article>`;
       }).join(''));
       if (focused) document.querySelector(`[data-tool-select="${CSS.escape(focused)}"]`)?.focus({ preventScroll: true });
       $('#tools-selected-count').textContent = `${this.selected.size} selected`;
@@ -252,7 +252,7 @@
       $('#tools-note').textContent = this.data?.guest_busy
         ? 'An installation is still running inside the guest. Its status will be checked again shortly.'
         : stopped ? 'Start & install boots this machine and installs your selected tools. Versions shown are last observed.'
-          : this.data?.checked_at ? `Last checked ${this.formatDate(this.data.checked_at)}${this.data.stale ? ' · cached observations' : ''}. Open a new session after installing to load shell integration.`
+          : this.data?.checked_at ? `Last checked ${this.formatDate(this.data.checked_at)}${this.data.stale ? ' · cached observations' : ''}. Use Get started for commands and shell initialization.`
             : 'Tools are installed directly inside this VM. Guest internet access is required for downloads.';
       this.renderJob();
     }

tests/browser_tool_server.py

@@ -0,0 +1,130 @@
+"""Browser fixture with the real API/job workers and simulated guest execution."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shlex
+import threading
+from dataclasses import replace
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import patch
+
+import uvicorn
+
+from nest.server import create_app
+from tests.browser_session_server import BrowserLifecycle
+from uvm.config import Settings
+from uvm.domain import new_vm_id
+from uvm.errors import ConflictError, GuestExecutionError
+from uvm.guest_exec import ExecResult
+from uvm.state import StateStore
+from uvm.tool_store import ToolJobStore
+from uvm.tools import ToolService
+
+
+class FixtureConnection:
+    def __init__(self):
+        self.closed = threading.Event()
+
+    def close(self):
+        self.closed.set()
+
+
+class ToolBrowserLifecycle(BrowserLifecycle):
+    def __init__(self, settings, state, *, delay=1.2):
+        super().__init__()
+        self.settings, self.state, self.delay = settings, state, delay
+        self.tools = ToolJobStore(settings, state)
+        self.installed = {}
+        self.attempts = {}
+        for index, vm in enumerate(self.records.values()):
+            vm.runtime_owned = True
+            vm.pid, vm.process_start_time, vm.boot_id = index + 100, "1", "fixture-boot"
+            settings.vm_dir(vm.id).mkdir(parents=True, mode=0o700, exist_ok=True)
+        with state.transaction() as document:
+            document.vms.update(self.records)
+
+    def create(self, spec):
+        vm = replace(self.vm, id=new_vm_id(), name=spec.name, username=spec.username, cpu=spec.cpu,
+                     ram_mib=spec.ram_mib, storage_mib=spec.storage_mib, ports=spec.ports)
+        with self.state.operation_lock():
+            self.settings.vm_dir(vm.id).mkdir(mode=0o700)
+            self.records[vm.id] = vm
+            with self.state.transaction() as document:
+                document.vms[vm.id] = vm
+            if spec.tools:
+                self.tools.enqueue_locked(vm.id, spec.tools, source="create")
+        return vm
+
+    def queue_tools(self, vm_id, selections, **options):
+        self.state.get(vm_id)
+        with self.state.operation_lock():
+            return self.tools.enqueue_locked(vm_id, selections, **options)
+
+    def prepare_tool_install(self, vm_id, *, start_if_stopped=False, restart=False):
+        with self.state.operation_lock():
+            vm = self.records[vm_id]
+            if vm.status == "stopped" and not (start_if_stopped or restart):
+                raise ConflictError("start the VM first")
+            vm.status = "running"
+            with self.state.transaction() as document:
+                document.vms[vm.id] = vm
+            return vm
+
+    def execute(self, vm_id, command, *, expected_identity=None, on_connect=None, **_options):
+        self.state.get(vm_id)
+        payload = json.loads(shlex.split(command)[-1])
+        connection = FixtureConnection()
+        if on_connect:
+            on_connect(connection)
+        installed = self.installed.setdefault(vm_id, {"curl"})
+
+        def observation(tool):
+            versions = {"nodejs": "24.21.0", "jdk": "21.0.12", "php": "8.5.11", "conda": "26.5.3", "micromamba": "2.9.0"}
+            version = tool["version"] + ".15" if tool["id"] == "python" else versions.get(tool["id"], "2.0.0")
+            return {"key": tool["key"], "id": tool["id"], "version": tool["version"], "installed": tool["key"] in installed,
+                    "detected_version": version if tool["key"] in installed else None, "error": None, "usage": tool["usage"]}
+
+        if payload["action"] == "probe":
+            result = {"ok": True, "busy": False, "boot_id": "guest-fixture", "supported": True,
+                      "tools": [observation(tool) for tool in payload["tools"]]}
+        else:
+            if connection.closed.wait(self.delay):
+                raise GuestExecutionError("fixture connection closed", outcome_unknown=True)
+            tool = payload["tools"][0]
+            key = (vm_id, tool["key"])
+            self.attempts[key] = self.attempts.get(key, 0) + 1
+            if tool["id"] == "php" and self.attempts[key] == 1:
+                result = {"ok": False, "error": "Simulated mirror failure", "output": "<script>bad()</script>\nTry again later."}
+            else:
+                installed.add(tool["key"])
+                result = {"ok": True, "status": "installed", "tool": observation(tool),
+                          "output": f"Verified and installed {tool['key']} inside the guest."}
+        return ExecResult(0, json.dumps(result), "", False, False)
+
+
+def fixture_application(base, *, delay=1.2):
+    settings = replace(Settings(), base=base, api_token="browser-test-token", tool_workers=2)
+    state = StateStore(settings)
+    lifecycle = ToolBrowserLifecycle(settings, state, delay=delay)
+    return SimpleNamespace(settings=settings, state_store=state, lifecycle=lifecycle,
+                           tools=ToolService(settings, state, lifecycle))
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", type=Path, required=True)
+    parser.add_argument("--port", type=int, required=True)
+    args = parser.parse_args()
+    application = fixture_application(args.base)
+    with patch("uvm.tools.require_root"):
+        api = create_app(application, host="127.0.0.1")
+        server = uvicorn.Server(uvicorn.Config(api, host="127.0.0.1", port=args.port, log_level="warning"))
+        api.state.uvicorn_server = server
+        server.run()
+
+
+if __name__ == "__main__":
+    main()

tests/browser_tools.cjs

@@ -0,0 +1,106 @@
+// Optional browser integration: real API/job persistence, simulated guest commands.
+const assert = require('node:assert/strict');
+const { spawn } = require('node:child_process');
+const { mkdtemp, rm } = require('node:fs/promises');
+const os = require('node:os');
+const path = require('node:path');
+const { chromium } = require('playwright');
+
+(async () => {
+  const temporary = await mkdtemp(path.join(os.tmpdir(), 'nest-tool-browser-'));
+  const port = process.env.NEST_BROWSER_TEST_PORT || '18766';
+  const origin = `http://127.0.0.1:${port}`;
+  const server = spawn(process.env.NEST_TEST_PYTHON || 'python3',
+    ['-B', '-m', 'tests.browser_tool_server', '--base', temporary, '--port', port],
+    { cwd: path.resolve(__dirname, '..'), stdio: ['ignore', 'pipe', 'pipe'] });
+  let logs = '';
+  server.stdout.on('data', (data) => { logs = (logs + data).slice(-16000); });
+  server.stderr.on('data', (data) => { logs = (logs + data).slice(-16000); });
+  let browser;
+  let page;
+  try {
+    let ready = false;
+    for (let attempt = 0; attempt < 100; attempt++) {
+      if (server.exitCode !== null) throw new Error(`Fixture server exited: ${logs}`);
+      try { if ((await fetch(`${origin}/health`)).ok) { ready = true; break; } } catch {}
+      await new Promise((resolve) => setTimeout(resolve, 100));
+    }
+    assert.ok(ready, logs);
+    browser = await chromium.launch({ headless: true });
+    page = await browser.newPage({ viewport: { width: 1440, height: 1100 } });
+    const errors = [];
+    page.on('pageerror', (error) => errors.push(error.message));
+    const installed = (key) => page.waitForFunction((value) =>
+      document.querySelector(`[data-install-tool="${value}"]`)?.textContent === 'Installed', key);
+    await page.goto(`${origin}/dashboard#create`);
+    await page.locator('#token-input').fill('browser-test-token');
+    await page.locator('#auth-btn').click();
+    await page.locator('[data-create-tool="python-3.12"]').check();
+    await page.locator('[data-create-tool="git"]').check();
+    await page.locator('[data-create-tool="nodejs"]').check();
+    assert.equal(await page.locator('[data-create-tool]').count(), 12);
+    assert.match(await page.locator('#create-summary-details').textContent(), /Git, Node.js, Python 3.12/);
+    const layout = await page.evaluate(() => ({
+      parent: document.querySelector('.create-tools-section').parentElement.className,
+      children: document.querySelector('#create-form').children.length,
+    }));
+    assert.equal(layout.parent, 'create-fields');
+    assert.equal(layout.children, 2);
+    await page.locator('#create-name').fill('tool-workspace');
+    await page.locator('#create-submit').click();
+    await page.waitForURL((url) => /#vm\/vm-[a-f0-9]+\/tools$/.test(url.hash));
+    const createdUrl = page.url();
+    await page.locator('#tools-job').waitFor({ state: 'visible' });
+    // A reload can occur while the server-owned worker is still installing.
+    await page.reload();
+    await page.waitForFunction(() => document.querySelector('#tools-job-title').textContent === 'Complete');
+    await installed('curl');
+    assert.equal(await page.locator('.tool-installed').count(), 4); // Includes image-provided curl.
+    assert.ok(await page.locator('[data-install-tool="python-3.12"]').isDisabled());
+    await page.locator('#tools-job-items details').first().locator('summary').click();
+    assert.match(await page.locator('#tools-job-items').textContent(), /Verified and installed git/);
+    await page.locator('[data-install-tool="php"]').click();
+    await page.waitForFunction(() => document.querySelector('#tools-job-title').textContent === 'Failed');
+    await page.locator('#tools-job-items details[data-output="php"] summary').click();
+    assert.match(await page.locator('.tool-output').textContent(), /<script>bad/);
+    assert.equal(await page.locator('#tools-job-items script').count(), 0);
+    await page.locator('#tools-retry').click();
+    await page.waitForFunction(() => document.querySelector('#tools-job-title').textContent === 'Complete');
+    await installed('php');
+    assert.ok(await page.locator('[data-install-tool="php"]').isDisabled());
+    assert.equal(await page.locator('#tools-job-select option').count(), 3);
+
+    // The same click boots a stopped VM before installing its selected tool.
+    await page.goto(`${origin}/dashboard#vm/vm-other/tools`);
+    await page.locator('[data-install-tool="htop"]').waitFor();
+    assert.equal(await page.locator('[data-install-tool="htop"]').textContent(), 'Start & install');
+    await page.locator('[data-install-tool="htop"]').click();
+    await installed('htop');
+    await page.waitForFunction(() => document.querySelector('#tools-job-title').textContent === 'Complete');
+    assert.ok(await page.locator('[data-install-tool="htop"]').isDisabled());
+    assert.ok(!(await page.locator('#tools-job-items').textContent()).includes('python-3.12'));
+
+    await page.goto(createdUrl);
+    await page.locator('[data-tool-select="wget"]').check();
+    await page.locator('[data-tool-select="micromamba"]').check();
+    await page.locator('#tools-install-selected').click();
+    await installed('wget');
+    await installed('micromamba');
+    await page.waitForFunction(() => document.querySelector('#tools-job-title').textContent === 'Complete');
+    await page.setViewportSize({ width: 390, height: 844 });
+    await page.waitForTimeout(250);
+    assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth), 'mobile layout overflows');
+    if (process.env.NEST_BROWSER_SCREENSHOT) await page.screenshot({ path: process.env.NEST_BROWSER_SCREENSHOT, fullPage: true });
+    assert.deepEqual(errors, []);
+    console.log('Browser tools passed: creation, background/reload, detection, failure/retry, safe output, history, Start & install, VM isolation, multi-select, and mobile layout.');
+  } catch (error) {
+    if (page) console.error('Tool UI:', await page.locator('#tools-error').textContent(), await page.locator('#tools-job-message').textContent());
+    console.error(logs);
+    throw error;
+  } finally {
+    await browser?.close();
+    server.kill('SIGTERM');
+    if (server.exitCode === null) await new Promise((resolve) => server.once('exit', resolve));
+    await rm(temporary, { recursive: true, force: true });
+  }
+})().catch((error) => { console.error(error); process.exitCode = 1; });

tests/integration/test_guest_tools.py

@@ -0,0 +1,96 @@
+"""Opt-in tool installation smoke test against a prepared, disposable microVM."""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+import unittest
+import uuid
+from urllib.parse import quote
+from urllib.request import Request, urlopen
+
+
+@unittest.skipUnless(os.environ.get("UVM_TOOLS_INTEGRATION") == "1",
+                     "set UVM_TOOLS_INTEGRATION=1 on an isolated disposable VM host")
+class GuestToolsIntegrationTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.api_url = os.environ.get("UVM_API_URL", "http://127.0.0.1:8000").rstrip("/")
+        cls.token = os.environ["UVM_API_TOKEN"]
+        cls.vm_id = os.environ["UVM_TOOLS_VM_ID"]
+        cls.path = "/vms/" + quote(cls.vm_id, safe="")
+        cls.vm = cls.request("GET", cls.path)
+        if (cls.vm["id"] != cls.vm_id or cls.vm["observed_status"] != "running"
+                or cls.vm["username"] == "root" or cls.vm["ports"] != []):
+            raise RuntimeError("provide a disposable running Ubuntu 22.04 VM with a non-root user and no open TCP ports")
+
+    @classmethod
+    def request(cls, method, path, body=None, *, timeout=120):
+        request = Request(cls.api_url + path, method=method,
+                          data=json.dumps(body).encode() if body is not None else None,
+                          headers={"X-UVM-Token": cls.token, "Content-Type": "application/json"})
+        with urlopen(request, timeout=timeout) as response:
+            return json.load(response)
+
+    def test_install_catalog_and_use_runtimes_as_the_guest_user(self):
+        catalog = self.request("GET", "/tools")
+        selections = [{"id": tool["id"], "version": version} for tool in catalog for version in tool["versions"] or [None]]
+        body = {"tools": selections, "request_id": str(uuid.uuid4())}
+        job = self.request("POST", self.path + "/tools/install", body)
+        deadline = time.monotonic() + 3600
+        while job["status"] in {"queued", "starting", "waiting", "installing"} and time.monotonic() < deadline:
+            time.sleep(2)
+            job = self.request("GET", self.path + "/tools/jobs/" + job["id"])
+        self.assertEqual(job["status"], "succeeded", json.dumps(job, indent=2))
+        self.assertEqual(self.request("POST", self.path + "/tools/install", body)["id"], job["id"])
+        inventory = self.request("GET", self.path + "/tools?refresh=true")
+        self.assertEqual(len(inventory["inventory"]), 12)
+        self.assertTrue(all(item["installed"] for item in inventory["inventory"]), inventory)
+
+        script = r'''set -eu
+test "$(id -u)" != 0
+git --version
+node -e 'if (1 + 1 !== 2) process.exit(1)'
+npm --version
+wget --version >/dev/null
+curl --version >/dev/null
+htop --version
+php -r 'if (2 + 2 !== 4) exit(1);'
+work=$(mktemp -d "$HOME/nest-tools-smoke.XXXXXX")
+name=$(basename "$work")
+export MAMBA_ROOT_PREFIX="$HOME/micromamba"
+cleanup() {
+    conda env remove -n "$name" -y >/dev/null 2>&1 || true
+    micromamba env remove -n "$name" -y >/dev/null 2>&1 || true
+    rm -rf -- "$work"
+}
+trap cleanup EXIT
+printf 'class Hello { public static void main(String[] args) { System.out.println("ok"); } }\n' > "$work/Hello.java"
+javac "$work/Hello.java"
+test "$(java -cp "$work" Hello)" = ok
+for version in 3.11 3.12 3.13; do
+    "python$version" -m venv "$work/venv-$version"
+    "$work/venv-$version/bin/python" -c 'import ssl, sqlite3, sys; assert sys.prefix != sys.base_prefix'
+    "$work/venv-$version/bin/python" -m pip --version
+done
+conda create -y -n "$name" python=3.13
+conda run -n "$name" python -c 'import sys; assert sys.version_info[:2] == (3, 13)'
+eval "$(conda shell.posix hook)"
+conda activate "$name"
+python -c 'import sys; assert sys.version_info[:2] == (3, 13)'
+conda deactivate
+micromamba create -y -n "$name" -c conda-forge python=3.12
+micromamba run -n "$name" python -c 'import sys; assert sys.version_info[:2] == (3, 12)'
+eval "$(micromamba shell hook --shell posix)"
+micromamba activate "$name"
+python -c 'import sys; assert sys.version_info[:2] == (3, 12)'
+micromamba deactivate
+'''
+        result = self.request("POST", self.path + "/exec",
+                              {"command": script, "sudo": False, "timeout_seconds": 1200}, timeout=1220)
+        self.assertFalse(result["timed_out"], result)
+        self.assertEqual(result["exit_code"], 0, result)
+        after = self.request("GET", self.path)
+        for key in ("id", "guest_ip", "mac", "username", "ports", "cpu", "ram_mib", "storage_mib"):
+            self.assertEqual(after[key], self.vm[key])

tests/test_tool_api.py

@@ -11,7 +11,6 @@ from tests.test_server import FASTAPI_AVAILABLE, FakeLifecycle
 from uvm.config import Settings
 from uvm.server import create_api
 from uvm.state import StateStore
-from uvm.tool_store import ToolJobStore
 from uvm.tools import ToolService
 
 if FASTAPI_AVAILABLE:

tests/test_tool_catalog.py

@@ -1,15 +1,15 @@
 from __future__ import annotations
 
+import ast
 import hashlib
 import io
-import os
 import shlex
 import tarfile
 import tempfile
 import unittest
 from pathlib import Path
 from types import SimpleNamespace
-from unittest.mock import Mock, patch
+from unittest.mock import patch
 
 from uvm import guest_tools
 from uvm.errors import ValidationError
@@ -28,6 +28,7 @@ class ToolCatalogTests(unittest.TestCase):
             argv = shlex.split(command)
             self.assertEqual(argv[:3], ["/usr/bin/python3", "-I", "-c"])
             compile(argv[3], "guest-tools", "exec")
+            ast.parse(argv[3], feature_version=(3, 10))
             for item in recipes:
                 if "url" in item:
                     self.assertTrue(item["url"].startswith("https://"))

tests/test_tool_flow.py

@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+import tempfile
+import time
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+from tests.test_server import FASTAPI_AVAILABLE
+
+if FASTAPI_AVAILABLE:
+    from fastapi.testclient import TestClient
+    from tests.browser_tool_server import fixture_application
+    from uvm.server import create_api
+
+
+@unittest.skipUnless(FASTAPI_AVAILABLE, "FastAPI and httpx are required")
+class ToolFlowTests(unittest.TestCase):
+    def setUp(self):
+        temporary = self.enterContext(tempfile.TemporaryDirectory())
+        self.application = fixture_application(Path(temporary), delay=0.01)
+        self.enterContext(patch("uvm.tools.require_root"))
+        self.headers = {"X-UVM-Token": "browser-test-token"}
+
+    def wait_for(self, condition):
+        deadline = time.monotonic() + 5
+        while time.monotonic() < deadline:
+            if value := condition():
+                return value
+            time.sleep(0.01)
+        self.fail("background tool worker did not reach the expected state")
+
+    def test_create_and_later_install_use_real_workers_and_durable_job_responses(self):
+        with TestClient(create_api(self.application, host="127.0.0.1")) as client:
+            response = client.post("/vms", headers=self.headers, json={"name": "tools-vm", "tools": [
+                {"id": "git"}, {"id": "python", "version": "3.12"},
+            ]})
+            self.assertEqual(response.status_code, 201, response.text)
+            vm_id = response.json()["id"]
+            self.wait_for(lambda: self.application.tools.store.read(vm_id)["jobs"][-1]["status"] == "succeeded")
+            snapshot = client.get(f"/vms/{vm_id}/tools?refresh=true", headers=self.headers)
+            self.assertEqual(snapshot.status_code, 200, snapshot.text)
+            installed = {item["key"] for item in snapshot.json()["inventory"] if item["installed"]}
+            self.assertEqual(installed, {"git", "python-3.12", "curl"})
+            self.assertEqual(snapshot.json()["latest_job"]["source"], "create")
+            later = client.post("/vms/vm-other/tools/install", headers=self.headers,
+                                json={"tools": [{"id": "htop"}], "start_if_stopped": True})
+            self.assertEqual(later.status_code, 202, later.text)
+            job_id = later.json()["id"]
+            self.wait_for(lambda: self.application.tools.store.get_job("vm-other", job_id)["status"] == "succeeded")
+            self.assertEqual(self.application.state_store.get("vm-other").status, "running")
+            self.assertEqual(client.get(f"/vms/{vm_id}/tools/jobs/{job_id}", headers=self.headers).status_code, 404)
+
+    def test_queued_jobs_are_dispatched_on_server_start(self):
+        with self.application.state_store.operation_lock():
+            job = self.application.tools.store.enqueue_locked("vm-test", [{"id": "git"}])
+        with TestClient(create_api(self.application, host="127.0.0.1")):
+            self.wait_for(lambda: self.application.tools.store.get_job("vm-test", job["id"])["status"] == "succeeded")
+        self.assertEqual(self.application.lifecycle.attempts[("vm-test", "git")], 1)
+
+    def test_shutdown_wakes_command_waiter_and_keeps_an_interrupted_job(self):
+        self.application.lifecycle.delay = 30
+        with TestClient(create_api(self.application, host="127.0.0.1")) as client:
+            response = client.post("/vms/vm-test/tools/install", headers=self.headers, json={"tools": [{"id": "git"}]})
+            job_id = response.json()["id"]
+            self.wait_for(lambda: self.application.tools.store.get_job("vm-test", job_id)["status"] == "installing"
+                          and bool(self.application.tools._sessions))
+            started = time.monotonic()
+        self.assertLess(time.monotonic() - started, 3)
+        job = self.application.tools.store.get_job("vm-test", job_id)
+        self.assertEqual(job["status"], "interrupted")
+        self.assertTrue(job["outcome_unknown"])
+        self.assertTrue(all(not thread.is_alive() for thread in self.application.tools._threads))

tests/test_tool_jobs.py

@@ -1,9 +1,7 @@
 from __future__ import annotations
 
 import json
-import os
 import tempfile
-import threading
 import unittest
 from concurrent.futures import ThreadPoolExecutor
 from dataclasses import replace

uvm/config.py

@@ -2,8 +2,8 @@
 
 from __future__ import annotations
 
-import os
 import math
+import os
 from dataclasses import dataclass, field
 from ipaddress import IPv4Address, IPv4Network
 from pathlib import Path
@@ -62,7 +62,7 @@ class Settings:
             raise ConfigurationError("tool_workers must be an integer between 1 and 8")
         for name in ("tool_ready_timeout_s", "tool_install_timeout_s"):
             value = getattr(self, name)
-            if type(value) not in (int, float) or not math.isfinite(value) or not 0 < value <= 3600:
+            if type(value) not in (int, float) or not 0 < value <= 3600 or not math.isfinite(value):
                 raise ConfigurationError(f"{name} must be finite, greater than zero, and at most 3600 seconds")
 
         # Resolve defaults at construction time, while preserving individual URL overrides.

uvm/guest_tools.py

@@ -281,7 +281,7 @@ class Installer:
                 raise ToolError(f"installation directory is already in use: {prefix}")
             regular_file(marker)
             previous = json.loads(marker.read_bytes())
-            if any(previous.get(key) != value for key, value in identity.items()):
+            if not isinstance(previous, dict) or any(previous.get(key) != value for key, value in identity.items()):
                 raise ToolError(f"installation directory belongs to a different release: {prefix}")
             if tool["kind"] != "conda" or previous.get("complete"):
                 return prefix
@@ -348,6 +348,10 @@ class Installer:
             if manifest.exists():
                 regular_file(manifest)
                 self.managed = json.loads(manifest.read_bytes())
+                if (not isinstance(self.managed, dict) or any(
+                        not isinstance(values, list) or len(values) > 2 or any(not isinstance(value, str) for value in values)
+                        for values in self.managed.values())):
+                    raise ToolError("invalid managed tool file manifest")
             with tempfile.TemporaryDirectory(prefix=".install-", dir=ROOT) as name:
                 temporary = Path(name)
                 if tool["kind"] == "php":
@@ -369,11 +373,13 @@ class Installer:
 
 def main(payload):
     installer = Installer(payload["username"])
+    info = {}
     try:
         if os.geteuid() != 0:
             raise ToolError("tool installation and inspection require guest root")
         info = platform_info()
-        if not info["supported"] or info["architecture"] != payload["architecture"]:
+        info["supported"] = info["supported"] and info["architecture"] == payload["architecture"]
+        if not info["supported"]:
             raise ToolError("one-click tools require an Ubuntu 22.04 guest on the selected x86_64/ARM64 architecture")
         if payload.get("boot_id") and payload["boot_id"] != info["boot_id"]:
             raise ToolError("the guest rebooted during installation; retry on the current boot")
@@ -385,7 +391,7 @@ def main(payload):
             raise ToolError("unsupported tool operation")
         result["ok"] = True
     except (ToolError, OSError, ValueError, KeyError, tarfile.TarError, subprocess.SubprocessError) as error:
-        result = {"ok": False, "error": str(error)[:2048]}
+        result = {"ok": False, "error": str(error)[:2048], "supported": info.get("supported")}
     result.update(output=installer.log.decode("utf-8", errors="replace"), truncated=installer.truncated)
     print(json.dumps(result, ensure_ascii=True))
 

uvm/tool_catalog.py

@@ -152,11 +152,11 @@ def recipe(selection: ToolSelection, architecture: str) -> dict:
                       url=("https://github.com/conda-forge/miniforge/releases/download/26.5.3-0/"
                            f"Miniforge3-26.5.3-0-Linux-{architecture}.sh"), sha256=_CONDA_SHA256[architecture],
                       links={"conda": "bin/conda", "mamba": "bin/mamba"},
-                      usage="conda create -n myenv python=3.13; conda activate myenv")
+                      usage='eval "$(conda shell.posix hook)"; conda create -n myenv python=3.13; conda activate myenv')
     elif tool == "micromamba":
         arch = "64" if architecture == "x86_64" else "aarch64"
         result.update(kind="binary", release="micromamba-2.9.0", packages=["ca-certificates"],
                       url=f"https://github.com/mamba-org/micromamba-releases/releases/download/2.9.0-0/micromamba-linux-{arch}",
                       sha256=_MICROMAMBA_SHA256[architecture], links={"micromamba": "micromamba"},
-                      usage="micromamba create -n myenv -c conda-forge python=3.13; micromamba activate myenv")
+                      usage='eval "$(micromamba shell hook --shell posix)"; micromamba create -n myenv -c conda-forge python=3.13; micromamba activate myenv')
     return result

uvm/tool_store.py

@@ -56,6 +56,8 @@ class ToolJobStore:
             descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
         except FileNotFoundError:
             return self._empty()
+        except OSError as error:
+            raise StateError("could not open private tool installation history") from error
         try:
             with os.fdopen(descriptor, "rb") as stream:
                 metadata = os.fstat(stream.fileno())

uvm/tools.py

@@ -77,12 +77,19 @@ class ToolService:
             self._owner = descriptor
             try:
                 self.store.recover()
-                self._workers_left = self.settings.tool_workers
+                for index in range(self.settings.tool_workers):
-                self._threads = [threading.Thread(target=self._worker, name=f"uvm-tools-{index}", daemon=True)
+                    thread = threading.Thread(target=self._worker, name=f"uvm-tools-{index}", daemon=True)
-                                 for index in range(self.settings.tool_workers)]
+                    self._workers_left += 1
-                for thread in self._threads:
+                    try:
                         thread.start()
                     except BaseException:
+                        self._workers_left -= 1
+                        raise
+                    self._threads.append(thread)
+            except BaseException:
+                self._stop.set()
+                self.notify()
+                if not self._workers_left:
                     self._release_owner()
                 raise
 
@@ -181,8 +188,11 @@ class ToolService:
         data = self.store.read(vm_id)
         running = self.lifecycle.inspect(vm.id).observed_status == "running"
         if refresh and running:
+            supported = data["supported"]
             try:
                 result = self._execute(vm, self._payload(vm, "probe", all_selections()), 90)
+                if type(result.get("supported")) is bool:
+                    supported = result["supported"]
                 if not result["ok"]:
                     raise UvmError(_text(result.get("error"), 2048))
                 inventory = self._inventory(result, all_selections())
@@ -192,7 +202,8 @@ class ToolService:
                 data = self.store.read(vm_id)
             except UvmError as error:
                 with self.store.transaction(vm_id) as current:
-                    current.update(error=_text(error, 2048), agent_ready=not isinstance(error, GuestAgentUnavailableError))
+                    current.update(error=_text(error, 2048), agent_ready=not isinstance(error, GuestAgentUnavailableError),
+                                   supported=supported)
                 data = self.store.read(vm_id)
         if not running:
             data.update(agent_ready=False, guest_busy=False)
@@ -258,7 +269,7 @@ class ToolService:
             def ready(current, data):
                 observed = {item["key"]: item for item in data["inventory"]}
                 observed.update({item["key"]: item for item in initial})
-                data.update(inventory=list(observed.values()), checked_at=time.time(), agent_ready=True,
+                data.update(inventory=list(observed.values()), agent_ready=True,
                             guest_busy=False, supported=True, error=None)
             self._update(job, ready)
             for index, selection in enumerate(selections):
@@ -289,7 +300,7 @@ class ToolService:
                         inventory[selection.key] = observed
                     elif selection.key in inventory:
                         inventory[selection.key].update(installed=False, error="Refresh to verify after the failed installation.")
-                    data.update(inventory=list(inventory.values()), checked_at=time.time())
+                    data.update(inventory=list(inventory.values()))
                 self._update(job, finish_item)
                 submitted = False
 
@@ -317,3 +328,11 @@ class ToolService:
                 self._update(job, failed)
             except NotFoundError:
                 pass
+        except Exception:
+            logging.getLogger(__name__).exception("Unexpected tool installation failure")
+            try:
+                self._update(job, lambda current, data: self.store.interrupt(
+                    current, "Unexpected installer failure; check the management server log and refresh before retrying.",
+                    time.time(), outcome_unknown=submitted))
+            except NotFoundError:
+                pass