feat: enhance guest network configuration and recovery process for UVM
This commit is contained in:
+43
-3
@@ -336,7 +336,7 @@ a ruleset/attachment snapshot, not application health or a reachability test.
|
||||
3. Reserve ID/name/IP/MAC/ports with `status=starting`, `operation=preparing`.
|
||||
4. Create the owned mode-0700 runtime directory, sparse-copy a mode-0600 private
|
||||
disk, resize ext4 to the requested storage capacity, provision credentials/host
|
||||
keys, and persist `disk_ready=True`.
|
||||
keys and the managed guest network profile, then persist `disk_ready=True`.
|
||||
5. Ensure bridge/egress networking and prepare an owned, attached TAP **down**.
|
||||
6. Apply/read back the nftables bridge policy, then activate the TAP.
|
||||
7. Write Firecracker config; persist `operation=spawning`, `spawn_pending=True`,
|
||||
@@ -373,6 +373,14 @@ not needed. It never recopies the rootfs, reprovisions credentials, or
|
||||
regenerates guest SSH host keys. Name/IP/MAC, ports, and guest disk changes
|
||||
survive stop/start; guest RAM does not.
|
||||
|
||||
After runtime cleanup confirms process exit and any pending resize finishes,
|
||||
start specializes an original UVM Ubuntu network profile with the retained
|
||||
guest address/gateway. This one-time migration repairs older guests that lost
|
||||
their default route. It uses a checked sparse copy of the retained private
|
||||
disk; it never replaces that disk with the shared template. Guest-edited
|
||||
profiles are preserved. The image layer's network checks and writes run only
|
||||
under the lifecycle operation lock, before the TAP and VMM are started.
|
||||
|
||||
Start on a healthy running VM validates its retained network, synchronizes
|
||||
policy, then restores shared bridge/NAT/forwarding rules without spawning another
|
||||
VMM. A shared-network setup failure retains the running VM for retry. An ambiguous
|
||||
@@ -500,7 +508,9 @@ must prevent `InstanceStart`. Without guest entropy, systemd's random-seed
|
||||
service and OpenSSH can remain in startup while the VM process and IP respond.
|
||||
|
||||
The guest receives a static IP via the kernel `ip=` boot argument. The netmask
|
||||
comes from `Settings.network`, not a hard-coded `/24` value.
|
||||
comes from `Settings.network`, not a hard-coded `/24` value. The Ubuntu
|
||||
systemd-networkd profile also receives explicit `Address=` and `Gateway=`
|
||||
settings so it can maintain the same configuration after boot or reconfiguration.
|
||||
ARM64 guests also receive `keep_bootcon`, as recommended by Firecracker's
|
||||
getting-started guide; both architectures use `console=ttyS0`.
|
||||
|
||||
@@ -566,6 +576,30 @@ UVM replaces conventional RSA, ECDSA, and Ed25519 host keys in each private
|
||||
disk and removes the known public login key from Firecracker's bionic image.
|
||||
A VM must never rely on reusable SSH private keys from a shared template.
|
||||
|
||||
### Guest Network Profile
|
||||
|
||||
`GUEST_NETWORK_TEMPLATE` in `images.py` is shared with the Ubuntu builder. It
|
||||
uses `KeepConfiguration=yes`: in Ubuntu 22.04's systemd-networkd, `static`
|
||||
retains permanent addresses and `proto static` routes, but the kernel's
|
||||
`ip=`-configured default route uses `proto boot` and is otherwise removed.
|
||||
This can leave SSH to the private IP working while all off-subnet destinations
|
||||
fail locally with `Network is unreachable`.
|
||||
|
||||
`ImageStore.configure_guest_network()` fills in the assigned address/prefix and
|
||||
gateway during creation and stopped-disk resume. It only specializes the exact
|
||||
shipped profile from recognized UVM Jammy v1/v2/v3 images. Missing, customized,
|
||||
symlinked, hard-linked, or unexpectedly owned/mode-set profiles are left intact.
|
||||
An already-specialized profile is a no-op. Custom images retain responsibility
|
||||
for their own guest networking.
|
||||
|
||||
For migration, the image layer sparse-copies the private disk and runs
|
||||
`e2fsck -f -p` on the copy. It rechecks the profile after journal replay so
|
||||
committed guest edits cannot be overwritten based on stale offline reads.
|
||||
The updated profile is read back and the filesystem checked before fsync and
|
||||
atomic disk replacement. Failed staging keeps the original disk available for
|
||||
retry. The built template's recipe marker is v3; cached older templates are
|
||||
also handled by per-VM provisioning without requiring an installation refresh.
|
||||
|
||||
### Integrity Model
|
||||
|
||||
[`integrity.py`](uvm/integrity.py) has two concepts:
|
||||
@@ -985,6 +1019,10 @@ and stop-on-error behavior. Tests simulate Ubuntu removing `policy-rc.d` after
|
||||
bootstrap. They do not run debootstrap or boot a VM.
|
||||
Lifecycle resize tests use fake processes/networking to verify ordering,
|
||||
single-lock ownership, identity preservation, pending intent, and failure recovery.
|
||||
`test_guest_network.py` exercises real offline ext4 profiles, explicit guest
|
||||
addresses/gateways, custom subnets, idempotent migration, journal-replay
|
||||
rechecks, and retention of guest data, credentials and keys on failures.
|
||||
Lifecycle tests require confirmed process exit before network-profile repair.
|
||||
|
||||
### Isolated Packet Tests
|
||||
|
||||
@@ -1017,7 +1055,8 @@ on an isolated Linux host:
|
||||
|
||||
1. Strict checked installation.
|
||||
2. `create` and successful guest boot.
|
||||
3. Guest IP reachability and egress.
|
||||
3. Guest IP reachability and egress, including the guest's default route after
|
||||
systemd-networkd startup and reconfiguration.
|
||||
4. SSH with unique guest host keys, `sudo -i` for a custom login, and
|
||||
`sudo apt update && sudo apt install htop` in a newly built Ubuntu guest.
|
||||
5. Stop/start/restart with retained guest data, SSH keys, and credentials.
|
||||
@@ -1055,6 +1094,7 @@ on an isolated Linux host:
|
||||
| VM is `dead` in `list` | PID/start token in state, `/proc/<pid>`, VM log |
|
||||
| Cannot stop legacy VM | Missing start/boot identity or failed protected binary/socket match; never guess a PID |
|
||||
| Guest has no egress | Host connectivity/routes, bridge/TAP state, `uvm-egress` iptables rules; `start` refreshes networking on a running VM |
|
||||
| Guest says `Network is unreachable` | Guest `ip -4 route`; the default route must use the UVM bridge gateway. Restart once with updated code to migrate an original Ubuntu profile. |
|
||||
| Policy missing/drift/unknown | Actual nft JSON and ownership, STP/group mask, permissions; explicit sync after fixing cause |
|
||||
| Retained-disk start fails | `disk_ready`, safe private path, kernel/VMM integrity, process identity, retained network, pending operation |
|
||||
| API server refuses to start | `UVM_API_TOKEN`, requested host, TLS cert/key files |
|
||||
|
||||
@@ -423,6 +423,19 @@ template retains apt's package metadata and enables `main` and `universe`,
|
||||
including updates/security repositories. Guest DNS defaults to `1.1.1.1` and
|
||||
`8.8.8.8` in `/etc/resolv.conf`.
|
||||
|
||||
UVM specializes the Ubuntu guest's `/etc/systemd/network/10-uvm.network` with
|
||||
its assigned `Address=` and `Gateway=` before boot. `KeepConfiguration=yes`
|
||||
also retains the kernel-created default route while systemd-networkd starts.
|
||||
Older templates used `KeepConfiguration=static`, which could retain the guest
|
||||
IP but remove the default route, producing `ping: connect: Network is unreachable`.
|
||||
|
||||
The updated CLI repairs an older VM's original, unedited UVM profile on its
|
||||
next stopped-disk boot, including `restart`. The repair is prepared and checked
|
||||
on a sparse copy before publication; guest-edited profiles and custom images
|
||||
are preserved. New VMs also receive the fix when using a cached older Ubuntu
|
||||
template. See [guest route recovery](USAGE.md#guest-reports-network-is-unreachable)
|
||||
for immediate recovery inside a running guest.
|
||||
|
||||
Existing accounts retain their UID, groups, home directory, and shell. A new
|
||||
user reuses a same-named group if present; otherwise UVM creates one with an
|
||||
unused GID. `/etc/gshadow` is updated when present. The default `root` account
|
||||
@@ -469,6 +482,8 @@ sudo ./uvm.py start web
|
||||
|
||||
On a running VM, `start` refreshes its private policy and shared host networking
|
||||
without rebooting it. New VM boots set up the same networking automatically.
|
||||
For an older guest missing its default route, use `restart` once to apply the
|
||||
offline guest-profile repair, or follow the guest route recovery steps above.
|
||||
|
||||
Private VM creation and start require an RFC 1918 subnet (`10.0.0.0/8`,
|
||||
`172.16.0.0/12`, or `192.168.0.0/16`). Retained VMs prevent incompatible
|
||||
@@ -516,7 +531,7 @@ is mode `0600`. Treat it as a secret, do not include it in bug reports, and
|
||||
replace the default `root` password immediately.
|
||||
|
||||
Credential provisioning applies only to VMs created by this version. Existing
|
||||
VM disks are not modified automatically.
|
||||
guest accounts, passwords, and SSH host keys are retained on subsequent boots.
|
||||
|
||||
Older registries are read and migrated on the next write. A missing/null
|
||||
`ports` policy remains `None`/`unmanaged`, not an empty allowlist or an
|
||||
|
||||
@@ -1203,6 +1203,48 @@ without a reboot. Use the updated CLI checkout, or restart the API server to
|
||||
load the updated code before using its start endpoint. Subsequent host
|
||||
disconnects, reconnects, and route changes require no UVM command.
|
||||
|
||||
### Guest Reports `Network is Unreachable`
|
||||
|
||||
An immediate `ping: connect: Network is unreachable` inside the VM indicates
|
||||
that the guest has no usable route to that destination. Inspect the guest's
|
||||
networking from its SSH shell:
|
||||
|
||||
```sh
|
||||
ip -4 address show dev eth0
|
||||
ip -4 route
|
||||
```
|
||||
|
||||
For the default UVM subnet, the route must include `default via 10.42.0.1 dev
|
||||
eth0`. The guest gateway is the **UVM bridge address**, even when the host's
|
||||
own uplink uses a different gateway.
|
||||
|
||||
Older UVM Ubuntu images used `KeepConfiguration=static`. Ubuntu's network
|
||||
manager could preserve the guest address but remove its kernel-created
|
||||
`proto boot` default route. Inside the running guest, restore it immediately:
|
||||
|
||||
```sh
|
||||
sudo ip -4 route replace default via 10.42.0.1 dev eth0 proto static
|
||||
ping -c 3 1.1.1.1
|
||||
```
|
||||
|
||||
For a persistent repair, run the updated CLI **on the host**:
|
||||
|
||||
```sh
|
||||
sudo ./uvm.py restart 10.42.0.2
|
||||
```
|
||||
|
||||
This stops the VM, repairs its original UVM network profile on the retained
|
||||
disk, and boots it with explicit address/gateway settings. A stopped VM's
|
||||
`start` applies the same repair; `start` on an already-running VM refreshes
|
||||
host rules only. Per-VM provisioning also repairs cached older Ubuntu
|
||||
templates when creating new VMs. Use the configured `UVM_GATEWAY` and guest IP
|
||||
in place of these defaults for a custom subnet.
|
||||
|
||||
Guest-edited profiles and custom images are preserved. For those, persist the
|
||||
correct `Address=` and `Gateway=` in the guest's own network configuration.
|
||||
|
||||
### Verify Guest Internet Access
|
||||
|
||||
Check DNS and HTTPS from the guest:
|
||||
|
||||
```sh
|
||||
@@ -1215,6 +1257,7 @@ Common issues:
|
||||
| Symptom | First checks |
|
||||
|---|---|
|
||||
| Guest cannot reach the internet | Confirm the host can reach the same destination, bridge/TAP are up, and `uvm-egress` NAT/FORWARD rules exist; use `start VM` to refresh sharing. |
|
||||
| Guest immediately reports `Network is unreachable` | Inspect the guest's default route and apply the guest route recovery steps above. |
|
||||
| IP connectivity works but DNS fails | Check guest `/etc/resolv.conf` and DNS reachability. The Ubuntu template uses `1.1.1.1` and `8.8.8.8`; host loopback resolvers such as `127.0.0.53` are not reachable as guest loopback addresses. |
|
||||
| Requested guest IP is rejected | Confirm it is inside the configured subnet and not used, gateway, network, or broadcast. |
|
||||
| Create says TAP already exists | Inspect the interface; do not delete it unless you know it is stale UVM state. |
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import configparser
|
||||
import hashlib
|
||||
import shutil
|
||||
import tempfile
|
||||
import unittest
|
||||
from dataclasses import replace
|
||||
from ipaddress import IPv4Address, IPv4Network
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from uvm.config import Settings
|
||||
from uvm.errors import UvmError, ValidationError
|
||||
from uvm.images import GUEST_NETWORK_TEMPLATE, ImageStore
|
||||
from uvm.system import CommandResult, CommandRunner
|
||||
|
||||
|
||||
PROFILE = "/etc/systemd/network/10-uvm.network"
|
||||
# The shipped v1/v2 profile that retained the IP but lost the proto-boot route.
|
||||
LEGACY_PROFILE = (
|
||||
"[Match]\nName=eth0\n\n[Network]\n"
|
||||
"KeepConfiguration=static\nDHCP=no\nLinkLocalAddressing=no\nIPv6AcceptRA=no\n"
|
||||
)
|
||||
|
||||
|
||||
class GuestNetworkValidationTests(unittest.TestCase):
|
||||
def test_invalid_guest_addresses_fail_before_disk_access(self) -> None:
|
||||
runner = Mock(spec=CommandRunner)
|
||||
store = ImageStore(Settings(), runner)
|
||||
for address in ("invalid", "::1", "192.0.2.2", "10.42.0.0", "10.42.0.255", "10.42.0.1"):
|
||||
with self.subTest(address=address), self.assertRaises(ValidationError):
|
||||
store.configure_guest_network(Path("/nonexistent/rootfs.ext4"), address)
|
||||
runner.run.assert_not_called()
|
||||
|
||||
def test_shared_or_linked_disks_are_rejected_before_guest_commands(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as name:
|
||||
settings = Settings(base=Path(name))
|
||||
settings.images_dir.mkdir()
|
||||
settings.rootfs_image.write_bytes(b"shared template")
|
||||
disk = settings.base / "rootfs.ext4"
|
||||
disk.write_bytes(b"private disk")
|
||||
linked = settings.base / "linked.ext4"
|
||||
linked.hardlink_to(disk)
|
||||
symlink = settings.base / "symlink.ext4"
|
||||
symlink.symlink_to(disk)
|
||||
runner = Mock(spec=CommandRunner)
|
||||
store = ImageStore(settings, runner)
|
||||
for candidate in (settings.rootfs_image, disk, linked, symlink):
|
||||
with self.subTest(candidate=candidate), self.assertRaisesRegex(UvmError, "unsafe"):
|
||||
store.configure_guest_network(candidate, "10.42.0.2")
|
||||
runner.run.assert_not_called()
|
||||
|
||||
|
||||
@unittest.skipUnless(all(shutil.which(tool) for tool in ("mkfs.ext4", "debugfs", "e2fsck", "cp")),
|
||||
"e2fsprogs and coreutils are required for offline guest networking checks")
|
||||
class OfflineGuestNetworkTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
temporary = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temporary.cleanup)
|
||||
self.base = Path(temporary.name)
|
||||
self.settings = Settings(base=self.base / "uvm")
|
||||
self.settings.images_dir.mkdir(parents=True)
|
||||
self.disk = self.base / "VM with spaces" / "rootfs.ext4"
|
||||
self.disk.parent.mkdir()
|
||||
self.runner = CommandRunner(emit=None)
|
||||
self.store = ImageStore(self.settings, self.runner)
|
||||
self.retained = {
|
||||
"/etc/shadow": "adm:$6$original-password-hash:1:0:99999:7:::\n",
|
||||
"/etc/ssh/ssh_host_ed25519_key": "original private host key\n",
|
||||
"/home/adm/data": "retained guest application data\n",
|
||||
}
|
||||
|
||||
def make_disk(self, *, marker="uvm-ubuntu-jammy-v2\n", profile=LEGACY_PROFILE, symlink=False) -> None:
|
||||
root = self.base / "tree"
|
||||
if root.exists():
|
||||
shutil.rmtree(root)
|
||||
files = dict(self.retained)
|
||||
if marker is not None:
|
||||
files["/etc/uvm-image"] = marker
|
||||
if profile is not None:
|
||||
files["/etc/alternate.network" if symlink else PROFILE] = profile
|
||||
for path, contents in files.items():
|
||||
local = root / path.lstrip("/")
|
||||
local.parent.mkdir(parents=True, exist_ok=True)
|
||||
local.write_text(contents, encoding="utf-8")
|
||||
local.chmod(0o644)
|
||||
if symlink:
|
||||
local = root / PROFILE.lstrip("/")
|
||||
local.parent.mkdir(parents=True, exist_ok=True)
|
||||
local.symlink_to("/etc/alternate.network")
|
||||
with self.settings.rootfs_image.open("wb") as image:
|
||||
image.truncate(32 * 1024 * 1024)
|
||||
self.runner.run(("mkfs.ext4", "-q", "-F", "-b", "4096", "-d", root, self.settings.rootfs_image), capture=True)
|
||||
if profile is not None:
|
||||
for field in ("uid", "gid"):
|
||||
self.runner.run(("debugfs", "-w", "-R", f"set_inode_field {PROFILE} {field} 0",
|
||||
self.settings.rootfs_image), capture=True)
|
||||
self.runner.run(("cp", "--sparse=always", self.settings.rootfs_image, self.disk), capture=True)
|
||||
self.disk.chmod(0o600)
|
||||
|
||||
def guest_file(self, path: str) -> str:
|
||||
return self.runner.run(("debugfs", "-R", f"cat {path}", self.disk), capture=True).stdout
|
||||
|
||||
def snapshot(self) -> tuple[int, bytes]:
|
||||
return self.disk.stat().st_ino, hashlib.sha256(self.disk.read_bytes()).digest()
|
||||
|
||||
def test_templates_gain_an_explicit_address_gateway_and_boot_route_preservation(self) -> None:
|
||||
for marker, profile in (("uvm-ubuntu-jammy-v1\n", LEGACY_PROFILE),
|
||||
("uvm-ubuntu-jammy-v2\n", LEGACY_PROFILE),
|
||||
("uvm-ubuntu-jammy-v3\n", GUEST_NETWORK_TEMPLATE)):
|
||||
with self.subTest(marker=marker):
|
||||
self.make_disk(marker=marker, profile=profile)
|
||||
original_template = hashlib.sha256(self.settings.rootfs_image.read_bytes()).digest()
|
||||
self.assertTrue(self.store.configure_guest_network(self.disk, "10.42.0.2"))
|
||||
config = configparser.ConfigParser()
|
||||
config.read_string(self.guest_file(PROFILE))
|
||||
self.assertEqual(config["Match"]["Name"], "eth0")
|
||||
self.assertEqual(config["Network"]["Address"], "10.42.0.2/24")
|
||||
self.assertEqual(config["Network"]["Gateway"], "10.42.0.1")
|
||||
self.assertTrue(config["Network"].getboolean("KeepConfiguration"))
|
||||
self.assertFalse(config["Network"].getboolean("DHCP"))
|
||||
for path, contents in self.retained.items():
|
||||
self.assertEqual(self.guest_file(path), contents)
|
||||
self.assertEqual(hashlib.sha256(self.settings.rootfs_image.read_bytes()).digest(), original_template)
|
||||
self.assertEqual(self.disk.stat().st_size, 32 * 1024 * 1024)
|
||||
self.assertEqual(self.disk.stat().st_mode & 0o777, 0o600)
|
||||
self.runner.run(("e2fsck", "-f", "-n", self.disk), capture=True)
|
||||
|
||||
before = self.snapshot()
|
||||
with patch.object(self.store, "_copy_sparse") as copy:
|
||||
self.assertFalse(self.store.configure_guest_network(self.disk, "10.42.0.2"))
|
||||
copy.assert_not_called()
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
|
||||
def test_custom_subnets_and_gateways_are_used_in_the_guest_profile(self) -> None:
|
||||
settings = replace(self.settings, network=IPv4Network("10.50.0.0/16"), gateway=IPv4Address("10.50.1.1"))
|
||||
self.make_disk()
|
||||
self.assertTrue(ImageStore(settings, self.runner).configure_guest_network(self.disk, "10.50.8.9"))
|
||||
config = configparser.ConfigParser()
|
||||
config.read_string(self.guest_file(PROFILE))
|
||||
self.assertEqual(config["Network"]["Address"], "10.50.8.9/16")
|
||||
self.assertEqual(config["Network"]["Gateway"], "10.50.1.1")
|
||||
|
||||
def test_custom_images_missing_profiles_and_guest_edits_are_left_intact(self) -> None:
|
||||
for marker, profile, symlink in (
|
||||
(None, LEGACY_PROFILE, False), ("custom-image\n", LEGACY_PROFILE, False),
|
||||
("uvm-ubuntu-jammy-v2\n", None, False),
|
||||
("uvm-ubuntu-jammy-v2\n", LEGACY_PROFILE + "# guest customization\n", False),
|
||||
("uvm-ubuntu-jammy-v2\n", LEGACY_PROFILE + "Gateway=10.42.0.9\n", False),
|
||||
("uvm-ubuntu-jammy-v2\n", LEGACY_PROFILE, True),
|
||||
):
|
||||
with self.subTest(marker=marker, profile=profile, symlink=symlink):
|
||||
self.make_disk(marker=marker, profile=profile, symlink=symlink)
|
||||
before = self.snapshot()
|
||||
with patch.object(self.store, "_copy_sparse") as copy:
|
||||
self.assertFalse(self.store.configure_guest_network(self.disk, "10.42.0.2"))
|
||||
copy.assert_not_called()
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
|
||||
def test_a_failed_or_interrupted_profile_write_keeps_the_original_disk(self) -> None:
|
||||
for error in (UvmError("profile readback failed"), KeyboardInterrupt()):
|
||||
with self.subTest(error=type(error).__name__):
|
||||
self.make_disk()
|
||||
before = self.snapshot()
|
||||
write = self.store._write_guest_file
|
||||
|
||||
def fail(candidate, path, contents):
|
||||
self.assertNotEqual(candidate, self.disk)
|
||||
write(candidate, path, contents)
|
||||
raise error
|
||||
|
||||
with patch.object(self.store, "_write_guest_file", side_effect=fail), self.assertRaises(type(error)):
|
||||
self.store.configure_guest_network(self.disk, "10.42.0.2")
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
self.assertEqual(list(self.disk.parent.iterdir()), [self.disk])
|
||||
|
||||
def test_filesystem_check_failure_prevents_publication(self) -> None:
|
||||
self.make_disk()
|
||||
before = self.snapshot()
|
||||
run = self.runner.run
|
||||
|
||||
def check_failure(command, **options):
|
||||
if command[0] == "e2fsck":
|
||||
return CommandResult(tuple(map(str, command)), 4, stderr="unrepaired filesystem errors")
|
||||
return run(command, **options)
|
||||
|
||||
with patch.object(self.runner, "run", side_effect=check_failure):
|
||||
with self.assertRaisesRegex(UvmError, "unrepaired filesystem errors"):
|
||||
self.store.configure_guest_network(self.disk, "10.42.0.2")
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
self.assertEqual(list(self.disk.parent.iterdir()), [self.disk])
|
||||
|
||||
def test_guest_edits_revealed_after_journal_replay_prevent_migration(self) -> None:
|
||||
self.make_disk()
|
||||
before = self.snapshot()
|
||||
edited = self.base / "edited.network"
|
||||
edited.write_text(LEGACY_PROFILE + "Gateway=10.42.0.9\n", encoding="utf-8")
|
||||
run = self.runner.run
|
||||
|
||||
def replay(command, **options):
|
||||
result = run(command, **options)
|
||||
if command[:3] == ("e2fsck", "-f", "-p"):
|
||||
run(("debugfs", "-w", "-R", f"rm {PROFILE}", command[-1]), capture=True)
|
||||
run(("debugfs", "-w", "-R", f"write {edited} {PROFILE}", command[-1]), capture=True)
|
||||
for field in ("uid", "gid"):
|
||||
run(("debugfs", "-w", "-R", f"set_inode_field {PROFILE} {field} 0", command[-1]), capture=True)
|
||||
return result
|
||||
|
||||
with patch.object(self.runner, "run", side_effect=replay):
|
||||
self.assertFalse(self.store.configure_guest_network(self.disk, "10.42.0.2"))
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
self.assertEqual(list(self.disk.parent.iterdir()), [self.disk])
|
||||
@@ -158,6 +158,7 @@ class InstallerArchitectureTests(unittest.TestCase):
|
||||
for verified, marker, force in (
|
||||
(True, _ROOTFS_RECIPE, False), (False, _ROOTFS_RECIPE, False),
|
||||
(True, "", False), (True, "uvm-ubuntu-jammy-v1\n", False), (True, _ROOTFS_RECIPE, True),
|
||||
(True, "uvm-ubuntu-jammy-v2\n", False),
|
||||
):
|
||||
with self.subTest(verified=verified, marker=marker, force=force), tempfile.TemporaryDirectory() as temporary:
|
||||
settings = Settings(base=Path(temporary))
|
||||
@@ -335,7 +336,7 @@ class UbuntuRootfsBuildTests(unittest.TestCase):
|
||||
self.assertRegex(guest("stat /usr/bin/sudo").stdout, r"Mode:\s+0?4755")
|
||||
self.assertEqual(guest("cat /etc/uvm-image").stdout, _ROOTFS_RECIPE)
|
||||
self.assertIn("nameserver 1.1.1.1", guest("cat /etc/resolv.conf").stdout)
|
||||
self.assertIn("KeepConfiguration=static", guest("cat /etc/systemd/network/10-uvm.network").stdout)
|
||||
self.assertIn("KeepConfiguration=yes", guest("cat /etc/systemd/network/10-uvm.network").stdout)
|
||||
self.assertIn("jammy-security main universe", guest("cat /etc/apt/sources.list").stdout)
|
||||
for path in ("/usr/sbin/policy-rc.d", "/etc/ssh/ssh_host_rsa_key"):
|
||||
self.assertIn("File not found", guest(f"stat {path}").stderr)
|
||||
|
||||
+52
-1
@@ -307,6 +307,9 @@ class LifecycleTestCase(unittest.TestCase):
|
||||
self.images, "resize_vm_disk", side_effect=self.resize_vm_disk,
|
||||
))
|
||||
self.provision = self.enterContext(patch.object(self.images, "provision_credentials"))
|
||||
self.configure_network = self.enterContext(patch.object(
|
||||
self.images, "configure_guest_network", side_effect=self.configure_guest_network,
|
||||
))
|
||||
self.copy_disk = self.enterContext(patch.object(
|
||||
self.images, "create_vm_disk", wraps=self.images.create_vm_disk,
|
||||
))
|
||||
@@ -340,6 +343,14 @@ class LifecycleTestCase(unittest.TestCase):
|
||||
raise AssertionError("cannot resize a running or ambiguous VM")
|
||||
self.disk_sizes[disk] = storage_mib * 1024 * 1024
|
||||
|
||||
def configure_guest_network(self, disk: Path, guest_ip: str) -> None:
|
||||
vm = self.store.get(disk.parent.name)
|
||||
self.host.call("images.configure_network", vm.id)
|
||||
if self.process.observe(vm) != "gone":
|
||||
raise AssertionError("cannot configure a running or ambiguous VM disk")
|
||||
if guest_ip != vm.guest_ip:
|
||||
raise AssertionError("guest network must use the VM's retained address")
|
||||
|
||||
def create_vm(self, **kwargs) -> VmRecord:
|
||||
return self.service.create(VmSpec(cpu=1, ram_mib=512, **kwargs))
|
||||
|
||||
@@ -481,7 +492,7 @@ class LifecycleTests(LifecycleTestCase):
|
||||
def test_boot_persists_intent_and_identity_with_policy_before_tap_and_vmm(self) -> None:
|
||||
vm = self.create_vm()
|
||||
self.assert_call_order(
|
||||
vm, "network.prepare_tap", "firewall.apply", "firewall.readback",
|
||||
vm, "images.configure_network", "network.prepare_tap", "firewall.apply", "firewall.readback",
|
||||
"network.activate_tap", "process.start", "process.spawn", "process.ready",
|
||||
"client.configure", "firewall.status",
|
||||
)
|
||||
@@ -499,6 +510,46 @@ class LifecycleTests(LifecycleTestCase):
|
||||
self.assertEqual(spawned.operation, "configuring")
|
||||
self.assertEqual(self.host.records["client.configure", vm.id].pid, vm.pid)
|
||||
|
||||
def test_guest_network_configuration_failure_rolls_back_before_marking_the_disk_ready(self) -> None:
|
||||
self.host.failures["images.configure_network", None] = UvmError("guest network verification failed")
|
||||
with self.assertRaisesRegex(UvmError, "guest network verification failed"):
|
||||
self.create_vm()
|
||||
self.assertEqual(self.store.load().vms, {})
|
||||
self.assertFalse(self.network.created_taps)
|
||||
self.assertFalse(self.process.started)
|
||||
self.assertEqual(list(self.settings.vms_dir.iterdir()), [])
|
||||
|
||||
def test_retained_guest_network_is_configured_after_exit_and_before_boot(self) -> None:
|
||||
vm = self.create_vm()
|
||||
self.host.events.clear()
|
||||
self.service.restart(vm.id)
|
||||
self.assert_call_order(vm, "images.configure_network", "process.start", "client.configure")
|
||||
actions = self.host.actions(vm.id)
|
||||
self.assertLess(max(index for index, action in enumerate(actions) if action == "process.gone"),
|
||||
actions.index("images.configure_network"))
|
||||
self.configure_network.assert_called_with(Path(vm.disk), vm.guest_ip)
|
||||
self.assertEqual(self.provision.call_count, 1)
|
||||
|
||||
def test_live_start_does_not_modify_the_guest_disk(self) -> None:
|
||||
vm = self.create_vm()
|
||||
self.configure_network.reset_mock()
|
||||
self.service.start(vm.id)
|
||||
self.configure_network.assert_not_called()
|
||||
|
||||
def test_failed_retained_network_configuration_keeps_disk_for_retry(self) -> None:
|
||||
vm = self.create_vm()
|
||||
self.service.stop(vm.id)
|
||||
disk = Path(vm.disk)
|
||||
disk.write_bytes(b"retained data and credentials")
|
||||
self.host.failures["images.configure_network", vm.id] = UvmError("network verification failed")
|
||||
with self.assertRaisesRegex(UvmError, "network verification failed"):
|
||||
self.service.start(vm.id)
|
||||
self.assertEqual(disk.read_bytes(), b"retained data and credentials")
|
||||
self.assertEqual(self.process.observe(self.store.get(vm.id)), "gone")
|
||||
self.assertEqual(len(self.process.started), 1)
|
||||
self.assertEqual(self.service.start(vm.id).status, "running")
|
||||
self.assertEqual(self.provision.call_count, 1)
|
||||
|
||||
def test_failed_initial_policy_never_activates_tap_or_starts_vmm(self) -> None:
|
||||
self.host.failures["firewall.apply", None] = UvmError("policy unavailable")
|
||||
with self.assertRaisesRegex(UvmError, "policy unavailable"):
|
||||
|
||||
+100
@@ -8,6 +8,7 @@ import stat
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from ipaddress import IPv4Address
|
||||
from pathlib import Path
|
||||
|
||||
from .config import MAX_STORAGE_MIB, MIN_STORAGE_MIB, Settings
|
||||
@@ -35,6 +36,16 @@ _HOST_KEY_TYPES = (
|
||||
("ed25519", None),
|
||||
)
|
||||
|
||||
# Keep the kernel's proto-boot default route until networkd takes over. On
|
||||
# Ubuntu 22.04, KeepConfiguration=static preserves addresses but drops that route.
|
||||
GUEST_NETWORK_TEMPLATE = (
|
||||
"[Match]\nName=eth0\n\n[Network]\n"
|
||||
"KeepConfiguration=yes\nDHCP=no\nLinkLocalAddressing=no\nIPv6AcceptRA=no\n"
|
||||
)
|
||||
_LEGACY_GUEST_NETWORK_TEMPLATE = GUEST_NETWORK_TEMPLATE.replace(
|
||||
"KeepConfiguration=yes", "KeepConfiguration=static",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class GuestAssets:
|
||||
@@ -150,6 +161,91 @@ class ImageStore:
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
def configure_guest_network(self, disk: Path, guest_ip: str) -> bool:
|
||||
"""Specialize an unedited UVM profile on a proven-offline private disk.
|
||||
|
||||
Lifecycle owns the operation lock and confirms process exit. Stage and
|
||||
check a sparse copy so failed migrations retain the original disk. A
|
||||
completed migration, custom image, or guest-edited profile is a no-op.
|
||||
"""
|
||||
try:
|
||||
address = IPv4Address(guest_ip)
|
||||
except ValueError as error:
|
||||
raise ValidationError("guest network configuration requires a valid IPv4 address") from error
|
||||
if address not in self._settings.network or address in {
|
||||
self._settings.network.network_address,
|
||||
self._settings.network.broadcast_address,
|
||||
self._settings.gateway,
|
||||
}:
|
||||
raise ValidationError("guest network configuration requires a usable IP inside the UVM subnet")
|
||||
|
||||
try:
|
||||
metadata = disk.lstat()
|
||||
if (not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1
|
||||
or disk.resolve() == self._settings.rootfs_image.resolve()):
|
||||
raise UvmError(f"private VM disk is unsafe to configure: {disk}")
|
||||
with tempfile.TemporaryDirectory(prefix="uvm-guest-network-") as name:
|
||||
files = Path(name)
|
||||
if self._unconfigured_guest_network(disk, files) is None:
|
||||
return False
|
||||
|
||||
descriptor, candidate = tempfile.mkstemp(prefix=".rootfs-network-", suffix=".ext4", dir=disk.parent)
|
||||
os.close(descriptor)
|
||||
temporary = Path(candidate)
|
||||
try:
|
||||
self._copy_sparse(disk, temporary)
|
||||
temporary.chmod(0o600)
|
||||
checked = self._runner.run(("e2fsck", "-f", "-p", temporary), check=False, capture=True)
|
||||
if checked.returncode not in (0, 1):
|
||||
detail = checked.stderr.strip() or checked.stdout.strip()
|
||||
raise UvmError(f"filesystem check failed before guest network configuration: {detail}")
|
||||
# Journal replay may reveal guest edits not visible to the
|
||||
# initial debugfs read. Recheck ownership on the recovered copy.
|
||||
profile = self._unconfigured_guest_network(temporary, files)
|
||||
if profile is None:
|
||||
return False
|
||||
profile.path.write_text(
|
||||
GUEST_NETWORK_TEMPLATE
|
||||
+ f"Address={address}/{self._settings.network.prefixlen}\n"
|
||||
+ f"Gateway={self._settings.gateway}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
self._write_guest_file(temporary, "/etc/systemd/network/10-uvm.network", profile)
|
||||
self._runner.run(("e2fsck", "-f", "-n", temporary), capture=True)
|
||||
with temporary.open("rb") as image:
|
||||
os.fsync(image.fileno())
|
||||
os.replace(temporary, disk)
|
||||
directory = os.open(disk.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(directory)
|
||||
finally:
|
||||
os.close(directory)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
return True
|
||||
except (OSError, UnicodeError) as error:
|
||||
raise UvmError(f"could not configure guest networking: {error}") from error
|
||||
|
||||
def _unconfigured_guest_network(self, disk: Path, temporary: Path) -> _GuestFile | None:
|
||||
marker = self._read_guest_file(
|
||||
disk, "/etc/uvm-image", temporary / "marker", required=False, regular_only=True,
|
||||
)
|
||||
if marker is None or marker.path.read_text(encoding="utf-8") not in {
|
||||
"uvm-ubuntu-jammy-v1\n", "uvm-ubuntu-jammy-v2\n", "uvm-ubuntu-jammy-v3\n",
|
||||
}:
|
||||
return None
|
||||
profile = self._read_guest_file(
|
||||
disk, "/etc/systemd/network/10-uvm.network", temporary / "network", required=False, regular_only=True,
|
||||
)
|
||||
if (profile is None or (profile.uid, profile.gid, profile.mode, profile.links) != (0, 0, 0o644, 1)
|
||||
or profile.has_extended_attributes):
|
||||
return None
|
||||
if profile.path.read_text(encoding="utf-8") not in {
|
||||
_LEGACY_GUEST_NETWORK_TEMPLATE, GUEST_NETWORK_TEMPLATE,
|
||||
}:
|
||||
return None
|
||||
return profile
|
||||
|
||||
@staticmethod
|
||||
def _validate_storage(storage_mib: int) -> None:
|
||||
if type(storage_mib) is not int or not MIN_STORAGE_MIB <= storage_mib <= MAX_STORAGE_MIB:
|
||||
@@ -397,12 +493,16 @@ class ImageStore:
|
||||
destination: Path,
|
||||
*,
|
||||
required: bool = True,
|
||||
regular_only: bool = False,
|
||||
) -> _GuestFile | None:
|
||||
stat_result = self._runner.run(
|
||||
("debugfs", "-R", f"stat {guest_path}", disk),
|
||||
check=False,
|
||||
capture=True,
|
||||
)
|
||||
file_type = re.search(r"Type:\s+(\S+)", stat_result.stdout)
|
||||
if regular_only and file_type is not None and file_type.group(1) != "regular":
|
||||
return None
|
||||
mode_match = _MODE_PATTERN.search(stat_result.stdout)
|
||||
owner_match = _OWNER_PATTERN.search(stat_result.stdout)
|
||||
flags_match = _FLAGS_PATTERN.search(stat_result.stdout)
|
||||
|
||||
+3
-6
@@ -14,14 +14,14 @@ from pathlib import Path
|
||||
|
||||
from .config import Settings
|
||||
from .errors import UvmError
|
||||
from .images import GuestAssets
|
||||
from .images import GUEST_NETWORK_TEMPLATE, GuestAssets
|
||||
from .integrity import load_manifest, require_checksum, sha256_file, verify_file, write_manifest
|
||||
from .state import StateStore
|
||||
from .system import CommandRunner, check_kvm, ensure_data_directories, require_root
|
||||
|
||||
|
||||
_UBUNTU_RELEASE = "jammy"
|
||||
_ROOTFS_RECIPE = "uvm-ubuntu-jammy-v2\n"
|
||||
_ROOTFS_RECIPE = "uvm-ubuntu-jammy-v3\n"
|
||||
_GUEST_PACKAGES = (
|
||||
"ca-certificates", "openssh-server", "sudo", "systemd-sysv", "udev",
|
||||
"dbus", "libpam-systemd", "iproute2", "iputils-ping", "curl",
|
||||
@@ -383,10 +383,7 @@ class Installer:
|
||||
"etc/resolv.conf": "nameserver 1.1.1.1\nnameserver 8.8.8.8\n",
|
||||
"etc/machine-id": "",
|
||||
"etc/uvm-image": _ROOTFS_RECIPE,
|
||||
"etc/systemd/network/10-uvm.network": (
|
||||
"[Match]\nName=eth0\n\n[Network]\n"
|
||||
"KeepConfiguration=static\nDHCP=no\nLinkLocalAddressing=no\nIPv6AcceptRA=no\n"
|
||||
),
|
||||
"etc/systemd/network/10-uvm.network": GUEST_NETWORK_TEMPLATE,
|
||||
}
|
||||
for relative, contents in files.items():
|
||||
path = root / relative
|
||||
|
||||
@@ -89,6 +89,7 @@ class LifecycleService:
|
||||
self._save(vm)
|
||||
disk = self._images.create_vm_disk(assets.rootfs, Path(vm.disk), storage_mib=spec.storage_mib)
|
||||
self._images.provision_credentials(disk, vm.username, vm.password)
|
||||
self._images.configure_guest_network(disk, vm.guest_ip)
|
||||
vm.disk_ready = True
|
||||
self._save(vm)
|
||||
self._boot(vm, assets.kernel)
|
||||
@@ -188,6 +189,7 @@ class LifecycleService:
|
||||
try:
|
||||
self._cleanup_runtime(vm)
|
||||
self._apply_pending_resize(vm)
|
||||
self._images.configure_guest_network(Path(vm.disk), vm.guest_ip)
|
||||
vm.network, vm.bridge = str(self._settings.network), self._settings.bridge
|
||||
self._boot(vm, kernel)
|
||||
return vm
|
||||
|
||||
Reference in New Issue
Block a user