kstyagi/nest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor VM lifecycle management and enhance validation

Browse Source 
- Introduced a new method `installed_kernel` in `ImageStore` to handle kernel verification and checksum validation.
- Updated `Installer` to include additional packages: `nftables` and `conntrack`.
- Refined `LifecycleService` to improve error handling and state management during VM creation, starting, stopping, and deletion.
- Enhanced port management with new methods for setting and syncing ports, including validation for port rules.
- Improved error handling in `StateStore` to raise specific `NotFoundError` and `ConflictError` exceptions.
- Updated network management to ensure proper TAP device handling and firewall integration.
- Enhanced API responses in `vms.py` to include policy status for better client-side handling.
- Added validation for VM names and ports in `validation.py` to enforce stricter rules and prevent conflicts.
This commit is contained in:
kstyagi
2026-09-09 17:27:50 +05:30
parent 4ce14c3378
commit d8a25209b9
31 changed files with 9576 additions and 1331 deletions
Download Patch File Download Diff File Expand all files Collapse all files
FIREWALL.md
+735
View File
@@ -0,0 +1,735 @@
# UVM Local Firewall Design
> Status: proposed design. This document describes a host-local firewall layer
> to be implemented after the VM lifecycle and registry work. It does not
> describe functionality currently available in `uvm`.
## 1. Decision
UVM should use **native nftables**, managed locally through an internal Python
adapter that invokes the `nft` command through the existing `CommandRunner`.
The initial firewall layer will enforce per-VM **private IPv4 TCP allowlists**
at the UVM bridge/TAP boundary. It will not require a cloud firewall, remote
agent, external controller, or long-running UVM firewall daemon.
This choice fits UVM because nftables provides:
- Native Linux bridge filtering, which sees VM-to-VM traffic on `uvm0`.
- Stateful connection tracking in the bridge family on supported kernels.
- Atomic rule transactions through `nft --file -`.
- Named sets and a dedicated table, allowing UVM to update only resources it
owns.
- A local CLI and JSON inspection interface that can be used safely through
UVM's existing subprocess boundary.
The recommended implementation is a focused adapter, not a general firewall
framework or a public Python SDK.
## 2. Scope
### In scope
- Enforce a per-VM TCP allowlist for traffic to the VM's private IPv4 address.
- Cover host-to-VM and VM-to-VM traffic on the existing UVM bridge.
- Preserve ordinary guest-initiated egress and replies to that egress.
- Make `close` block subsequent packets on existing service connections as
well as new connections.
- Install the policy before a VM TAP becomes usable.
- Let explicit lifecycle or policy-sync operations reconcile UVM-owned
firewall state from the local registry after interruption or firewall-rule
loss.
- Work entirely on the Linux host that runs UVM.
### Explicitly out of scope
- Public IP allocation, port forwarding, DNAT, reverse proxies, DNS, TLS
termination, load balancing, or cloud security groups.
- A remote firewall API, external controller, or multi-host coordination.
- Automatic integration with Docker, UFW, firewalld, Kubernetes, libvirt, or
a site-managed firewall. Those need separately tested integration profiles.
- IPv6 service policies, UDP service policies, source-CIDR ACLs, rate limits,
or egress restrictions in the first version.
- A claim that the current Firecracker launch model is a hardened,
multi-tenant isolation boundary.
Opening TCP/80 in this design means that TCP/80 is reachable at the guest's
private UVM address by permitted local paths. It does not create a public
listener, start a guest service, configure TLS, or bypass a guest firewall.
### Supported topology profile
The first version supports a UVM-dedicated private bridge only. Preflight must
verify all of these conditions before activating a managed TAP:
- `UVM_NETWORK` is wholly contained in one RFC1918 range and does not overlap
another local address or non-default route, apart from the expected UVM
bridge address and connected bridge route. The host's generic default route
is expected and is not a conflict.
- `UVM_BRIDGE` has only UVM-owned TAP member ports. It has no physical uplink,
foreign veth, libvirt, container, or otherwise unknown bridge port.
- New inbound service connections originate from the UVM bridge gateway or a
valid address in `UVM_NETWORK`. This is a fixed private-path boundary, not a
user-configurable source-CIDR feature.
- UVM itself installs no DNAT, port-forward, or public-address rule.
The source restriction permits host-to-guest traffic, whose source is normally
the bridge gateway address, and VM-to-VM traffic. It does not affect replies
to guest-initiated external connections, which are identified as reply traffic
through connection tracking.
A private RFC1918 address is not itself a complete public-exposure guarantee.
An administrator can still add host routing, source NAT, DNAT, or another
firewall rule that changes reachability. UVM does not discover or override
such external policy; it only guarantees the scoped bridge policy after it has
successfully applied and verified it.
## 3. Existing UVM Topology
UVM currently creates one TAP per Firecracker VM and attaches every TAP to the
same Linux bridge. The host bridge address is the guest default gateway.
```text
guest eth0
|
Firecracker virtio-net
|
per-VM TAP: uvm-<suffix>
|
Linux bridge: uvm0 (10.42.0.1/24 by default)
|
host IPv4 routing and existing egress NAT
|
host uplink
```
Relevant current code:
- [`uvm/network.py`](uvm/network.py) allocates guest addresses and MACs,
creates the bridge/TAPs, enables IPv4 forwarding, and installs the current
egress `iptables` rules.
- [`uvm/lifecycle.py`](uvm/lifecycle.py) coordinates VM creation, rollback,
stop, and destruction under the shared operation lock.
- [`uvm/state.py`](uvm/state.py) owns the atomic registry and the operation
lock.
- [`uvm/system.py`](uvm/system.py) provides the testable `CommandRunner`
boundary for local host commands.
An IP-family `FORWARD` rule alone is insufficient for the future port policy.
Traffic between two VM TAPs is normally switched by the Linux bridge at layer
2 and does not traverse an `inet`/`ip` forwarding hook. Host-to-guest traffic
also follows bridge output rather than the ordinary routed-forward path.
Native nftables bridge filtering is designed for these paths and does not
require enabling the legacy `br_netfilter` module. The Linux bridge
documentation specifically discourages `br_netfilter` for new filtering
designs in favor of nftables.
## 4. Candidate Evaluation
All options below are open source and can run entirely on the local Linux
host. The comparison is about fit for UVM's intended lifecycle-managed policy,
not whether the tool is generally useful.
| Option | Local operation | Advantages | Limitations for UVM | Decision |
|---|---|---|---|---|
| Native nftables | Kernel Netfilter plus local `nft` CLI; no daemon required | Bridge-family hooks, conntrack, named sets, atomic batches, precise ownership | Requires a supported kernel and careful coexistence rules | Adopt |
| `iptables-nft` compatibility frontend | Local CLI backed by nftables | Already used by existing UVM egress code; familiar syntax | Does not give a clean bridge-policy model and makes native ownership/order harder to reason about | Do not extend for the new layer |
| `iptables-legacy` plus ebtables/ipset | Local CLI tools | Available on older systems | Separate legacy data path, incremental updates, more components, and bridge IP filtering commonly depends on legacy `br_netfilter` behavior | Do not use |
| firewalld | Local daemon with D-Bus API | Zones, policies, IP sets, local programmatic control | Daemon ownership, runtime/permanent configuration, and policy interactions add operational complexity; it is not optimized for UVM's per-TAP hard-close contract | Optional future adapter only |
| UFW | Local Ubuntu-oriented CLI | Simple operator workflow and routed rules | CLI/rule-number model is not a suitable authoritative dynamic policy API; reload/default-policy changes have broad effects | Do not use as UVM's backend |
| eBPF/XDP or tc/eBPF | Local kernel programs and maps | High performance and dynamic maps | Requires a custom data plane, attachment/pinning lifecycle, verifier-compatible programs, and does not naturally provide UVM's conntrack semantics | Future performance option, not v1 |
### Why not reuse the current iptables implementation?
The existing `NetworkManager._ensure_masquerade()` method incrementally adds
egress NAT and forwarding rules. It does not manage a UVM-owned firewall
table, does not filter bridge-forwarded VM-to-VM frames, and does not provide
atomic whole-policy replacement.
On systems where `iptables` points to the nftables compatibility backend,
native nftables and `iptables-nft` share the kernel nftables data plane but
still create separately managed rule structures. Adding the new policy through
native nftables while leaving current egress rules unchanged is the smallest
safe migration. A later, separately tested change may migrate UVM's existing
egress rules into a dedicated nftables table.
UVM must never change a system-wide `iptables` alternative, flush a global
ruleset, or adopt unrelated rules just because they resemble UVM's topology.
### Why not use firewalld or UFW as the primary backend?
firewalld is a valid local-only solution on hosts where it is already the
authoritative firewall manager. Its D-Bus API and policy model make a future
adapter possible. It adds a daemon and a separate permanent/runtime
configuration model, however, and a firewalld reload intentionally preserves
established connections. That does not by itself meet UVM's close-port
contract.
UFW is well suited to static host policy and can express routed rules. Its own
manual states that its command interface is not intended to expose complete
firewall functionality. Dynamic per-VM reconciliation would depend on parsing
or manipulating a higher-level CLI-managed ruleset, and it still would not
solve bridge-path filtering cleanly.
If a host already uses firewalld or UFW, UVM must not write that manager's
generated tables directly. The operator must choose and test a dedicated
integration profile before using UVM's firewall layer on that host.
## 5. Recommended Security Contract
The v1 policy is deliberately narrow:
| Item | Contract |
|---|---|
| Protocol | IPv4 TCP only |
| Inbound access | Default deny for managed VMs; only explicitly allowed guest TCP ports pass |
| Default new-VM policy | Persist TCP/22 as an ordinary initial allowlist entry unless creation explicitly disables default ports |
| Egress | Guest-initiated TCP, UDP, DNS, and required control traffic remain allowed |
| Existing service connections after close | Subsequent packets are blocked in both directions after the policy update commits |
| IPv6 | Drop IPv6 traffic on managed TAPs until an explicit IPv6 policy exists |
| Guest identity | Bind each managed TAP to its recorded MAC and IPv4 address; reject spoofed source identity |
| New inbound source | Only the UVM bridge gateway or an address in `UVM_NETWORK` may initiate a service connection |
| Legacy records | A missing policy is `unmanaged`, not an implicit allow-all or deny-all policy |
The policy must distinguish an explicit empty list from a legacy record:
```text
open_tcp_ports = None -> legacy/unmanaged; do not claim enforcement
open_tcp_ports = [] -> managed deny-all inbound policy
open_tcp_ports = [22] -> managed policy allowing SSH only
```
HTTP and HTTPS are user-facing labels for TCP/80 and TCP/443. They do not
change packet treatment.
An active unmanaged VM may not share `uvm0` with managed VMs. Before managed
enforcement is enabled, all active legacy records must be stopped and migrated
to an explicit policy, or the operation must fail. This avoids treating an
unfiltered legacy TAP as a trustworthy peer for MAC/IP/ARP identity checks.
### What close guarantees
When a managed VM closes TCP/443, the resulting policy must match both:
- Original-direction packets traveling to that VM's TCP/443 service.
- Reply-direction packets leaving that service on an already established flow.
The implementation must not place a broad `ct state established,related
accept` rule before this service-policy check. Instead, it checks the
connection's original destination address and destination port for every
service-flow packet.
This gives the following guarantee:
> Once the atomic firewall update commits, later packets traversing the VM's
> service path are blocked while the port remains closed.
It cannot retract data already delivered, gracefully close a guest process, or
guarantee an immediately visible close for a completely idle TCP socket. The
design may use `reject with tcp reset` after packet-path tests establish that
the behavior is appropriate; a drop rule is the conservative baseline.
Connection tracking must remain available while enforcing a live close. A
targeted `conntrack` cleanup is useful before a deleted VM IP is reused, but
it is not the primary live-close mechanism. Deleting a conntrack entry too
early can discard the original tuple needed to distinguish a service reply
from a new guest-originated connection.
## 6. nftables Architecture
UVM will own one dedicated bridge-family table, conceptually:
```text
table bridge uvm
```
The table contains only UVM-generated rules, sets, and chains. It must never
use `flush ruleset`, modify a built-in host policy, or clear tables owned by a
different tool.
```mermaid
flowchart LR
LIFE[LifecycleService] --> FIREWALL[NftFirewall]
LIFE --> NETWORK[NetworkManager]
FIREWALL --> RUNNER[CommandRunner]
RUNNER --> NFT[nftables bridge table]
NETWORK --> BRIDGE[uvm0 bridge and TAPs]
NFT --> BRIDGE
BRIDGE --> VM1[Firecracker VM A]
BRIDGE --> VM2[Firecracker VM B]
```
### Hook Coverage
| Bridge hook | UVM use | Required policy action |
|---|---|---|
| `prerouting` | Frame arrives from a managed TAP | Validate expected source MAC/IP before bridge learning; allow only necessary ARP; reject unsupported IPv6 |
| `input` | VM frame enters the host stack, including the gateway path | Apply the VM's source-side policy and allow guest-originated traffic/relevant replies |
| `output` | Host stack sends a frame toward a VM | Apply the destination VM's inbound policy or allow a reply to guest-originated traffic |
| `forward` | Frame moves directly between bridge ports | Apply both source-VM and destination-VM policy; never let one VM's acceptance skip the other VM's check |
The base chains have an `accept` policy so that UVM does not impose a global
deny policy on non-UVM bridge ports. The UVM dispatch rules select only known,
registry-backed TAP names. A denial occurs inside a selected VM's policy path.
Once a frame is selected as to or from a managed TAP, an unmatched frame is a
terminal denial. The v1 policy permits only untagged IPv4 and the narrowly
required ARP traffic on that attachment. It drops IPv6, VLAN-tagged frames,
and other unsupported EtherTypes in both directions. This is separate from
the base-chain policy for non-UVM bridge ports.
For VM-to-VM traffic, dispatch into source and destination policy chains must
use `jump` and `return` semantics. A source-side success returns to the
dispatcher so the destination VM's inbound policy still runs. It must not use
an early terminal `accept` that bypasses the destination check.
### Per-VM Rule Model
Each managed VM contributes a typed policy record:
```text
VM ID
TAP name
guest IPv4 address
guest MAC address
open TCP ports
runtime mode: active or quarantined
```
The generated rules maintain two logical checks:
| Chain role | Original-direction traffic | Reply-direction traffic |
|---|---|---|
| Toward VM | Allow a new TCP flow only from the private-path sources and when its destination port is in the VM's current allowlist | Allow a reply only when it belongs to a guest-originated connection |
| From VM | Allow guest-originated traffic | Allow a service reply only when the original destination VM/port is still in the allowlist |
The implementation uses connection tracking's original tuple, rather than the
packet's current source/destination port alone, for service replies. For
example, a web server reply is sent *from* TCP/443 to a client ephemeral port;
the original tuple still identifies TCP/443 as the guest service.
The actual nft syntax is an implementation detail and must be generated from
typed data. It will use `ct direction`, `ct original ip daddr`, and `ct
original proto-dst` only after an isolated integration test verifies the
target kernel's bridge conntrack behavior.
### Identity and Layer-2 Controls
Filtering only TCP ports is not enough if a guest can spoof another guest's
source identity. For every managed TAP, the policy must:
- Accept a frame from the TAP only when its Ethernet source equals the
persisted guest MAC.
- Accept an IPv4 frame from the TAP only when its source equals the persisted
guest IPv4 address.
- Bind ARP sender hardware and protocol addresses from the TAP to that same
MAC/IP pair; allow only the required target/address-probe forms after packet
testing.
- Drop IPv6, tagged Ethernet frames, and unsupported EtherTypes both to and
from the managed TAP while they have no corresponding policy.
The identity check belongs before or at the earliest bridge ingress point. It
must be validated against the actual bridge forwarding database behavior on
the target kernel. This is meaningful network-policy hardening, but it does
not turn UVM into a complete hostile-tenant platform. Stronger tenant
isolation may eventually require per-VM L2/L3 segments, bridge-port
isolation, VLANs, or network namespaces.
### Ownership and Rule Rendering
UVM owns only its dedicated table and resources carrying a fixed versioned
UVM marker. The adapter must:
1. Refuse an unexpected pre-existing UVM table or an ownership-marker
mismatch rather than adopting it.
2. Render the complete desired contents of its own table from current state.
3. Apply the generated table in one nftables transaction.
4. Inspect the owned table after a mutation when a stronger confirmation is
required.
5. Remove a per-VM dispatch entry only after its TAP is confirmed absent or
safely quarantined.
The exact table replacement batch must be tested against the supported nft
version. It may rebuild only `table bridge uvm`; it must never rebuild the
host's whole nftables ruleset.
## 7. Persistence and Reconciliation
The JSON registry remains the source of desired policy. Kernel firewall state
is observed state and is never treated as durable truth.
The future `VmRecord` model needs to persist at least:
```text
name (if VM naming is added)
open_tcp_ports: list[int] | None
policy_intent / pending lifecycle phase
policy generation
```
`None` preserves the legacy/unmanaged distinction. New VMs always have an
explicit policy, including a deliberately empty policy.
The registry also needs a topology fingerprint at the state level containing
the configured network CIDR, gateway, and bridge name. Lifecycle and firewall
preflight compare the current `Settings` against that persisted fingerprint
before touching interfaces or rules. A legacy registry without this fingerprint
requires an explicit migration; it must not infer that a changed `/24` and
`/16`, for example, describe the same UVM topology.
UVM should not persist an authoritative `policy_enforced: true` boolean. It
would become stale after a reboot, manual firewall change, partial failure, or
host firewall reload. Instead, `NftFirewall.reconcile()` renders the desired
state from the registry and inspects the UVM-owned table when needed.
Reconciliation has no continuous trigger in this design. It runs during
create, start, port mutations, and an explicit future policy-sync operation.
Read-only list/inspect commands may report an owned table as missing or
unverified, but must not silently mutate the host. If an active VM loses its
UVM firewall table between operations, its policy is unknown until a successful
explicit reconciliation; UVM must not claim it remains enforced.
### Mutation Ordering
All firewall mutations use the existing `operations.lock`. State and kernel
updates cannot be one cross-resource transaction, so durable intent and
observed enforcement must remain distinct:
| Action | Required order |
|---|---|
| Create | Persist explicit desired policy, create the TAP down, install/verify policy, activate TAP, then launch Firecracker |
| Start | Reconcile the saved policy before activating the retained TAP or launching a new VMM |
| Open port | Persist desired policy first, atomically apply it, then clear the pending state |
| Close port | Persist the more restrictive desired policy first, atomically apply it, then clear the pending state |
| Stop | Persist stop intent, quarantine the TAP, confirm VMM exit, remove/deactivate TAP, retain saved policy |
| Delete | Quarantine, confirm exit, remove TAP, remove UVM policy for the VM, clear stale flow state before IP reuse, then remove registry data |
If an open operation cannot be applied, the desired state may say "open" while
the host remains more restrictive. UVM reports failure and retries through
reconciliation; it must not claim the port is open. If a close operation
cannot be confirmed, UVM retains its restrictive intent and attempts to
quarantine the attachment rather than restoring the old policy.
There is an unavoidable crash window after restrictive intent is persisted and
before the nftables transaction commits. During that window, an already-running
VM can still have its old permissive kernel policy. An interrupted close stays
pending and is not reported as enforced until an explicit reconciliation or
quarantine successfully commits. The operation lock cannot close this window
after the UVM process exits.
### Quarantine
An empty inbound allowlist is not a quarantine because it still permits a VM
to initiate outbound traffic. A quarantine mode drops traffic both to and from
the owned TAP. It is required while a stop, delete, or failed boot leaves
process/TAP ownership unresolved.
Do not remove this guard merely because a cleanup operation was attempted. It
remains until the TAP is gone or a successful start replaces it with the
saved active policy.
## 8. Python Integration
### Internal Adapter Boundary
Add a focused internal module such as `uvm/firewall.py`. It should be
constructed in [`uvm/app.py`](uvm/app.py), similarly to the current
`NetworkManager` and Firecracker process manager.
```python
firewall = NftFirewall(settings, runner)
lifecycle = LifecycleService(settings, state_store, images, network, process, firewall)
```
The exact constructor arrangement can vary, but these boundaries must remain:
| Component | Responsibility |
|---|---|
| `LifecycleService` | Owns ordering, operation locking, persistence intent, rollback, and VM status transitions |
| `NetworkManager` | Owns bridge validation plus TAP preparation, activation, and deletion |
| `NftFirewall` | Owns only UVM's generated nftables policy and its local inspection/reconciliation |
| `CommandRunner` | Runs local commands without a shell and makes command construction testable |
| CLI/API routers | Validate user input and call services; never send raw firewall syntax or commands |
The adapter should expose typed operations, for example:
```python
@dataclass(frozen=True, slots=True)
class VmFirewallPolicy:
vm_id: str
tap: str
guest_ip: IPv4Address
guest_mac: str
open_tcp_ports: tuple[int, ...]
quarantined: bool = False
class NftFirewall:
def preflight(self) -> None: ...
def reconcile(self, policies: Iterable[VmFirewallPolicy]) -> FirewallObservation: ...
def inspect(self) -> FirewallObservation: ...
def purge_vm_flows(self, vm: VmFirewallPolicy) -> None: ...
```
The public surface should accept domain objects, not interface names or raw
rules supplied by a CLI/API caller. There should be no `apply_raw`, `flush`,
or generic arbitrary-command method.
### Command Transport
The first implementation should call the local `nft` executable through
`CommandRunner`, rather than add a Python package dependency:
```python
runner.run(
("nft", "--check", "--file", "-"),
input_text=ruleset,
capture=True,
timeout=settings.firewall_timeout_s,
)
runner.run(
("nft", "--file", "-"),
input_text=ruleset,
capture=True,
timeout=settings.firewall_timeout_s,
)
```
This follows the same engineering model as Firecracker's Unix-socket adapter:
a narrow typed UVM boundary hides an external host interface. For nftables,
the host interface is a local command/Netfilter transaction rather than HTTP.
`nft --check` and `nft --file -` are separate operations. The checked batch is
only a validation aid; UVM may claim a policy change only after the apply
transaction succeeds and the adapter records its observation.
Benefits of the CLI approach:
- No shell interpolation or raw shell scripts.
- Atomic batch application through `nft --file -`.
- Syntax/preflight checking through `nft --check`.
- Easy fake-runner unit tests and visible CLI traces.
- No additional Python runtime dependency.
Inspection should use JSON rather than human-formatted output:
```python
runner.run(
("nft", "--json", "list", "table", "bridge", "uvm"),
capture=True,
timeout=settings.firewall_timeout_s,
)
```
`python3-nftables`/libnftables is a viable future transport if avoiding a
subprocess becomes necessary. It must not be introduced casually: package API
and kernel compatibility need a version-pinned integration suite. Direct
Netlink programming is not justified for the first implementation.
### Errors
Introduce a `FirewallError(UvmError)` for semantic firewall failures such as
unsupported local capabilities, unexpected ownership, or an unsafe observed
table. Preserve `CommandError` for an `nft` or `conntrack` command failure.
Errors must include the failed operation and safe diagnostics, but never
include raw untrusted rule input because the adapter has none. The CLI and API
can retain their existing `UvmError` rendering conventions.
## 9. Installation and Host Requirements
The installer remains the only component that invokes `apt-get`. The future
package list should add:
```text
nftables
conntrack
```
`conntrack` is used for targeted stale-flow cleanup before a deleted VM IP is
reallocated. It is not a substitute for the firewall's live close policy.
On Ubuntu-compatible hosts, the installation step is conceptually:
```sh
apt-get install --yes nftables conntrack
```
Do not automatically enable `nftables.service`, write `/etc/nftables.conf`,
or install a system-wide persistence service. UVM dynamically owns only its
dedicated table and recreates/reconciles it during lifecycle operations.
### Required Host Capabilities
| Requirement | Reason |
|---|---|
| Linux with native nftables support | UVM uses local Netfilter tables |
| Kernel with bridge-family connection tracking support | Required for original-tuple service checks; upstream support starts in Linux 5.3 |
| `nft`, `conntrack`, and `ip` commands | Local policy, selected flow cleanup, and TAP/bridge management |
| `CONFIG_BRIDGE` and `CONFIG_TUN` | Existing UVM bridge/TAP topology |
| Netfilter nftables, conntrack, bridge, reject, and relevant protocol modules | Required to load/apply the generated table |
| Root or `CAP_NET_ADMIN` in the bridge network namespace | Interface and firewall administration |
| No flowtable/hardware offload that bypasses this filter path | Required for reliable close enforcement |
Preflight should verify the actual generated ruleset with `nft --check` on the
target host. Kernel-version checks alone are not enough; distributions can
build or package Netfilter capabilities differently.
UVM must not enable `br_netfilter` or change `net.bridge.bridge-nf-call-*`
sysctls. Native bridge nftables works without that legacy compatibility layer.
## 10. Configuration Approach
The first version should avoid a generic `UVM_FIREWALL_RULES` setting or any
configuration value containing raw nftables syntax.
The generated policy derives from existing UVM configuration and state:
| Source | Firewall use |
|---|---|
| `UVM_NETWORK` | Validates guest IPv4 addresses and UVM subnet membership |
| `UVM_GATEWAY` | Establishes the bridge/guest topology for ARP and gateway traffic |
| `UVM_BRIDGE` | Selects the UVM bridge and its managed TAP attachments |
| `VmRecord.tap`, `VmRecord.mac`, `VmRecord.guest_ip` | Binds policy to one owned VM attachment |
| Persisted `open_tcp_ports` | Creates the per-VM TCP allowlist |
One bounded command timeout, such as `UVM_FIREWALL_TIMEOUT_S`, is reasonable
as a `Settings` value. It should default to a small value and apply only to
UVM's local command invocations.
Changing `UVM_NETWORK`, `UVM_GATEWAY`, or `UVM_BRIDGE` after VMs exist is not
a firewall migration mechanism. UVM must fail rather than infer ownership of
old interfaces or rules from changed environment values.
## 11. Lifecycle Integration
The existing `NetworkManager.create_tap()` activates a TAP immediately. The
firewall layer requires that operation to be split into preparation and
activation:
```text
create TAP down -> attach to bridge -> install policy -> bring TAP up
```
### Create and Start
```mermaid
sequenceDiagram
participant L as LifecycleService
participant S as StateStore
participant N as NetworkManager
participant F as NftFirewall
participant P as Firecracker
L->>S: persist VM and explicit policy intent
L->>N: ensure bridge; prepare TAP down
L->>F: reconcile active policy including new VM
F-->>L: policy confirmed
L->>N: activate TAP
L->>P: spawn, configure, and start VMM
L->>S: persist running state
```
A policy failure prevents TAP activation and Firecracker launch. A startup
failure after policy installation leaves the policy in place until lifecycle
cleanup confirms that the TAP is gone. This is safer than briefly booting an
unfiltered guest.
### Stop, Delete, and Failure Cleanup
```text
stop/delete intent
-> quarantine owned TAP
-> terminate verified Firecracker process and confirm exit
-> deactivate/delete TAP
-> stop: retain desired port policy for a later start
-> delete: remove VM policy, purge stale flows, remove disk/state
```
If process identity or TAP ownership is unresolved, keep the quarantine guard,
disk, and registry entry. Do not delete a possibly live VM disk or remove its
network guard simply because a cleanup command raised an error.
Shared table infrastructure may remain after the last VM is deleted. It is
not a resource leak: it is a UVM-owned baseline that can be reconciled by the
next lifecycle operation. It must contain no stale per-VM dispatch/policy
entries after successful deletion.
## 12. Coexistence and Security Boundaries
The initial supported profile is a host where UVM may own `table bridge uvm`
and no other component modifies that table, using the dedicated private bridge
profile described above.
This does not mean UVM owns the host firewall. In particular:
- A terminal `drop` in UVM's bridge table can protect a UVM TAP.
- An `accept` in one nftables base chain does not promise that later hooks or
another firewall manager will accept the packet.
- UVM must not change global default policies, Docker chains, firewalld zones,
UFW framework files, or a host's `nftables.service` configuration.
- If Docker, Podman, UFW, firewalld, libvirt, or site firewall tooling is
active, allowed traffic may still be blocked by that tooling. UVM should
report its own enforcement state honestly rather than claim reachability.
The present design controls the local private network path only. It does not
protect against a privileged host process, a guest root exploit, raw host
network administration, or future external DNAT rules. Source-IP allowlists
and public exposure, if added, require a separate `inet`-family and ingress
design.
## 13. Testing and Release Gates
### Unit Tests
Add a focused `tests/test_firewall.py` with a stateful fake runner. Test:
- Generated table ownership, no `flush ruleset`, and no built-in chain policy
changes.
- Exact TAP-scoped dispatch and VM policy rendering.
- One atomic batch per policy update.
- Validation failures before host mutation.
- Idempotent reconciliation and recovery from missing UVM-owned rules.
- Refusal to adopt an unexpected table or marker mismatch.
- No broad `established,related` acceptance before service policy checks.
- Close-policy rendering that blocks both request and reply service packets.
- Quarantine and policy retention across stop/start/delete.
- Lifecycle ordering: policy before TAP activation and Firecracker spawn.
- Failure injection at state, nft, process, TAP, and cleanup boundaries.
### Isolated Network-Namespace Proof
Before integrating with real Firecracker, build a disposable network-namespace
test using a bridge and veth peers standing in for TAPs. It must prove:
1. Host-to-VM traffic reaches an allowed TCP port and fails at a closed port.
2. VM-to-VM traffic is governed by the destination VM's policy.
3. Guest-initiated TCP/UDP egress and replies still work.
4. Closing a port during one-way server streaming blocks subsequent reply
packets as well as new requests.
5. A guest outbound connection using a source port equal to a closed service
port is not falsely blocked as inbound service traffic.
6. ARP/gateway behavior remains functional.
7. IPv6 cannot bypass the IPv4-only policy.
8. MAC/IP spoofing from a managed TAP is rejected.
9. Tagged IPv4/IPv6 and unsupported EtherTypes are denied at managed TAPs.
10. An active unmanaged TAP prevents managed-policy activation.
11. Unrelated bridge ports and host firewall tables are unaffected.
Only after that proof should an isolated Firecracker test verify real TAP/MAC
behavior, default SSH access, an allowed guest listener, and live close.
## 14. References
- [Linux bridge Netfilter documentation](https://docs.kernel.org/networking/bridge.html#netfilter)
- [nftables bridge filtering](https://wiki.nftables.org/wiki-nftables/index.php/Bridge_filtering)
- [nftables atomic rule replacement](https://wiki.nftables.org/wiki-nftables/index.php/Atomic_rule_replacement)
- [nftables connection-tracking metadata](https://wiki.nftables.org/wiki-nftables/index.php/Matching_connection_tracking_stateful_metainformation)
- [firewalld concepts and policies](https://firewalld.org/documentation/concepts.html)
- [UFW manual](https://manpages.ubuntu.com/manpages/noble/man8/ufw.8.html)
- [iptables manual](https://man7.org/linux/man-pages/man8/iptables.8.html)
- [Linux `bpf(2)` manual](https://man7.org/linux/man-pages/man2/bpf.2.html)