- Introduced a new method `installed_kernel` in `ImageStore` to handle kernel verification and checksum validation. - Updated `Installer` to include additional packages: `nftables` and `conntrack`. - Refined `LifecycleService` to improve error handling and state management during VM creation, starting, stopping, and deletion. - Enhanced port management with new methods for setting and syncing ports, including validation for port rules. - Improved error handling in `StateStore` to raise specific `NotFoundError` and `ConflictError` exceptions. - Updated network management to ensure proper TAP device handling and firewall integration. - Enhanced API responses in `vms.py` to include policy status for better client-side handling. - Added validation for VM names and ports in `validation.py` to enforce stricter rules and prevent conflicts.
736 lines
33 KiB
Markdown
736 lines
33 KiB
Markdown
# UVM Local Firewall Design
|
|
|
|
> Status: proposed design. This document describes a host-local firewall layer
|
|
> to be implemented after the VM lifecycle and registry work. It does not
|
|
> describe functionality currently available in `uvm`.
|
|
|
|
## 1. Decision
|
|
|
|
UVM should use **native nftables**, managed locally through an internal Python
|
|
adapter that invokes the `nft` command through the existing `CommandRunner`.
|
|
|
|
The initial firewall layer will enforce per-VM **private IPv4 TCP allowlists**
|
|
at the UVM bridge/TAP boundary. It will not require a cloud firewall, remote
|
|
agent, external controller, or long-running UVM firewall daemon.
|
|
|
|
This choice fits UVM because nftables provides:
|
|
|
|
- Native Linux bridge filtering, which sees VM-to-VM traffic on `uvm0`.
|
|
- Stateful connection tracking in the bridge family on supported kernels.
|
|
- Atomic rule transactions through `nft --file -`.
|
|
- Named sets and a dedicated table, allowing UVM to update only resources it
|
|
owns.
|
|
- A local CLI and JSON inspection interface that can be used safely through
|
|
UVM's existing subprocess boundary.
|
|
|
|
The recommended implementation is a focused adapter, not a general firewall
|
|
framework or a public Python SDK.
|
|
|
|
## 2. Scope
|
|
|
|
### In scope
|
|
|
|
- Enforce a per-VM TCP allowlist for traffic to the VM's private IPv4 address.
|
|
- Cover host-to-VM and VM-to-VM traffic on the existing UVM bridge.
|
|
- Preserve ordinary guest-initiated egress and replies to that egress.
|
|
- Make `close` block subsequent packets on existing service connections as
|
|
well as new connections.
|
|
- Install the policy before a VM TAP becomes usable.
|
|
- Let explicit lifecycle or policy-sync operations reconcile UVM-owned
|
|
firewall state from the local registry after interruption or firewall-rule
|
|
loss.
|
|
- Work entirely on the Linux host that runs UVM.
|
|
|
|
### Explicitly out of scope
|
|
|
|
- Public IP allocation, port forwarding, DNAT, reverse proxies, DNS, TLS
|
|
termination, load balancing, or cloud security groups.
|
|
- A remote firewall API, external controller, or multi-host coordination.
|
|
- Automatic integration with Docker, UFW, firewalld, Kubernetes, libvirt, or
|
|
a site-managed firewall. Those need separately tested integration profiles.
|
|
- IPv6 service policies, UDP service policies, source-CIDR ACLs, rate limits,
|
|
or egress restrictions in the first version.
|
|
- A claim that the current Firecracker launch model is a hardened,
|
|
multi-tenant isolation boundary.
|
|
|
|
Opening TCP/80 in this design means that TCP/80 is reachable at the guest's
|
|
private UVM address by permitted local paths. It does not create a public
|
|
listener, start a guest service, configure TLS, or bypass a guest firewall.
|
|
|
|
### Supported topology profile
|
|
|
|
The first version supports a UVM-dedicated private bridge only. Preflight must
|
|
verify all of these conditions before activating a managed TAP:
|
|
|
|
- `UVM_NETWORK` is wholly contained in one RFC1918 range and does not overlap
|
|
another local address or non-default route, apart from the expected UVM
|
|
bridge address and connected bridge route. The host's generic default route
|
|
is expected and is not a conflict.
|
|
- `UVM_BRIDGE` has only UVM-owned TAP member ports. It has no physical uplink,
|
|
foreign veth, libvirt, container, or otherwise unknown bridge port.
|
|
- New inbound service connections originate from the UVM bridge gateway or a
|
|
valid address in `UVM_NETWORK`. This is a fixed private-path boundary, not a
|
|
user-configurable source-CIDR feature.
|
|
- UVM itself installs no DNAT, port-forward, or public-address rule.
|
|
|
|
The source restriction permits host-to-guest traffic, whose source is normally
|
|
the bridge gateway address, and VM-to-VM traffic. It does not affect replies
|
|
to guest-initiated external connections, which are identified as reply traffic
|
|
through connection tracking.
|
|
|
|
A private RFC1918 address is not itself a complete public-exposure guarantee.
|
|
An administrator can still add host routing, source NAT, DNAT, or another
|
|
firewall rule that changes reachability. UVM does not discover or override
|
|
such external policy; it only guarantees the scoped bridge policy after it has
|
|
successfully applied and verified it.
|
|
|
|
## 3. Existing UVM Topology
|
|
|
|
UVM currently creates one TAP per Firecracker VM and attaches every TAP to the
|
|
same Linux bridge. The host bridge address is the guest default gateway.
|
|
|
|
```text
|
|
guest eth0
|
|
|
|
|
Firecracker virtio-net
|
|
|
|
|
per-VM TAP: uvm-<suffix>
|
|
|
|
|
Linux bridge: uvm0 (10.42.0.1/24 by default)
|
|
|
|
|
host IPv4 routing and existing egress NAT
|
|
|
|
|
host uplink
|
|
```
|
|
|
|
Relevant current code:
|
|
|
|
- [`uvm/network.py`](uvm/network.py) allocates guest addresses and MACs,
|
|
creates the bridge/TAPs, enables IPv4 forwarding, and installs the current
|
|
egress `iptables` rules.
|
|
- [`uvm/lifecycle.py`](uvm/lifecycle.py) coordinates VM creation, rollback,
|
|
stop, and destruction under the shared operation lock.
|
|
- [`uvm/state.py`](uvm/state.py) owns the atomic registry and the operation
|
|
lock.
|
|
- [`uvm/system.py`](uvm/system.py) provides the testable `CommandRunner`
|
|
boundary for local host commands.
|
|
|
|
An IP-family `FORWARD` rule alone is insufficient for the future port policy.
|
|
Traffic between two VM TAPs is normally switched by the Linux bridge at layer
|
|
2 and does not traverse an `inet`/`ip` forwarding hook. Host-to-guest traffic
|
|
also follows bridge output rather than the ordinary routed-forward path.
|
|
|
|
Native nftables bridge filtering is designed for these paths and does not
|
|
require enabling the legacy `br_netfilter` module. The Linux bridge
|
|
documentation specifically discourages `br_netfilter` for new filtering
|
|
designs in favor of nftables.
|
|
|
|
## 4. Candidate Evaluation
|
|
|
|
All options below are open source and can run entirely on the local Linux
|
|
host. The comparison is about fit for UVM's intended lifecycle-managed policy,
|
|
not whether the tool is generally useful.
|
|
|
|
| Option | Local operation | Advantages | Limitations for UVM | Decision |
|
|
|---|---|---|---|---|
|
|
| Native nftables | Kernel Netfilter plus local `nft` CLI; no daemon required | Bridge-family hooks, conntrack, named sets, atomic batches, precise ownership | Requires a supported kernel and careful coexistence rules | Adopt |
|
|
| `iptables-nft` compatibility frontend | Local CLI backed by nftables | Already used by existing UVM egress code; familiar syntax | Does not give a clean bridge-policy model and makes native ownership/order harder to reason about | Do not extend for the new layer |
|
|
| `iptables-legacy` plus ebtables/ipset | Local CLI tools | Available on older systems | Separate legacy data path, incremental updates, more components, and bridge IP filtering commonly depends on legacy `br_netfilter` behavior | Do not use |
|
|
| firewalld | Local daemon with D-Bus API | Zones, policies, IP sets, local programmatic control | Daemon ownership, runtime/permanent configuration, and policy interactions add operational complexity; it is not optimized for UVM's per-TAP hard-close contract | Optional future adapter only |
|
|
| UFW | Local Ubuntu-oriented CLI | Simple operator workflow and routed rules | CLI/rule-number model is not a suitable authoritative dynamic policy API; reload/default-policy changes have broad effects | Do not use as UVM's backend |
|
|
| eBPF/XDP or tc/eBPF | Local kernel programs and maps | High performance and dynamic maps | Requires a custom data plane, attachment/pinning lifecycle, verifier-compatible programs, and does not naturally provide UVM's conntrack semantics | Future performance option, not v1 |
|
|
|
|
### Why not reuse the current iptables implementation?
|
|
|
|
The existing `NetworkManager._ensure_masquerade()` method incrementally adds
|
|
egress NAT and forwarding rules. It does not manage a UVM-owned firewall
|
|
table, does not filter bridge-forwarded VM-to-VM frames, and does not provide
|
|
atomic whole-policy replacement.
|
|
|
|
On systems where `iptables` points to the nftables compatibility backend,
|
|
native nftables and `iptables-nft` share the kernel nftables data plane but
|
|
still create separately managed rule structures. Adding the new policy through
|
|
native nftables while leaving current egress rules unchanged is the smallest
|
|
safe migration. A later, separately tested change may migrate UVM's existing
|
|
egress rules into a dedicated nftables table.
|
|
|
|
UVM must never change a system-wide `iptables` alternative, flush a global
|
|
ruleset, or adopt unrelated rules just because they resemble UVM's topology.
|
|
|
|
### Why not use firewalld or UFW as the primary backend?
|
|
|
|
firewalld is a valid local-only solution on hosts where it is already the
|
|
authoritative firewall manager. Its D-Bus API and policy model make a future
|
|
adapter possible. It adds a daemon and a separate permanent/runtime
|
|
configuration model, however, and a firewalld reload intentionally preserves
|
|
established connections. That does not by itself meet UVM's close-port
|
|
contract.
|
|
|
|
UFW is well suited to static host policy and can express routed rules. Its own
|
|
manual states that its command interface is not intended to expose complete
|
|
firewall functionality. Dynamic per-VM reconciliation would depend on parsing
|
|
or manipulating a higher-level CLI-managed ruleset, and it still would not
|
|
solve bridge-path filtering cleanly.
|
|
|
|
If a host already uses firewalld or UFW, UVM must not write that manager's
|
|
generated tables directly. The operator must choose and test a dedicated
|
|
integration profile before using UVM's firewall layer on that host.
|
|
|
|
## 5. Recommended Security Contract
|
|
|
|
The v1 policy is deliberately narrow:
|
|
|
|
| Item | Contract |
|
|
|---|---|
|
|
| Protocol | IPv4 TCP only |
|
|
| Inbound access | Default deny for managed VMs; only explicitly allowed guest TCP ports pass |
|
|
| Default new-VM policy | Persist TCP/22 as an ordinary initial allowlist entry unless creation explicitly disables default ports |
|
|
| Egress | Guest-initiated TCP, UDP, DNS, and required control traffic remain allowed |
|
|
| Existing service connections after close | Subsequent packets are blocked in both directions after the policy update commits |
|
|
| IPv6 | Drop IPv6 traffic on managed TAPs until an explicit IPv6 policy exists |
|
|
| Guest identity | Bind each managed TAP to its recorded MAC and IPv4 address; reject spoofed source identity |
|
|
| New inbound source | Only the UVM bridge gateway or an address in `UVM_NETWORK` may initiate a service connection |
|
|
| Legacy records | A missing policy is `unmanaged`, not an implicit allow-all or deny-all policy |
|
|
|
|
The policy must distinguish an explicit empty list from a legacy record:
|
|
|
|
```text
|
|
open_tcp_ports = None -> legacy/unmanaged; do not claim enforcement
|
|
open_tcp_ports = [] -> managed deny-all inbound policy
|
|
open_tcp_ports = [22] -> managed policy allowing SSH only
|
|
```
|
|
|
|
HTTP and HTTPS are user-facing labels for TCP/80 and TCP/443. They do not
|
|
change packet treatment.
|
|
|
|
An active unmanaged VM may not share `uvm0` with managed VMs. Before managed
|
|
enforcement is enabled, all active legacy records must be stopped and migrated
|
|
to an explicit policy, or the operation must fail. This avoids treating an
|
|
unfiltered legacy TAP as a trustworthy peer for MAC/IP/ARP identity checks.
|
|
|
|
### What close guarantees
|
|
|
|
When a managed VM closes TCP/443, the resulting policy must match both:
|
|
|
|
- Original-direction packets traveling to that VM's TCP/443 service.
|
|
- Reply-direction packets leaving that service on an already established flow.
|
|
|
|
The implementation must not place a broad `ct state established,related
|
|
accept` rule before this service-policy check. Instead, it checks the
|
|
connection's original destination address and destination port for every
|
|
service-flow packet.
|
|
|
|
This gives the following guarantee:
|
|
|
|
> Once the atomic firewall update commits, later packets traversing the VM's
|
|
> service path are blocked while the port remains closed.
|
|
|
|
It cannot retract data already delivered, gracefully close a guest process, or
|
|
guarantee an immediately visible close for a completely idle TCP socket. The
|
|
design may use `reject with tcp reset` after packet-path tests establish that
|
|
the behavior is appropriate; a drop rule is the conservative baseline.
|
|
|
|
Connection tracking must remain available while enforcing a live close. A
|
|
targeted `conntrack` cleanup is useful before a deleted VM IP is reused, but
|
|
it is not the primary live-close mechanism. Deleting a conntrack entry too
|
|
early can discard the original tuple needed to distinguish a service reply
|
|
from a new guest-originated connection.
|
|
|
|
## 6. nftables Architecture
|
|
|
|
UVM will own one dedicated bridge-family table, conceptually:
|
|
|
|
```text
|
|
table bridge uvm
|
|
```
|
|
|
|
The table contains only UVM-generated rules, sets, and chains. It must never
|
|
use `flush ruleset`, modify a built-in host policy, or clear tables owned by a
|
|
different tool.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
LIFE[LifecycleService] --> FIREWALL[NftFirewall]
|
|
LIFE --> NETWORK[NetworkManager]
|
|
FIREWALL --> RUNNER[CommandRunner]
|
|
RUNNER --> NFT[nftables bridge table]
|
|
NETWORK --> BRIDGE[uvm0 bridge and TAPs]
|
|
NFT --> BRIDGE
|
|
BRIDGE --> VM1[Firecracker VM A]
|
|
BRIDGE --> VM2[Firecracker VM B]
|
|
```
|
|
|
|
### Hook Coverage
|
|
|
|
| Bridge hook | UVM use | Required policy action |
|
|
|---|---|---|
|
|
| `prerouting` | Frame arrives from a managed TAP | Validate expected source MAC/IP before bridge learning; allow only necessary ARP; reject unsupported IPv6 |
|
|
| `input` | VM frame enters the host stack, including the gateway path | Apply the VM's source-side policy and allow guest-originated traffic/relevant replies |
|
|
| `output` | Host stack sends a frame toward a VM | Apply the destination VM's inbound policy or allow a reply to guest-originated traffic |
|
|
| `forward` | Frame moves directly between bridge ports | Apply both source-VM and destination-VM policy; never let one VM's acceptance skip the other VM's check |
|
|
|
|
The base chains have an `accept` policy so that UVM does not impose a global
|
|
deny policy on non-UVM bridge ports. The UVM dispatch rules select only known,
|
|
registry-backed TAP names. A denial occurs inside a selected VM's policy path.
|
|
|
|
Once a frame is selected as to or from a managed TAP, an unmatched frame is a
|
|
terminal denial. The v1 policy permits only untagged IPv4 and the narrowly
|
|
required ARP traffic on that attachment. It drops IPv6, VLAN-tagged frames,
|
|
and other unsupported EtherTypes in both directions. This is separate from
|
|
the base-chain policy for non-UVM bridge ports.
|
|
|
|
For VM-to-VM traffic, dispatch into source and destination policy chains must
|
|
use `jump` and `return` semantics. A source-side success returns to the
|
|
dispatcher so the destination VM's inbound policy still runs. It must not use
|
|
an early terminal `accept` that bypasses the destination check.
|
|
|
|
### Per-VM Rule Model
|
|
|
|
Each managed VM contributes a typed policy record:
|
|
|
|
```text
|
|
VM ID
|
|
TAP name
|
|
guest IPv4 address
|
|
guest MAC address
|
|
open TCP ports
|
|
runtime mode: active or quarantined
|
|
```
|
|
|
|
The generated rules maintain two logical checks:
|
|
|
|
| Chain role | Original-direction traffic | Reply-direction traffic |
|
|
|---|---|---|
|
|
| Toward VM | Allow a new TCP flow only from the private-path sources and when its destination port is in the VM's current allowlist | Allow a reply only when it belongs to a guest-originated connection |
|
|
| From VM | Allow guest-originated traffic | Allow a service reply only when the original destination VM/port is still in the allowlist |
|
|
|
|
The implementation uses connection tracking's original tuple, rather than the
|
|
packet's current source/destination port alone, for service replies. For
|
|
example, a web server reply is sent *from* TCP/443 to a client ephemeral port;
|
|
the original tuple still identifies TCP/443 as the guest service.
|
|
|
|
The actual nft syntax is an implementation detail and must be generated from
|
|
typed data. It will use `ct direction`, `ct original ip daddr`, and `ct
|
|
original proto-dst` only after an isolated integration test verifies the
|
|
target kernel's bridge conntrack behavior.
|
|
|
|
### Identity and Layer-2 Controls
|
|
|
|
Filtering only TCP ports is not enough if a guest can spoof another guest's
|
|
source identity. For every managed TAP, the policy must:
|
|
|
|
- Accept a frame from the TAP only when its Ethernet source equals the
|
|
persisted guest MAC.
|
|
- Accept an IPv4 frame from the TAP only when its source equals the persisted
|
|
guest IPv4 address.
|
|
- Bind ARP sender hardware and protocol addresses from the TAP to that same
|
|
MAC/IP pair; allow only the required target/address-probe forms after packet
|
|
testing.
|
|
- Drop IPv6, tagged Ethernet frames, and unsupported EtherTypes both to and
|
|
from the managed TAP while they have no corresponding policy.
|
|
|
|
The identity check belongs before or at the earliest bridge ingress point. It
|
|
must be validated against the actual bridge forwarding database behavior on
|
|
the target kernel. This is meaningful network-policy hardening, but it does
|
|
not turn UVM into a complete hostile-tenant platform. Stronger tenant
|
|
isolation may eventually require per-VM L2/L3 segments, bridge-port
|
|
isolation, VLANs, or network namespaces.
|
|
|
|
### Ownership and Rule Rendering
|
|
|
|
UVM owns only its dedicated table and resources carrying a fixed versioned
|
|
UVM marker. The adapter must:
|
|
|
|
1. Refuse an unexpected pre-existing UVM table or an ownership-marker
|
|
mismatch rather than adopting it.
|
|
2. Render the complete desired contents of its own table from current state.
|
|
3. Apply the generated table in one nftables transaction.
|
|
4. Inspect the owned table after a mutation when a stronger confirmation is
|
|
required.
|
|
5. Remove a per-VM dispatch entry only after its TAP is confirmed absent or
|
|
safely quarantined.
|
|
|
|
The exact table replacement batch must be tested against the supported nft
|
|
version. It may rebuild only `table bridge uvm`; it must never rebuild the
|
|
host's whole nftables ruleset.
|
|
|
|
## 7. Persistence and Reconciliation
|
|
|
|
The JSON registry remains the source of desired policy. Kernel firewall state
|
|
is observed state and is never treated as durable truth.
|
|
|
|
The future `VmRecord` model needs to persist at least:
|
|
|
|
```text
|
|
name (if VM naming is added)
|
|
open_tcp_ports: list[int] | None
|
|
policy_intent / pending lifecycle phase
|
|
policy generation
|
|
```
|
|
|
|
`None` preserves the legacy/unmanaged distinction. New VMs always have an
|
|
explicit policy, including a deliberately empty policy.
|
|
|
|
The registry also needs a topology fingerprint at the state level containing
|
|
the configured network CIDR, gateway, and bridge name. Lifecycle and firewall
|
|
preflight compare the current `Settings` against that persisted fingerprint
|
|
before touching interfaces or rules. A legacy registry without this fingerprint
|
|
requires an explicit migration; it must not infer that a changed `/24` and
|
|
`/16`, for example, describe the same UVM topology.
|
|
|
|
UVM should not persist an authoritative `policy_enforced: true` boolean. It
|
|
would become stale after a reboot, manual firewall change, partial failure, or
|
|
host firewall reload. Instead, `NftFirewall.reconcile()` renders the desired
|
|
state from the registry and inspects the UVM-owned table when needed.
|
|
|
|
Reconciliation has no continuous trigger in this design. It runs during
|
|
create, start, port mutations, and an explicit future policy-sync operation.
|
|
Read-only list/inspect commands may report an owned table as missing or
|
|
unverified, but must not silently mutate the host. If an active VM loses its
|
|
UVM firewall table between operations, its policy is unknown until a successful
|
|
explicit reconciliation; UVM must not claim it remains enforced.
|
|
|
|
### Mutation Ordering
|
|
|
|
All firewall mutations use the existing `operations.lock`. State and kernel
|
|
updates cannot be one cross-resource transaction, so durable intent and
|
|
observed enforcement must remain distinct:
|
|
|
|
| Action | Required order |
|
|
|---|---|
|
|
| Create | Persist explicit desired policy, create the TAP down, install/verify policy, activate TAP, then launch Firecracker |
|
|
| Start | Reconcile the saved policy before activating the retained TAP or launching a new VMM |
|
|
| Open port | Persist desired policy first, atomically apply it, then clear the pending state |
|
|
| Close port | Persist the more restrictive desired policy first, atomically apply it, then clear the pending state |
|
|
| Stop | Persist stop intent, quarantine the TAP, confirm VMM exit, remove/deactivate TAP, retain saved policy |
|
|
| Delete | Quarantine, confirm exit, remove TAP, remove UVM policy for the VM, clear stale flow state before IP reuse, then remove registry data |
|
|
|
|
If an open operation cannot be applied, the desired state may say "open" while
|
|
the host remains more restrictive. UVM reports failure and retries through
|
|
reconciliation; it must not claim the port is open. If a close operation
|
|
cannot be confirmed, UVM retains its restrictive intent and attempts to
|
|
quarantine the attachment rather than restoring the old policy.
|
|
|
|
There is an unavoidable crash window after restrictive intent is persisted and
|
|
before the nftables transaction commits. During that window, an already-running
|
|
VM can still have its old permissive kernel policy. An interrupted close stays
|
|
pending and is not reported as enforced until an explicit reconciliation or
|
|
quarantine successfully commits. The operation lock cannot close this window
|
|
after the UVM process exits.
|
|
|
|
### Quarantine
|
|
|
|
An empty inbound allowlist is not a quarantine because it still permits a VM
|
|
to initiate outbound traffic. A quarantine mode drops traffic both to and from
|
|
the owned TAP. It is required while a stop, delete, or failed boot leaves
|
|
process/TAP ownership unresolved.
|
|
|
|
Do not remove this guard merely because a cleanup operation was attempted. It
|
|
remains until the TAP is gone or a successful start replaces it with the
|
|
saved active policy.
|
|
|
|
## 8. Python Integration
|
|
|
|
### Internal Adapter Boundary
|
|
|
|
Add a focused internal module such as `uvm/firewall.py`. It should be
|
|
constructed in [`uvm/app.py`](uvm/app.py), similarly to the current
|
|
`NetworkManager` and Firecracker process manager.
|
|
|
|
```python
|
|
firewall = NftFirewall(settings, runner)
|
|
lifecycle = LifecycleService(settings, state_store, images, network, process, firewall)
|
|
```
|
|
|
|
The exact constructor arrangement can vary, but these boundaries must remain:
|
|
|
|
| Component | Responsibility |
|
|
|---|---|
|
|
| `LifecycleService` | Owns ordering, operation locking, persistence intent, rollback, and VM status transitions |
|
|
| `NetworkManager` | Owns bridge validation plus TAP preparation, activation, and deletion |
|
|
| `NftFirewall` | Owns only UVM's generated nftables policy and its local inspection/reconciliation |
|
|
| `CommandRunner` | Runs local commands without a shell and makes command construction testable |
|
|
| CLI/API routers | Validate user input and call services; never send raw firewall syntax or commands |
|
|
|
|
The adapter should expose typed operations, for example:
|
|
|
|
```python
|
|
@dataclass(frozen=True, slots=True)
|
|
class VmFirewallPolicy:
|
|
vm_id: str
|
|
tap: str
|
|
guest_ip: IPv4Address
|
|
guest_mac: str
|
|
open_tcp_ports: tuple[int, ...]
|
|
quarantined: bool = False
|
|
|
|
|
|
class NftFirewall:
|
|
def preflight(self) -> None: ...
|
|
|
|
def reconcile(self, policies: Iterable[VmFirewallPolicy]) -> FirewallObservation: ...
|
|
|
|
def inspect(self) -> FirewallObservation: ...
|
|
|
|
def purge_vm_flows(self, vm: VmFirewallPolicy) -> None: ...
|
|
```
|
|
|
|
The public surface should accept domain objects, not interface names or raw
|
|
rules supplied by a CLI/API caller. There should be no `apply_raw`, `flush`,
|
|
or generic arbitrary-command method.
|
|
|
|
### Command Transport
|
|
|
|
The first implementation should call the local `nft` executable through
|
|
`CommandRunner`, rather than add a Python package dependency:
|
|
|
|
```python
|
|
runner.run(
|
|
("nft", "--check", "--file", "-"),
|
|
input_text=ruleset,
|
|
capture=True,
|
|
timeout=settings.firewall_timeout_s,
|
|
)
|
|
runner.run(
|
|
("nft", "--file", "-"),
|
|
input_text=ruleset,
|
|
capture=True,
|
|
timeout=settings.firewall_timeout_s,
|
|
)
|
|
```
|
|
|
|
This follows the same engineering model as Firecracker's Unix-socket adapter:
|
|
a narrow typed UVM boundary hides an external host interface. For nftables,
|
|
the host interface is a local command/Netfilter transaction rather than HTTP.
|
|
|
|
`nft --check` and `nft --file -` are separate operations. The checked batch is
|
|
only a validation aid; UVM may claim a policy change only after the apply
|
|
transaction succeeds and the adapter records its observation.
|
|
|
|
Benefits of the CLI approach:
|
|
|
|
- No shell interpolation or raw shell scripts.
|
|
- Atomic batch application through `nft --file -`.
|
|
- Syntax/preflight checking through `nft --check`.
|
|
- Easy fake-runner unit tests and visible CLI traces.
|
|
- No additional Python runtime dependency.
|
|
|
|
Inspection should use JSON rather than human-formatted output:
|
|
|
|
```python
|
|
runner.run(
|
|
("nft", "--json", "list", "table", "bridge", "uvm"),
|
|
capture=True,
|
|
timeout=settings.firewall_timeout_s,
|
|
)
|
|
```
|
|
|
|
`python3-nftables`/libnftables is a viable future transport if avoiding a
|
|
subprocess becomes necessary. It must not be introduced casually: package API
|
|
and kernel compatibility need a version-pinned integration suite. Direct
|
|
Netlink programming is not justified for the first implementation.
|
|
|
|
### Errors
|
|
|
|
Introduce a `FirewallError(UvmError)` for semantic firewall failures such as
|
|
unsupported local capabilities, unexpected ownership, or an unsafe observed
|
|
table. Preserve `CommandError` for an `nft` or `conntrack` command failure.
|
|
|
|
Errors must include the failed operation and safe diagnostics, but never
|
|
include raw untrusted rule input because the adapter has none. The CLI and API
|
|
can retain their existing `UvmError` rendering conventions.
|
|
|
|
## 9. Installation and Host Requirements
|
|
|
|
The installer remains the only component that invokes `apt-get`. The future
|
|
package list should add:
|
|
|
|
```text
|
|
nftables
|
|
conntrack
|
|
```
|
|
|
|
`conntrack` is used for targeted stale-flow cleanup before a deleted VM IP is
|
|
reallocated. It is not a substitute for the firewall's live close policy.
|
|
|
|
On Ubuntu-compatible hosts, the installation step is conceptually:
|
|
|
|
```sh
|
|
apt-get install --yes nftables conntrack
|
|
```
|
|
|
|
Do not automatically enable `nftables.service`, write `/etc/nftables.conf`,
|
|
or install a system-wide persistence service. UVM dynamically owns only its
|
|
dedicated table and recreates/reconciles it during lifecycle operations.
|
|
|
|
### Required Host Capabilities
|
|
|
|
| Requirement | Reason |
|
|
|---|---|
|
|
| Linux with native nftables support | UVM uses local Netfilter tables |
|
|
| Kernel with bridge-family connection tracking support | Required for original-tuple service checks; upstream support starts in Linux 5.3 |
|
|
| `nft`, `conntrack`, and `ip` commands | Local policy, selected flow cleanup, and TAP/bridge management |
|
|
| `CONFIG_BRIDGE` and `CONFIG_TUN` | Existing UVM bridge/TAP topology |
|
|
| Netfilter nftables, conntrack, bridge, reject, and relevant protocol modules | Required to load/apply the generated table |
|
|
| Root or `CAP_NET_ADMIN` in the bridge network namespace | Interface and firewall administration |
|
|
| No flowtable/hardware offload that bypasses this filter path | Required for reliable close enforcement |
|
|
|
|
Preflight should verify the actual generated ruleset with `nft --check` on the
|
|
target host. Kernel-version checks alone are not enough; distributions can
|
|
build or package Netfilter capabilities differently.
|
|
|
|
UVM must not enable `br_netfilter` or change `net.bridge.bridge-nf-call-*`
|
|
sysctls. Native bridge nftables works without that legacy compatibility layer.
|
|
|
|
## 10. Configuration Approach
|
|
|
|
The first version should avoid a generic `UVM_FIREWALL_RULES` setting or any
|
|
configuration value containing raw nftables syntax.
|
|
|
|
The generated policy derives from existing UVM configuration and state:
|
|
|
|
| Source | Firewall use |
|
|
|---|---|
|
|
| `UVM_NETWORK` | Validates guest IPv4 addresses and UVM subnet membership |
|
|
| `UVM_GATEWAY` | Establishes the bridge/guest topology for ARP and gateway traffic |
|
|
| `UVM_BRIDGE` | Selects the UVM bridge and its managed TAP attachments |
|
|
| `VmRecord.tap`, `VmRecord.mac`, `VmRecord.guest_ip` | Binds policy to one owned VM attachment |
|
|
| Persisted `open_tcp_ports` | Creates the per-VM TCP allowlist |
|
|
|
|
One bounded command timeout, such as `UVM_FIREWALL_TIMEOUT_S`, is reasonable
|
|
as a `Settings` value. It should default to a small value and apply only to
|
|
UVM's local command invocations.
|
|
|
|
Changing `UVM_NETWORK`, `UVM_GATEWAY`, or `UVM_BRIDGE` after VMs exist is not
|
|
a firewall migration mechanism. UVM must fail rather than infer ownership of
|
|
old interfaces or rules from changed environment values.
|
|
|
|
## 11. Lifecycle Integration
|
|
|
|
The existing `NetworkManager.create_tap()` activates a TAP immediately. The
|
|
firewall layer requires that operation to be split into preparation and
|
|
activation:
|
|
|
|
```text
|
|
create TAP down -> attach to bridge -> install policy -> bring TAP up
|
|
```
|
|
|
|
### Create and Start
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant L as LifecycleService
|
|
participant S as StateStore
|
|
participant N as NetworkManager
|
|
participant F as NftFirewall
|
|
participant P as Firecracker
|
|
|
|
L->>S: persist VM and explicit policy intent
|
|
L->>N: ensure bridge; prepare TAP down
|
|
L->>F: reconcile active policy including new VM
|
|
F-->>L: policy confirmed
|
|
L->>N: activate TAP
|
|
L->>P: spawn, configure, and start VMM
|
|
L->>S: persist running state
|
|
```
|
|
|
|
A policy failure prevents TAP activation and Firecracker launch. A startup
|
|
failure after policy installation leaves the policy in place until lifecycle
|
|
cleanup confirms that the TAP is gone. This is safer than briefly booting an
|
|
unfiltered guest.
|
|
|
|
### Stop, Delete, and Failure Cleanup
|
|
|
|
```text
|
|
stop/delete intent
|
|
-> quarantine owned TAP
|
|
-> terminate verified Firecracker process and confirm exit
|
|
-> deactivate/delete TAP
|
|
-> stop: retain desired port policy for a later start
|
|
-> delete: remove VM policy, purge stale flows, remove disk/state
|
|
```
|
|
|
|
If process identity or TAP ownership is unresolved, keep the quarantine guard,
|
|
disk, and registry entry. Do not delete a possibly live VM disk or remove its
|
|
network guard simply because a cleanup command raised an error.
|
|
|
|
Shared table infrastructure may remain after the last VM is deleted. It is
|
|
not a resource leak: it is a UVM-owned baseline that can be reconciled by the
|
|
next lifecycle operation. It must contain no stale per-VM dispatch/policy
|
|
entries after successful deletion.
|
|
|
|
## 12. Coexistence and Security Boundaries
|
|
|
|
The initial supported profile is a host where UVM may own `table bridge uvm`
|
|
and no other component modifies that table, using the dedicated private bridge
|
|
profile described above.
|
|
|
|
This does not mean UVM owns the host firewall. In particular:
|
|
|
|
- A terminal `drop` in UVM's bridge table can protect a UVM TAP.
|
|
- An `accept` in one nftables base chain does not promise that later hooks or
|
|
another firewall manager will accept the packet.
|
|
- UVM must not change global default policies, Docker chains, firewalld zones,
|
|
UFW framework files, or a host's `nftables.service` configuration.
|
|
- If Docker, Podman, UFW, firewalld, libvirt, or site firewall tooling is
|
|
active, allowed traffic may still be blocked by that tooling. UVM should
|
|
report its own enforcement state honestly rather than claim reachability.
|
|
|
|
The present design controls the local private network path only. It does not
|
|
protect against a privileged host process, a guest root exploit, raw host
|
|
network administration, or future external DNAT rules. Source-IP allowlists
|
|
and public exposure, if added, require a separate `inet`-family and ingress
|
|
design.
|
|
|
|
## 13. Testing and Release Gates
|
|
|
|
### Unit Tests
|
|
|
|
Add a focused `tests/test_firewall.py` with a stateful fake runner. Test:
|
|
|
|
- Generated table ownership, no `flush ruleset`, and no built-in chain policy
|
|
changes.
|
|
- Exact TAP-scoped dispatch and VM policy rendering.
|
|
- One atomic batch per policy update.
|
|
- Validation failures before host mutation.
|
|
- Idempotent reconciliation and recovery from missing UVM-owned rules.
|
|
- Refusal to adopt an unexpected table or marker mismatch.
|
|
- No broad `established,related` acceptance before service policy checks.
|
|
- Close-policy rendering that blocks both request and reply service packets.
|
|
- Quarantine and policy retention across stop/start/delete.
|
|
- Lifecycle ordering: policy before TAP activation and Firecracker spawn.
|
|
- Failure injection at state, nft, process, TAP, and cleanup boundaries.
|
|
|
|
### Isolated Network-Namespace Proof
|
|
|
|
Before integrating with real Firecracker, build a disposable network-namespace
|
|
test using a bridge and veth peers standing in for TAPs. It must prove:
|
|
|
|
1. Host-to-VM traffic reaches an allowed TCP port and fails at a closed port.
|
|
2. VM-to-VM traffic is governed by the destination VM's policy.
|
|
3. Guest-initiated TCP/UDP egress and replies still work.
|
|
4. Closing a port during one-way server streaming blocks subsequent reply
|
|
packets as well as new requests.
|
|
5. A guest outbound connection using a source port equal to a closed service
|
|
port is not falsely blocked as inbound service traffic.
|
|
6. ARP/gateway behavior remains functional.
|
|
7. IPv6 cannot bypass the IPv4-only policy.
|
|
8. MAC/IP spoofing from a managed TAP is rejected.
|
|
9. Tagged IPv4/IPv6 and unsupported EtherTypes are denied at managed TAPs.
|
|
10. An active unmanaged TAP prevents managed-policy activation.
|
|
11. Unrelated bridge ports and host firewall tables are unaffected.
|
|
|
|
Only after that proof should an isolated Firecracker test verify real TAP/MAC
|
|
behavior, default SSH access, an allowed guest listener, and live close.
|
|
|
|
## 14. References
|
|
|
|
- [Linux bridge Netfilter documentation](https://docs.kernel.org/networking/bridge.html#netfilter)
|
|
- [nftables bridge filtering](https://wiki.nftables.org/wiki-nftables/index.php/Bridge_filtering)
|
|
- [nftables atomic rule replacement](https://wiki.nftables.org/wiki-nftables/index.php/Atomic_rule_replacement)
|
|
- [nftables connection-tracking metadata](https://wiki.nftables.org/wiki-nftables/index.php/Matching_connection_tracking_stateful_metainformation)
|
|
- [firewalld concepts and policies](https://firewalld.org/documentation/concepts.html)
|
|
- [UFW manual](https://manpages.ubuntu.com/manpages/noble/man8/ufw.8.html)
|
|
- [iptables manual](https://man7.org/linux/man-pages/man8/iptables.8.html)
|
|
- [Linux `bpf(2)` manual](https://man7.org/linux/man-pages/man2/bpf.2.html)
|