Architecture
katamaran is designed around one constraint: every extra process boundary, socket hop, and thread handoff between a Linux container and macOS shows up directly in developer workloads with tens of thousands of small files or short network round trips.
Process layout
Section titled “Process layout”On macOS, two processes run for an active VM:
katamaran daemonowns lifecycle management, configuration, the macOS sleep watcher, the NFS mount lifecycle, and the local control API (~/.katamaran/run/daemon.sock).katamaran-vmlinkslibkrundirectly on Apple’s Hypervisor.framework (HVF) and loadslibgvproxy.dylibin-process. Separating the VM into its own process keeps Hypervisor entitlements isolated and allows clean VM restarts without dropping the daemon or app connection.
Inside the Linux guest, kat-init runs as PID 1 from an uncompressed initramfs.cpio, mounts the read-only EROFS system image (/dev/vda) under a tmpfs overlay, mounts the three btrfs subvolumes of data.img (/dev/vdb), configures networking statically from kat.net=, and supervises containerd, dockerd, kat-nfs, LXC machines, and the memory reclaim controller.
Direct virtio-fs hypercalls and APFS backend
Section titled “Direct virtio-fs hypercalls and APFS backend”Standard virtio-fs routes every FUSE request through a virtqueue descriptor ring, wakes a host worker thread, and injects a completion interrupt into the guest vCPU. That round trip dominates small metadata operations (stat, open, readdir, unlink).
katamaran changes both the transport and the macOS server:
- Direct hypercalls (
KAT_FS_DIRECT): Small, bounded foreground FUSE requests pass their guest-physical buffer table directly in an ARM SMCCC vendor call. The vCPU thread handles the request inline and returns the reply length without touching virtqueues or injecting an interrupt. Requests that can block (such as iCloud dataless files, FIFOs, network volumes, or oversized I/O) fall back to the asynchronous worker pool, enforced bySyncPolicyand macOSsetiopolicy_npbounded opens. - APFS-tuned operations: Directory listings use macOS
getattrlistbulk(2). PlainFUSE_READDIRrequests onlyATTR_CMN_NAME,ATTR_CMN_OBJTYPE, andATTR_CMN_FILEID, which APFS serves from inline directory records without reading child inodes. Frequently used parent directories holdO_EVTONLYpinned descriptors (PinPool) so lookups, opens, and unlinks use*at()syscalls instead of/.vol/<dev>/<ino>volfs path resolution.OpenTrackerskipsfstatatonunlinkandrmdirwhen no open handle can refer to the target. - Long caching with FSEvents invalidation: Shares cache entries, attributes, negative lookups, unchanged file pages (
FOPEN_KEEP_CACHE), and unchanged directory listings (FOPEN_CACHE_DIR | FOPEN_KEEP_CACHE). A single macOS FSEvents stream watches shared trees and sends targeted FUSE node ID invalidations over virtio-vsock to/dev/kat_fsnotify, dropping stale guest caches and emitting Linux inotify events. - Caller ownership (
owner_caller): The host reports a fixed owner so attributes remain cacheable, while the guest FUSE driver substitutescurrent_fsuid()andcurrent_fsgid()ongetattr, checksaccess(2)locally against cached mode bits, and skips no-opchmodandchownFUSE_SETATTRcalls.
In-process networking
Section titled “In-process networking”katamaran-vm loads libgvproxy.dylib (gvisor-tap-vsock built with -buildmode=c-shared) into the VM process.
- Small outbound guest frames (
<= 4 KiB) are transmitted inline on the vCPU thread throughkat_gvproxy_txwithout waking a virtio-net worker thread. - Small inbound frames (
<= 2 KiB) copy headers and payload into a stack buffer and callinproc_net_rxdirectly withoutruntime.Pinneror slice allocations, while large frames (up to the 65,520-byte MTU) stay zero-copy with partial checksum offload (VIRTIO_NET_F_GUEST_CSUM). - TCP connections in the forwarder register persistent gVisor
waiter.Entryobjects and use 8 MiB macOS socket buffers with copy buffers that start at 64 KiB and grow to 1 MiB for bulk transfers. - DNS queries go through macOS
dns_sdwith answer caching and in-flight deduplication. - Inside the Linux guest,
kat-initdisablesbridge-nf-call-iptablesso container-to-container L2 bridge packets bypass IP netfilter hooks, and the guest kernel builds its arm64 vDSO with--hash-style=bothso both glibc and musl containers resolveclock_gettimewithout syscalls.
Adaptive idle polling and QoS
Section titled “Adaptive idle polling and QoS”Waking a halted vCPU from WFI on Hypervisor.framework costs tens of microseconds of host thread wakeup latency. The guest kernel defines TIF_POLLING_NRFLAG on arm64 and polls adaptively before halting (growing up to 200 µs during busy request-response traffic and shrinking to 0 µs when idle), keeping cross-vCPU container traffic fast without burning idle CPU. Active vCPU, network, and filesystem worker threads run at QOS_CLASS_USER_INTERACTIVE so macOS keeps them on performance cores under background host load.
64 KiB memory reclaim
Section titled “64 KiB memory reclaim”On macOS, madvise(MADV_FREE) does not immediately remove dirty hypervisor-backed pages from a process’s physical footprint. To give memory back to macOS reliably:
- At boot,
libkrunmaps guest RAM into separate 64 KiB XNU VM map entries by mapping the untouched region once with Tag 1 and overwriting only the odd 64 KiB blocks with Tag 2. Each block gets its own VM object on first fault. - In the guest,
kat-initsetspage_reporting_order = 4(64 KiB) on the virtio-balloon driver, trims idle file and reclaimable slab caches toward a 128 MiB budget through cgroup v2memory.reclaim, and runscompact_memoryonce after fragmentation exceeds 64 MiB. - When the balloon reports free 64 KiB blocks,
libkrununmaps them from Hypervisor.framework, replaces them with fresh anonymousmmap(MAP_FIXED)blocks (immediately freeing the old VM objects and lowering macOS physical footprint), and maps them back into the guest.