Fleet

Running detection on every host means having a way to configure and observe every host. Fleet is the control plane we built for that: enrollment, configuration, deployment, and observation, in three programs (fleetd, fleetserver, fleetctl) plus a managed WireGuard transport that hosts can opt out of. Per-site configuration lives on the server; edges hold the rendered output of each apply.

It is how we operate the decentralized SOC, not a product we are asking anyone to adopt on its own. This page is here so you can see the machinery rather than take it on faith.

Why not Ansible, Salt, or Puppet? For most fleets, use one of those. Three constraints pushed us elsewhere. Edges are often unreachable exactly when you want to change them, so the agent dials out and the server pushes onto that connection. No inbound port on the edge, and desired state lives on the server rather than in a runbook. Jails, ZFS datasets with a named retention policy, and per-jail port forwarding are first-class objects here instead of modules layered onto a Linux-shaped abstraction. And on a detection node, configuration drift is itself a security signal, so every managed file's hash is reported on heartbeat and compared server-side rather than discovered on the next run. Fleet is small because that is all it has to do.

Components

fleetd - device agent: Runs on every node as the unprivileged _fleetd user, with sudoers grants for the operations it needs (Bastille, pkg, sysrc, zfs, WireGuard). Certificate work runs inside the jail through the Bastille grant rather than as a separate one. Reconnects, heartbeats, applies, reports drift.
fleetserver - server: Headless Starlette REST API on :8888 for operators, binary enrollment broker on :8444 for fleetd. FleetDB tracks devices, platforms, deployments, audit log, drift, jail state and WireGuard topology. Overlays are files on the server, not rows.
fleetctl - operator CLI: Multi-namespace (production / staging / customer profiles), with fan-out for list verbs. Nouns: device, platform, enroll, overlay, secret, namespace, jail, app, appconfig, asset, ssh, tenant, pkg; plus apply, status, drift, vpn, logs, login. No web UI.

Platforms as YAML

A platform is a YAML document declaring a host’s sysrc, packages, services, ZFS layout, jails (with their packages, services, mounts, config files), and managed firewall rules. Operators ship a per-device overlay that overrides anything.

Platform	What it builds
`hosting`	Web and mail jails: Nextcloud, WordPress, Dolibarr, Stalwart. WAKE on the host watches every jail’s access logs.
`nas`	ZFS storage host. NFSv4 server, MariaDB, sanoid snapshots, fleet-backup, fleetserver. No jails.
`racing-boat`	Five jails: gateway, core (NATS), compute, persistence (TimescaleDB), vis. NMEA 2000 telemetry edge. Only platform that uses NATS.
`wake-edge`	Dedicated WAKE sensor: rsyslog + the WAKE modules (mmenrich / mmsigma / mmwake) + waked, plus `immesh` in rsyslog for pheromone gossip. For resource-constrained edges.
`soc-console`	Linux SOC operator console. WAKE collector, wakeweb, fleetd. Aggregates detections from every edge.
`vpn-hub`	Linux WireGuard hub. fleetserver renders `wg0.conf` on every peer mutation. Tunnel only.
`identity`	Keycloak OIDC and a co-located PostgreSQL jail.
`build-worker`	GitLab runner and Poudriere on amd64.

Platforms ship as a single package. Operators don’t edit them - they ship per-device overlays. Per-deployment content (configs, assets) lives in /usr/local/etc/marlinos/{app_configs,assets}/.

Architecture

             +----------------------------------------------+
             |  fleetserver  (REST API + enrollment broker) |
             |  ------------------------------------------- |
             |  FleetDB . audit . platforms . drift . vpn   |
             +-----------+-----------------+----------------+
                         | HTTP(S) :8888   | plain TCP :8444
                         | (operator API)  | (binary enrol)
                         |                 |
                +--------+--------+   +----+------------+
                |    fleetctl     |   |     fleetd      |
                |    (operator)   |   |  (every device) |
                +-----------------+   +--------+--------+
                                               |
                                  +------------+------------+
                                  |            |            |
                                  v            v            v
                          Bastille jails    sysrc/pkg    ZFS + sanoid

  The enrollment channel has no TLS.  Its authentication IS the transport:
  the broker binds to the WireGuard address only.  Hosts that opt out of
  the fleet mesh must therefore be reachable over a network you already
  trust.

WireGuard

Opt-out per host. Hosts with wireguard_enable: NO are managed over whatever transport they already have on the LAN. For hosts that join the mesh, fleet generates the keypair at enrollment, renders wg0.conf, and applies updates with wg syncconf, so peer changes do not drop the tunnel. Keys are generated by default; fleetctl vpn device key-override exists for adopting existing material.

Hub identity: The hub (vpn-hub platform) seeds its own keypair on first run, or adopts an existing on-disk wg0.conf if one is already present.
Device peers: Each enrolled device gets a per-device keypair, PSK, and tunnel IP. The peer block + private key get pushed inside the next platform apply.
Human peers: Laptops and phones aren’t fleetd nodes. fleetctl vpn user qr <name> hands a phone its config; fleetctl vpn user config <name> -o file writes a desktop wg0.conf.
External peers: Federation peers are declared with their pubkey, allowed-IPs, and optional tenant scope. They show up as direct [Peer] blocks on matching devices - no hairpinning.

Drift

Each file fleetd most recently wrote is hashed and reported on heartbeat, and the server compares those hashes on request. Jails are compared on interface, IP, release and jail type. Templates and RDR rules are reapplied every cycle rather than diffed, so they do not appear as drift.

# config-file drift on a single device
fleetctl drift edge-01

# --all + --watch across the fleet, with colour
fleetctl drift --all --watch

# jail drift (interface / ip / release / jail type)
fleetctl jail drift list edge-01

Enrolling a node

# on the operator workstation
fleetctl enroll list                       # pending requests
fleetctl enroll approve a1b2c3 --name edge-01   # prefix + a name

# assign a platform, push the overlay, apply
fleetctl device platform a1b2c3 racing-boat
fleetctl overlay push a1b2c3 -f overlays/my-boat.yaml
fleetctl apply a1b2c3