Architecture

Overview

clabernetes is a collection of kubernetes custom resources and controllers that reconcile those resources. The ultimate goal of the controllers is to render a network topology in the cluster: one device Pod per (containerlab) node running the actual network-device image as a regular application container, network node interfaces stitched together across pods, and the management interfaces of the nodes exposed.

There is no nested runtime: no Docker daemon inside the Pod, no containerlab executable, and no image import path. The kubelet pulls and runs every device image natively, and Kubernetes owns device lifecycle, logs, resources, and status. Kind behavior (defaults, generated files, lifecycle commands, readiness) comes exclusively from an unmodified, pinned containerlab Go module, so a containerlab module bump admits new kinds without any clabernetes kind code.

Everything below the "PRIMARY API" line works identically whether the custom resources were written by a person, by tooling, or by the Topology compiler.

The Primary API: Nodes and Links

A Node custom resource represents a single containerlab node: its spec is simply what a human would write for that node in a containerlab topology file (flat, verbatim containerlab vocabulary). A Link custom resource represents a single wire between two nodes. That's the whole authoritative api: apply Node and Link objects and the controllers render a running, wired lab, with no Topology object required.

The object name of a Node is the containerlab node name: the device pod hostname and the node's services derive from it, which also means the namespace is the topology boundary. Everything operational is controller-stamped into the statuses: expose port allocations and readiness on Nodes, wire id allocations on Links. Deployment policy (expose behavior, image pull defaults, primary-container resources, scheduling, persistence, probes) lives on NodeProfile (the retained CRD name for reusable direct-workload policy). Each Node can explicitly reference one same-namespace profile through spec.profileRef; an omitted reference uses global Config defaults, while a missing explicit reference prevents realization instead of silently falling back. Device privilege, capabilities, devices, and other kind-owned container requirements are not profile policy: they come from the imported package's plan.

Topology remains a supported, backward-compatible auxiliary resource. Its existing connectivity, expose, deployment, imagePull, and statusProbes fields are accepted and compiled into primitive resources. Because a Topology still embeds the whole source definition, large labs should define NodeProfile, Node, and Link resources directly instead of persisting a Topology.

Components

Controllers & Custom Resource Definitions

The "brains" of clabernetes is the manager deployment which runs three cooperating controllers:

• the node controller turns every Node (or network-mode: container:<primary> group of Nodes) into one Recreate Deployment of a direct device Pod, a per-node "fabric" service (<name>-wire, headless peer discovery for cross-worker wires), an expose service (<name>), one headless service per declared network alias, and an optional PVC, and stamps readiness/allocations into the Node status. Before rendering, it resolves image metadata and runs planning (below); a Node whose plan fails stays failed with a structured diagnostic and never mutates the last successfully applied workload.
• the link controller validates Links and allocates namespace-unique wire ids into their statuses (wire ids become the link identities on the fabric wire between connectivity sidecars; datagrams dispatch inside one receiving sidecar from a validated source, so the namespace is the whole allocation domain).
• the topology controller is the optional convenience layer: it compiles a Topology (a containerlab file plus knobs) into NodeProfile/Link/Node objects, expanding topology defaults/kinds/groups into each node so every emitted Node is self contained. It prunes emitted objects that fall out of the definition, protects them from drift, and aggregates fixed-size counts and readiness back into the Topology status. Profiles and Links are reconciled before Nodes.

The compiler normally emits one shared NodeProfile and places its explicit reference on every Node. A distinct per-node workload resource policy produces a complete dedicated profile, not an inheritance chain.

Compilation fails closed: any unrecognized or unrepresentable Containerlab field fails before resource creation with sorted structured diagnostics naming every unsupported field and location. Deliberately rejected vocabulary fails with a diagnostic stating why, and only constructs whose loss cannot change lab behavior inside the cluster are accepted with a warning instead; see Differences from containerlab for the full list. Explicit veth links may use brief node:interface endpoints or equivalent structured node/interface mappings; empty or malformed endpoint values fail before Link resources are emitted.

Planning

Imported containerlab code mixes intent with side effects, so kind hooks never run inside the long-running manager. By default, a pool of four planning Pods runs in the manager namespace. Each worker handles one request at a time in a fresh, deadline-bounded subprocess with a private workspace. The controller first resolves metadata for each logical Node's declared image, then opens a private request/response stream through Kubernetes exec to an available worker. Inside it, the imported registry and node hooks run against a generic recording runtime and a disposable workspace: container creation, files, mounts, lifecycle, exec, namespace, and interface operations are recorded rather than performed, and the result is a deterministic, runtime-neutral device plan.

The plan's image configuration comes from a registry metadata resolver in the manager that fetches only manifests and config blobs (never layers) using same-namespace pull Secrets and the controller-only Config.spec.imagePull.registryMetadataTrust and registryMetadataMirrors policies. Assigning another imported role to an already-known image is handled in worker memory. If setup genuinely discovers a new image reference or certificate requirement, the worker asks the manager over the same attached stream; the manager resolves or issues it, and that process reruns the relevant evaluation without another Pod. The worker has no service-account token or network access. A repeated request that adds no facts and the overall session deadline fail the attempt instead of driving normal startup. The kubelet performs the only real image pull, and readiness fails on a material digest mismatch between the planned image and the kubelet-reported imageID.

The ordinary path is therefore: resolve declared-image metadata, use an existing worker, then create the device Pod. There are no separate image-discovery Pods. Accepted finalized inputs and plans are stored in immutable, digest-named, owner-referenced ConfigMaps that contain no secret bytes, so unchanged Nodes skip planning. Different Nodes keep separate plans even when their kind and image match: names, addresses, links, payloads, certificates, and entropy remain request-specific. Private payloads and entropy travel over the exec stream, and each request's workspace is removed before the worker is reused. Imported Go state is discarded with the subprocess. A kernel lock prevents overlapping requests, including across manager restarts. Each execution has a five-minute maximum lifetime; failed executions can be retried without retaining partial plans.

Set plannerPool.replicas in the Helm values to change parallel planning capacity. Excess requests wait in controller reconciliation without creating more Pods. The Node controller currently permits 16 concurrent reconciles, so replicas above 16 cannot increase concurrent planning without also raising that controller limit and validating the added load. Transferred payloads are limited to 64 MiB in total per request. Set plannerPool.enabled: false to use the original disposable planning Pod path, including when larger aggregate payloads are required. The device Pod's planner preparation init container remains necessary in either mode.

Each cold plan owns one bounded mutable connectivity-revision ConfigMap so live-capable Link changes update interfaces without rolling the Pod.

Direct device Pods

Each rendered Pod runs, in order:

1. a preparation init container that re-runs the imported deterministic file generation against its mounted payload/certificate inputs, verifies every planned path, mode, and digest, and stages artifacts into plan-scoped volumes. It renders generated configuration with the runtime-completed management identity, so package templates emit the real management address, prefix, and gateway;
2. a restartable privileged connectivity init-sidecar whose startup probe gates device start until local initialization, the accepted cold-start interface set, and imported endpoint hooks are complete. Missing remote peers keep its readiness probe failing without restarting the helper or blocking application startup. It stays running to reconcile live Link changes and publish connectivity readiness; and
3. one regular application container per grouped logical Node or planned chassis component, running the unmodified device image. Kubernetes-native image pull policy, pull Secrets, environment, users, ports, resources, devices, capabilities, privilege, seccomp/AppArmor, sysctls, tmpfs, shared memory, and DNS all map from the plan onto native Pod fields.

A standalone Node has one Deployment/Pod; network-mode: container:<primary> adds application containers to the primary's Pod; a component-based node (i.e. a distributed SR-SIM chassis) expands into card containers that are directly visible as application containers of one Pod with validated namespace ownership. Container names are deterministic DNS labels derived from logical Node and component identity, so kubectl logs and kubectl exec target the actual device container (-c selects components).

VM-backed kinds stay ordinary application containers running their packaged qemu/vrnetlab entrypoint; the plan grants /dev/kvm, tap access, and resources, and clabernetes does not launch the VM itself.

Inter-Node Connectivity

The device always sees a plain interface leg; every kind tolerates that, including kinds that adopt the interfaces presented to them. All connectivity is realized by the Pod's own privileged connectivity sidecar; no node-resident agent exists. Before any device container starts, the sidecar preserves the CNI interface under a sidecar-owned name (c9s0), so the Kubernetes underlay survives whatever a device does to the interfaces it is shown.

Cross-pod wires terminate inside the Pod on that preserved underlay: each endpoint is a veth pair whose sidecar-side leg feeds the Pod's fabric wire, a sidecar-to-sidecar UDP transport (one socket per Pod) that segments whole frames to the local underlay MTU, reassembles them at the peer, propagates endpoint carrier state, and detects peer loss with heartbeats, all on one socket and one path so the carrier signal cannot diverge from the datapath in endpoint or route. Frames are addressed by the Link's allocated wire id; loss stays loss (nothing is retransmitted), so link MTUs never depend on the Pod network MTU and loss/convergence testing stays honest. Peer discovery uses the per-node headless fabric service (<remote node>-wire.<namespace>...), which publishes the current Pod address even before readiness; rewiring and Pod rescheduling converge on the sidecar's reconcile tick, and a peer that stops answering takes the link carrier-down like an unplugged cable. Same-Pod, loopback, and local veth state stays inside the Pod sandbox and is removed automatically with it. Host Links (host:<interface> endpoints) are veth pairs whose worker-side end the sidecar places through a read-only worker network-namespace handle (mounted only when the topology declares host Links); because a veth pair dies with either end, forced Pod deletion leaves no worker residue and nothing needs sweeping.

Management and DNS

Management identity is always addressed with containerlab semantics: every node's management address is controller-allocated, from the operator's management policy (profile mgmt subnets, static mgmt-ipv4/mgmt-ipv6), or from containerlab's default management subnet (172.20.20.0/24, gateway at the first usable address) when no policy is declared. The sidecar interposes a synthetic management interface carrying that address before the device starts: the device adopts it exactly as it would a physical management port, a sidecar-owned policy routing table keeps Kubernetes transport immune to device rewrites of the main table, and an nftables translation table (hooked ahead of anything a device programs) SNATs outbound management traffic to the Pod address and DNATs declared ports from it, so Service exposure keeps working. Inbound translated connections also get the Pod-local gateway as their source, so a device that only knows its connected management route (SR OS reads its routes from a Docker-shaped environment) can always answer. This is the same source identity containerlab's Docker port publishing presents. Management configuration renders at plan time through each kind's own imported templates using the allocated address, prefix, and gateway.

The management subnet is one shared subnet across the whole namespace, exactly as on containerlab's management network: a device dialing a peer's management address (a hardcoded telemetry target, a syslog collector, plain ping) reaches the peer device itself, any protocol, both directions, with source addresses intact. Nothing is flooded to get there: the sidecar's router leg answers the device's ARP for any peer with the gateway identity and routes the packet over a management VXLAN tunnel endpoint riding the preserved Pod underlay, holding exactly one neighbor entry and one forwarding entry per peer. Peers are published rather than discovered: the controller renders a namespace-wide peer directory (node names, aliases, management addresses, and the address of the Pod realizing each node) as a fixed set of ConfigMap shards that the kubelet projects into every Pod, so nodes joining, leaving, or moving between workers converge without any DNS lookup and without restarting any Pod (a rescheduled Pod's new address reaches its peers with the kubelet's ConfigMap sync, about a minute by default). The management gateway stays strictly Pod-local: every Pod answers the gateway address itself with one deterministic identity, and gateway traffic never crosses the mesh. Inside a device, node names, declared aliases, and chassis component names resolve to management addresses through the same peer directory written into each Pod's hosts file; the per-node and alias Services serve the rest of the cluster. Only ARP is emulated across Pods; other device broadcasts (gratuitous ARP, LLDP or DHCP on the management port) stay inside the Pod.

A device that firewalls its own network namespace (EOS installs a default-deny input policy) cannot sever these transports: the sidecar keeps accept rules for its mesh and wire UDP ports asserted at the head of any packet filter a device programs in the shared namespace, and re-asserts them on the reconcile tick when a device rewrites its firewall. On hardware the wire carrying management traffic is invisible to a control-plane ACL; these accepts preserve exactly that boundary, while the device keeps filtering its own management traffic above the transport.

Exposing Nodes

Ports listed in a node definition (plus a sensible default set unless auto-expose is disabled) are allocated into the Node status and exposed through a per-node service, LoadBalancer flavored by default, targeting the device Pod's ports directly. The assigned address is reflected back into the Node status.

Status and readiness

The node controller reads init/application container status directly and maps it back to each logical Node using applied plan identity and UID checks, so a stale Pod cannot update a replacement Node. OCI image healthchecks and declared healthcheck contracts translate into container startup/readiness behavior; profile statusProbes add TCP/SSH checks. Failures surface as bounded Node conditions and Kubernetes Events naming the logical Node, plan digest, and failing typed action.