Image pulling

Configure pull policy, private registry credentials, registry trust, and controller metadata mirrors.

The kubelet pulls device images like the image of any other Pod. c9s has no inner Docker daemon, image-puller Pods, or image import path, so registry access is configured the Kubernetes way: pull policy and pull Secrets on the workload, mirrors and trust in the worker container runtime.

Pull policy

The default pull policy is IfNotPresent. Change it in the singleton Config, in a NodeProfile, or in a Topology's spec.imagePull:

apiVersion: c9s.run/v1alpha1
kind: Config
metadata:
  name: clabernetes
spec:
  imagePull:
    policy: Always

The supported values are Always, IfNotPresent, and Never. A node that sets image-pull-policy in its containerlab definition keeps that value.

Private registry credentials

Create a Docker registry Secret in the namespace where the Node or Topology lives. The example creates a Secret named regcred in the lab namespace for the registry registry.example.com:

kubectl -n lab create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=myuser \
  --docker-password=mypass

Then set the Secret name as a global default in the Config, or reference it from a NodeProfile or Topology:

The Config default applies to every Node in the cluster:

apiVersion: c9s.run/v1alpha1
kind: Config
metadata:
  name: clabernetes
spec:
  imagePull:
    pullSecrets:
      - regcred

Secrets are namespaced, so the Secret must exist in every namespace that runs Nodes. A Node in a namespace without it is not planned: its PlanApplied condition reports ImagePullSecretMissing until the Secret appears. See centrally managed credentials for mirroring one Secret into many namespaces.

A NodeProfile that sets pullSecrets replaces the global list for its Nodes, and pullSecrets: [] opts them out of the global default.

With Helm, set the same default through globalConfig.imagePull.pullSecrets in the chart values.

c9s passes only the Secret name to the kubelet through Pod.spec.imagePullSecrets. The controller also reads the Secret to fetch the image manifest before it creates the Pod, so the Secret must be of type kubernetes.io/dockerconfigjson or kubernetes.io/dockercfg, which kubectl create secret docker-registry produces. Credential bytes are never copied into a plan or ConfigMap.

Centrally managed credentials

To manage registry credentials in one place, create the Secret once and mirror it into the lab namespaces with a tool such as External Secrets Operator or kubernetes-reflector. Together with the Config default, a new lab namespace then needs no registry setup at all.

With kubernetes-reflector, install it next to c9s:

helm upgrade --namespace c9s --install reflector oci://ghcr.io/emberstack/helm-charts/reflector

Create the regcred Secret in the c9s namespace and annotate it so the reflector mirrors it into every namespace:

kubectl -n c9s annotate secret regcred \
  reflector.v1.k8s.emberstack.com/reflection-allowed=true \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled=true

To mirror into selected namespaces only, add the reflector.v1.k8s.emberstack.com/reflection-auto-namespaces annotation with a comma-separated list of namespaces.

Registry mirrors, private CAs, HTTP, and proxies

Registry mirrors, proxies, private certificate authorities, and plain-HTTP registries are configured in the container runtime of every worker that can run a device Pod, not in c9s. The files and reload steps depend on the Kubernetes distribution and runtime.

Before it creates a device Pod, the c9s controller fetches the image manifest and configuration from the registry (never the layers) to plan the Pod. Public HTTPS registries need no extra setting. If the registry uses a private CA or plain HTTP, add a trust entry for it in the Config:

apiVersion: c9s.run/v1alpha1
kind: Config
metadata:
  name: clabernetes
spec:
  imagePull:
    registryMetadataTrust:
      - registry: registry.example.com
        caBundle: |-
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      - registry: registry.lab.example:5000
        plainHTTP: true

Entries match the exact host[:port] of the registry. Wildcards and paths are not accepted, and an entry sets either caBundle or plainHTTP. This trust applies only to the controller's metadata requests: configure the same CA or HTTP endpoint in the worker runtime as well, and prefer TLS with a private CA over plain HTTP. If the workers pull through a mirror, also declare it as a controller metadata mirror.

Controller metadata mirrors

Some clusters pull every image through a pull-through registry configured in the container runtime, for example Harbor proxy projects set up through containerd hosts.toml or Talos machine.registries.mirrors. The kubelet then never reaches the public registry, but the controller's metadata request would still dial it. Declare the same mirrors for the controller in the Config:

apiVersion: c9s.run/v1alpha1
kind: Config
metadata:
  name: clabernetes
spec:
  imagePull:
    registryMetadataMirrors:
      - registry: docker.io
        endpoint: https://harbor.example.com/v2/docker
        overridePath: true
      - registry: ghcr.io
        endpoint: https://harbor.example.com/v2/ghcr
        overridePath: true
    registryMetadataTrust:
      - registry: harbor.example.com
        caBundle: |-
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----

Each entry redirects the controller's manifest and configuration requests for one source registry to the mirror endpoint. The Docker Hub names docker.io, index.docker.io, and registry-1.docker.io share one entry. Only the connection changes: image references, resolved digests, and Pod image strings keep the original registry, so kubelets keep using their own mirror configuration. A failing mirror fails planning with a Node condition; there is no fallback to the public registry.

overridePath treats the endpoint path as the mirror's registry API root, replacing the standard /v2 prefix (containerd override_path semantics). A Harbor proxy project for ghcr.io at https://harbor.example.com/v2/ghcr serves ghcr.io/srl-labs/network-multitool at /v2/ghcr/srl-labs/network-multitool/.... An endpoint with a path requires overridePath; a hostname-only endpoint such as https://mirror.example.com forwards paths unchanged.

Trust follows the connection: a registryMetadataTrust entry for the mirror host applies to every registry fetched through that mirror, and the endpoint scheme selects TLS or plain HTTP. Pull Secret credentials still match the source registry in the image reference.

The authentication token realm a mirror advertises must be a DNS hostname: the controller refuses realms that point at loopback or private IP addresses. Harbor advertises its configured external URL as the realm, so a Harbor mirror that is reachable only by a private IP must be given a hostname.

Troubleshooting

Problems the controller finds before it creates the Pod are reported on the Node: PlanApplied becomes False with a reason such as ImagePullSecretMissing, ImagePullSecretUnreadable, or an OCIMetadata... reason for registry authentication and trust failures, plus a Warning event. Planning resumes on its own once the cause is fixed. Pull failures on the worker appear on the device Pod as the usual ErrImagePull or ImagePullBackOff events.

Inspect the Node, Pod, events, and resolved profile:

kubectl -n lab describe nodes.c9s.run router
kubectl -n lab get pods -l c9s.run/direct-workload=router
kubectl -n lab describe pod -l c9s.run/direct-workload=router
kubectl -n lab get nodeprofile private-images -o yaml

Test the kubelet path independently with a plain Pod that uses the same image and pull Secret:

kubectl -n lab run image-pull-test --restart=Never --image=registry.example.com/network/srlinux:latest \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}'
kubectl -n lab describe pod image-pull-test

If the test Pod fails, fix the Secret or the worker's registry configuration. c9s has no insecure-registry, pull-through, or image-import fallback of its own.