Kubernetes instancer

The Kubernetes instancer is the scalable backend for per-team challenge instances. An operator runs inside the cluster, turning a ChallengeInstance custom resource into a namespace, network policies, deployments, services, and Traefik routes.

The rCTF API only talks to the Kubernetes API server through the instancer/k8s-instancer provider. The operator handles every other moving piece.

Architecture#

A working deployment has three cooperating components:

A participant request flows through these in order:

Namespaces are deterministic and named inst-<challenge-id>-<team-id> so the controller can find them across restarts. Every child resource inherits owner references from the ChallengeInstance, so cluster-level garbage collection acts as a safety net behind the explicit finalizer.

Prerequisites#

The Terraform example assumes GKE plus Cloudflare for DNS and ACME. GCP Cloud DNS works as a drop-in alternative.

The instancer’s public hostname is <instancer_subdomain>.<instancer_zone> (or just <instancer_zone> when no subdomain is set). All per-instance hostnames live under a wildcard one level below.

Controller image#

The operator image is published at ghcr.io/otter-sec/rctf-new/k8s-controller, and the matching install.yaml ships in the repo at apps/k8s-controller/dist/install.yaml. The Terraform k8s module reads that file directly and substitutes the configured hostname into the INSTANCER_HOST placeholder, so there’s nothing to build or push before running $terraform apply.

Terraform variables#

The example terraform.tfvars lives in deploy/terraform/instancer/example/:

• deploy/

  • terraform/

    • instancer/

      • example/

        • main.tf Providers, GKE module wiring
        • dns.tf Cloudflare or GCP Cloud DNS record
        • tls.tf ACME wildcard certificate and Traefik TLSStore
        • rctf-instancer.tf k8s module call and rCTF ServiceAccount
        • variables.tf Input variables
        • terraform.tfvars.example Example values
      • modules/

        • gke/GKE cluster and Artifact Registry
        • k8s/Traefik, error pages, controller installer

Copy terraform.tfvars.example to terraform.tfvars and fill in:

A minimal Cloudflare-backed file looks like this:

1
cloudflare_api_token = "<cloudflare-api-token>"
2
letsencrypt_email_address = "ops@example.com"
3
instancer_zone = "ctf.example.com"
4
instancer_subdomain = "instances"
5
ctf_name = "Example CTF"
6

7
gcp_project_id = "example-ctf"
8
gcp_region = "us-central1"
9
gcp_zone = "us-central1-a"
10
gcp_instancer_cluster_name = "rctf-instancer"
11
gcp_instancer_machine_type = "e2-standard-4"
12
gcp_instancer_min_node_count = 1
13
gcp_instancer_max_node_count = 8

To use GCP Cloud DNS instead of Cloudflare, comment out the Cloudflare blocks in dns.tf and tls.tf, uncomment the google_dns_record_set and gcloud ACME blocks, and set gcp_dns_managed_zone_name.

Deployment#

$cd deploy/terraform/instancer/example
$cp terraform.tfvars.example terraform.tfvars
$$EDITOR terraform.tfvars
$terraform init

$terraform apply

$gcloud container clusters get-credentials rctf-instancer --project example-ctf --location us-central1
$kubectl get pods -n rctf-instancer-controller-system

$terraform output -raw rctf_instancer_api_url
$terraform output -raw rctf_instancer_auth_token
$terraform output -raw rctf_instancer_ca_certificate

instancerProvider:
  name: instancer/k8s-instancer
  options:
    apiUrl: https://203.0.113.10
    authToken: <rctf_instancer_auth_token>
    caCertificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

What Terraform provisions#

The example layers the GKE module, the k8s module, and the example-level resources in rctf-instancer.tf, dns.tf, and tls.tf:

Traefik is configured with three ports:

The wildcard certificate is provisioned manually instead of through cert-manager, so DNS provider credentials never have to live inside the cluster. The blast radius of a cluster compromise stays limited to whatever certificates Terraform has already issued.

Network policies#

The controller creates three NetworkPolicy resources in every instance namespace:

The exposed label is applied automatically based on whether a pod is named by any expose[] entry. The egress label comes from the per-pod egress: true flag in instancerConfig. Challenges that shouldn’t reach the internet leave it false.

Per-pod safety checklist#

Unlike Docker, Kubernetes’ PodSpec doesn’t have first-class fields for every resource cap, and the controller deploys what you give it verbatim. Set these on every pod you ship through instancerConfig.config.pods[]:

#!/bin/sh
ulimit -n 1024
exec /your/challenge "$@"

RBAC and the rCTF service account#

The example creates a single ServiceAccount named rctf in kube-system and a matching ClusterRole granting only what the API needs:

rule:
  api_groups: ['rctf-instancer.osec.io']
  resources: ['challengeinstances']
  verbs: ['create', 'get', 'delete', 'patch']

The rCTF API never reads or writes any other resource type. The kubernetes_secret_v1.rctf_token resource issues a kubernetes.io/service-account-token so the token doesn’t rotate. The value comes back through the rctf_instancer_auth_token Terraform output.

The controller itself runs with its own RBAC from apps/k8s-controller/config/. It needs broad permissions on namespaces, deployments, services, network policies, and Traefik IngressRoute, IngressRouteTCP, and Middleware resources so it can reconcile per-instance objects. The CRD lives in apps/k8s-controller/config/crd/bases/ and is generated by $make manifests.

Example challenge config (Konata)#

A complete instanced challenge as it would live in a Konata deployment repo. This is the web/mirror-temple config from the DiceCTF Quals 2026 challenge repository. Konata builds and pushes the image, then forwards instancer_config straight to rCTF, which hands it to the k8s-instancer provider.

challenges:
  - category: web
    name: mirror-temple
    author: arcblroth
    description: |
      stare long enough at the void and the void stares back
    attachments:
      files:
        - 'Dockerfile'
        - 'chall/src/'
    flags:
      rctf:
        file: flag.txt
    instancer_config:
      challenge_integration_id: '{{ challenge.name }}'
      timeout_milliseconds: 1800000
      extendable: true
      expose:
        - kind: https
          host_prefix: '{{ challenge.name }}'
          container_name: app
          container_port: 8080
      config:
        pods:
          - name: app
            egress: true
            ports:
              - protocol: TCP
                name: http-service
                port: 8080
            spec:
              restartPolicy: Always
              terminationGracePeriodSeconds: 0
              automountServiceAccountToken: false
              containers:
                - name: app
                  image: '{{ images[challenge.name] }}'
                  ports:
                    - containerPort: 8080
                  resources:
                    requests:
                      cpu: '500m'
                      memory: '500Mi'
                    limits:
                      cpu: '3'
                      memory: '2Gi'
                  readinessProbe:
                    tcpSocket:
                      port: 8080
                    initialDelaySeconds: 5
                    periodSeconds: 3
                  securityContext:
                    runAsNonRoot: true
                    readOnlyRootFilesystem: true
                    allowPrivilegeEscalation: false
                    capabilities:
                      drop:
                        - ALL

deployment:
  images:
    - build_context: .
      name: '{{ challenges[0].name }}'
      tag: latest
      registry_name: instancer-challenges
      platform: linux/amd64

Things worth pointing at in this example:

• egress: true on the pod opts it into the egress NetworkPolicy so the challenge can reach the public internet. Drop it for challenges that should be sealed off.
• Resource requests and limits are mandatory in practice. The controller schedules the pod normally, so an unset limit lets a single instance starve the node. Size them to the per-team load you expect at peak.
• readinessProbe keeps Traefik from routing to the pod before the app is up. Without it, the first request after creation often 502s while the container is still booting.
• securityContext locks the container down (read-only root FS, dropped capabilities, no root). The k8s-instancer namespace is already isolated by the per-namespace NetworkPolicy set, but a tight pod-level context is the second layer.
• {{ images[challenge.name] }} resolves to the fully-qualified registry path Konata pushed to (registries.instancer-challenges + the image name + tag).
• flags.rctf.file: flag.txt lets the flag live in a sibling file Konata reads at sync time, so the challenge directory stays self-contained.

For the rest of the Konata schema, see Konata.

Local development with Kind#

For controller iteration, the README in apps/k8s-controller/ uses Kind. Routing inside the cluster needs cloud-provider-kind so that LoadBalancer services get an external IP.

$go install sigs.k8s.io/cloud-provider-kind@latest

$cd apps/k8s-controller
$kind create cluster --name rctf --config kind-config.yaml

$cloud-provider-kind

$cd deploy/terraform/instancer/example
$terraform apply

$cd apps/k8s-controller
$make install
$make run ARGS="-instancer-host instancer.test"

$kubectl apply -f config/sample/rctf-instancer_v1_challengeinstance.yaml

Troubleshooting#

The controller exposes Kubernetes events through standard $kubectl describe output. Pair $kubectl describe challengeinstance <name> with the controller logs ($kubectl logs -n rctf-instancer-controller-system -l control-plane=controller-manager) to trace down any reconciliation failure.