Infrastructure

These are the infrastructure-side failure modes: cloud quota, cluster posture, per-challenge runtime limits, host hardening, and the proxy chain in front of the platform. Most of them are recoverable, but only if you find them before participants do.

Cloud providers#

Not requesting quota in advance#

New GCP, AWS, and Azure accounts often start with low default quotas. If you hit quota during the event, autoscaling stops and new instances may fail to start. Quota reviews can take multiple days, so request increases at least two weeks before the event.

For GCP, check project-level quota in the IAM quotas page. Pay attention to CPUS, IN_USE_ADDRESSES, PERSISTENT_DISK_SSD_GB, GPU quotas in the selected region, and the per-region GKE SUBNETWORKS cap.

Quota belongs to the project. Sponsored credits don’t automatically raise project limits. File the request with the event date, expected team count, expected concurrent workload, and selected region, then verify the approved limit in the same UI.

For AWS, Service Quotas requests are tied to an account-region pair. Request quota in every region you plan to use, especially for vCPU limits by instance family, Elastic IPs, and EBS volume capacity.

Region choice matters. Some instance families are easier to approve in one region than another, so pick the region only after checking whether the required capacity is realistic there.

For Azure, check regional vCPU limits by family before committing to a region. Like the other providers, credit or sponsorship doesn’t automatically grant the compute quota an event needs.

Ask for more than the steady-state estimate. Burst, retries, node replacement, and pre-pulled images can chew through more capacity than a simple team count would suggest.

Warning (Request headroom)

Ask for roughly 1.5 to 2x the worst-case estimate. The margin covers burst, retries, pre-pulled images, and replacement capacity during node churn.

GCP trial projects are a common edge case. If a project is still on a free-trial billing account, quota-increase requests can be rejected regardless of timing because trial projects are intentionally kept at lower limits.

Upgrade to a paid billing account before requesting quota increases. Free credits can still apply, but the billing account class matters during review.
Run a small always-on VM, such as an e2-small, for 3 to 7 days before filing the request. Consistent billable usage makes the project look like an active workload rather than a brand-new account.
File the request with context. Reference the active workload and the event date in the justification. A note like currently running production workloads, scaling up for an event on YYYY-MM-DD reads better than a bare capacity request. Projects that jump straight from no usage to a very large quota request are more likely to be denied or routed to a slower manual review queue.

Exposing the instance-metadata service to challenges#

Anything reachable from a challenge container can also reach http://169.254.169.254 on AWS, GCP, or Hetzner, or http://[fd00:ec2::254] on IPv6. Those metadata services can hand out short-lived credentials for the node’s service account.

Important (Block metadata service access)

Block egress from challenge containers to 169.254.0.0/16 and the IPv6 metadata addresses. Verify the rule from inside a test pod or challenge container, not only from the host.

For shared-remote VPS or bare-VM challenges without Kubernetes NetworkPolicy, drop IMDS at the host firewall or run the host without an attached service account.

From a test pod, run curl http://169.254.169.254. The request should fail. If the cluster is dual-stack, also test the IPv6 metadata address. On bare-metal clusters, this catches CNI configurations that silently ignore the intended policy.

For rCTF Kubernetes instancer deployments, the Kubernetes instancer ships a NetworkPolicy per instance namespace that excludes RFC1918, CGNAT, and link-local ranges by default. That covers IMDS on IPv4. The Terraform module also enables GKE workload_metadata_concealment. Opt into public egress with egress: true on the pod rather than removing the policy.

The bundled GKE Terraform module provisions an IPv4-only cluster, so the IPv6 IMDS address [fd00:ec2::254] is unreachable from challenge pods because the cluster has no IPv6. If you switch the cluster to dual-stack, the shipped NetworkPolicy has no IPv6 allow rule and will implicitly deny IPv6 egress. That keeps IPv6 IMDS blocked, but challenges that need IPv6 internet egress will also fail until you add an explicit IPv6 IPBlock with matching fd00::/8 and fe80::/10 exceptions.

Kubernetes#

Skipping per-pod resource limits#

A pod without resources.limits can use whatever the node has free. A fork bomb or memory leak in one instance can take down other instances on the same node.

The usual shape of this bug is a PodSpec with only requests, limits sized from a single happy-path test, and no terminationGracePeriodSeconds cap.

The k8s-instancer instancerConfig.config.pods[] field is a real PodSpec. Set resources.requests/limits directly.

Mounting broad service-account tokens#

Every pod in a namespace can receive the default service-account token at /var/run/secrets/kubernetes.io/serviceaccount/. If that service account has broad permissions, or if the cluster over-grants system:authenticated, a compromised challenge turns straight into Kubernetes API access.

Important (Disable tokens unless needed)

Set automountServiceAccountToken to false unless the challenge genuinely needs the in-cluster API. Audit system:authenticated cluster-role bindings with kubectl get clusterrolebindings -o yaml | grep -A4 system:authenticated, and remove bindings with meaningful verbs. Also check that temporary controller RBAC changes from debugging were reverted before the event.

The k8s-instancer instancerConfig.config.pods[] field accepts the standard PodSpec, including automountServiceAccountToken.

Runtime limits#

Missing runtime abuse limits#

CPU and memory limits aren’t enough on their own. A participant can run a fork bomb like :(){ :|:& };:, fill /tmp/ with dd if=/dev/zero of=/tmp/x, or open a large number of file descriptors from inside a challenge process. Those behaviors need runtime limits, not assumptions about participant behavior.

For Docker services, set pids_limit so fork bombs can’t exhaust the host kernel.pid_max. Set ulimits.nofile so a process can’t exhaust the file-descriptor table. If /tmp/ is writable, mount a size-limited tmpfs. Bare-metal challenges need the same storage cap with a tmpfs size= option.

For Kubernetes services, set resources.limits.ephemeral-storage on pods and configure kubelet podPidsLimit at the cluster level. Use readOnlyRootFilesystem where practical and set terminationGracePeriodSeconds so cleanup can’t be delayed indefinitely.

For nsjail services, use --rlimit_nproc, --rlimit_nofile, and --time_limit. Shared remote services should have a per-connection time limit so slow or long-lived TCP connections cannot starve the connection table.

Before release, test challenge services with intentionally abusive clients. Include :(){ :|:& };:, dd if=/dev/zero of=/tmp/x, for i in $(seq 100000); do exec {fd}<>/dev/null; done, and slow TCP connections where the challenge uses a shared remote.

The Docker instancer defaults to 6m memory, 1.0 CPU, 1024 PIDs, and 1024 nofile per service when not overridden. It also exposes mem_limit, cpus, pids_limit, and ulimits for tuning. The GKE Terraform module sets kubelet podPidsLimit to 1024 cluster-wide through gcp_instancer_pod_pids_limit, so a fork bomb inside a challenge pod hits that cap before the node kernel.pid_max.

The Kubernetes side otherwise leaves several settings to the challenge author. Ephemeral-storage limits, readOnlyRootFilesystem, FD ulimits, and terminationGracePeriodSeconds are not defaulted. See the per-pod safety checklist on the Kubernetes instancer page for the full list of fields to set.

When in doubt, treat every challenge image as already compromised. Assume participants can get code execution inside the challenge container, even when the intended solution isn’t RCE.

Host#

Improper host hardening#

A cloud VM with an unpatched kernel or unnecessary privileges can turn challenge code execution into host access. Container-escape and local privilege-escalation issues get patched regularly, but unpatched hosts stay exposed.

Run unattended upgrades (or an equivalent) on challenge hosts, then reboot into the newer kernel before the event. Don’t mount /var/run/docker.sock into challenge containers. Avoid --privileged, --cap-add=SYS_ADMIN, --pid=host, and --net=host unless someone has reviewed the security impact. Keep runc, Docker, and containerd on patched versions.

Public local privilege-escalation exploits can show up within days of disclosure. A container with Docker socket access can also start a container with docker run -v /:/host and walk out to the host filesystem. Pin minimum runtime versions and verify them before the event.

Challenge machine runs out of disk space#

Disk exhaustion is easy to miss because CPU and memory dashboards can stay quiet while logs, images, and caches grow in the background.

Docker json-file logs have no rotation unless max-size and max-file are set in /etc/docker/daemon.json, so a noisy challenge can fill /var/lib/docker/containers/*/.log for the lifetime of the container. journald grows in the same way unless SystemMaxUse= is configured. Rebuilds and redeploys also leave dangling images and build cache in /var/lib/docker/, so schedule docker system prune or keep an eye on disk usage before the event.

Network#

Client-IP extraction not tested end-to-end#

Every layer in front of the platform (Cloudflare, an L7 load balancer, nginx, Traefik) can rewrite the source address. If the chain isn’t configured correctly, the platform and challenges may see the proxy IP rather than the participant IP. The failure is often silent until rate limits or bans behave incorrectly. A bad chain can ban a shared Cloudflare edge IP and lock participants out, or make every request appear to come from 127.0.0.1. The challenge-side version is also important. A web challenge that trusts X-Forwarded-For without validating the upstream hop can be spoofed by participants.

Send a request through the real DNS from a known external IP and confirm the admin or audit log shows that IP. Then test a challenge through the same path and confirm the challenge sees the same address. A staging path without the full proxy chain isn’t enough for this check.

Important (Trust only the edge you control)

If USE_PROXY_HEADERS=true is enabled without a locked-down trusted-proxy CIDR, participants can spoof X-Forwarded-For. The same applies to nginx set_real_ip_from and Traefik forwardedHeaders.trustedIPs when the trusted range is wider than the proxy layer you control. With Cloudflare, restore CF-Connecting-IP so the platform does not log only Cloudflare edge IPs.

Not being prepared for traffic#

The first few minutes after start pile up concurrent registrations, logins, challenge reads, and leaderboard reads. Load-test those paths from a separate machine before the event, and size the challenge cluster for expected teams × overlapping instanced challenges.

Turn on captcha for sensitive unauthenticated actions, verify rate limits on sensitive platform paths, and configure trusted proxy IP ranges before the load test. The test should use the production proxy chain, not a local or staging shortcut.

rCTF helps with this through the leaderboard worker and Redis cache, nginx brotli and gzip support, immutable cache headers, Docker and Kubernetes instancers for per-team lifecycles, GKE module autoscaling, captcha integrations for reCAPTCHA, hCaptcha, and Turnstile, and default route rate limits.