jongminchungtech
KO
Documents in this area

Cilium Gateway API Part 4: Per-Node Cloud Public IPs Without an LB

Receive external traffic with Cilium hostNetwork and DNS when cloud SDN associates a distinct public IP with each node VM.

Updated Source

Key takeaways

  • The baseline associates a distinct public IP with each control-plane VM through provider SDN or virtual networking and publishes every public IP in DNS
  • A public IP missing from eth0 does not prove NAT; first inspect the destination address of the packet that the VM actually receives
  • A packet addressed to the fixed IP can reach Cilium's 0.0.0.0:80/443 hostNetwork listener, while a preserved public /32 can require additional Linux local-delivery configuration
  • Manage provider public IPs in cloud infrastructure inventory and DNS, not in LB-IPAM-oriented Gateway.spec.addresses
  • Multiple A/AAAA records do not select healthy nodes like a central LB, so the required RTO can demand health-checked DNS or an external LB

Separate per-node public IPs from guest addresses

  • A provider public IP and a guest OS interface address can exist in different management layers
    • The provider control plane associates 203.0.113.11 with the vNIC of cp-a
    • The guest can show only its fixed IP, 10.0.0.11/24, on eth0
    • This state neither means that the association is absent nor proves that NAT is in use
Provider SDN steering per-node public IPs to Cilium host network listeners
  • Treat the baseline data path as public IP → provider SDN or virtual switch → target VM vNIC and fixed IP → Cilium hostNetwork Envoy
    • Depending on the provider implementation, the VM can observe either the fixed IP or the public /32 as the final destination
    • Confirm address translation, routing, and virtual-NIC behavior in the documentation for the specific provider

Inspect the destination address inside the VM

  • Start by running tcpdump -ni any while sending an external request to confirm that the packet reaches the target VM
sudo tcpdump -ni any 'tcp port 80 or tcp port 443'
ip -br address
ip route show
kubectl get nodes -o wide
sudo ss -lntp '( sport = :80 or sport = :443 )'

  • When the destination is the node fixed IP, inspect the path to the wildcard hostNetwork listener

    • A SYN to an address such as 10.0.0.11:443 calls for checks of the local firewall and 0.0.0.0:443 listener
    • A SYN without a response points toward the listener, local firewall, or return path
    • No packet points toward the public-IP association, provider firewall, security group, or network ACL

  • When the destination remains the public /32, separately verify that Linux treats it as a local address

ip addr
ip route get local 203.0.113.11
ip route show table local
  • Enabling hostNetwork alone cannot receive a public destination that Linux does not recognize as local
    • Apply any required public /32, local route, or extra virtual NIC according to the provider documentation
    • Adding a provider-managed address to eth0 without that requirement can create an address conflict or incorrect return path
    • net.ipv4.ip_nonlocal_bind=1 permits binding to a nonlocal address but does not turn packets into local delivery, so it is not a general fix

Enable hostNetwork only on intended nodes

  • Cilium hostNetwork exposes each Gateway listener on all interfaces of selected nodes through 0.0.0.0 or ::
    • It automatically disables the default LoadBalancer Service mode because the modes are mutually exclusive
    • Listener ports must not clash across Gateways on the selected nodes
    • TCPRoute and UDPRoute are not compatible with host network mode
kubectl label node cp-a cp-b cp-c gateway.cilium.io/public=true
values-direct-public-ip.yaml
gatewayAPI:
  enabled: true
  hostNetwork:
    enabled: true
    nodes:
      matchLabels:
        gateway.cilium.io/public: "true"

envoy:
  enabled: true
  securityContext:
    capabilities:
      keepCapNetBindService: true
      envoy:
        - NET_BIND_SERVICE
  • Binding to 80/443, or any port at or below 1023, requires the NET_BIND_SERVICE capability
    • Add NET_BIND_SERVICE to the installed chart's capability defaults instead of replacing the existing list
    • The example shows only the key values for standalone Envoy DaemonSet mode
    • Embedded Envoy mode adds the capability under securityContext.capabilities.ciliumAgent
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --values values-direct-public-ip.yaml

kubectl -n kube-system rollout status ds/cilium-envoy

Separate Gateway addresses from provider inventory

  • Gateway.status.addresses in host network mode can contain InternalIP or ExternalIP values reported by Kubernetes Nodes

    • A Node that reports only its fixed address as InternalIP can produce only private Gateway status addresses
    • Cilium sorts node addresses and publishes no more than 16 of them in Gateway status
    • An absent provider public IP does not by itself indicate failed Envoy programming

  • Do not declare a provider public IP in LB-IPAM-oriented Gateway.spec.addresses

    • Cilium implements spec.addresses with LB IPAM to request a Service VIP
    • The field does not create a provider-SDN association or VM route
    • Keep per-node public-IP associations in cloud infrastructure inventory

  • Publish the distinct public IP associated with every node in DNS

app.example.com.  60  IN  A  203.0.113.11
app.example.com.  60  IN  A  203.0.113.12
app.example.com.  60  IN  A  203.0.113.13
  • Issue certificates for the Gateway listener hostname rather than the public IP
    • HTTP-01 requires the challenge Route to work through every DNS target
    • DNS-01 is less dependent on per-node HTTP reachability

Verify firewalls and source IP by observation

  • Allow the actual listener ports, 80/443, only from required source ranges in both provider and host firewalls

    • Restrict Kubernetes API 6443 to administrative VPN and node CIDRs
    • Do not expose etcd, kubelet, or Cilium management ports to the Internet

  • Do not assume that the provider network preserves the original source IP because the behavior can vary by implementation

    • Inspect the SYN source address in a packet capture on the VM
    • Inspect the downstream remote address and forwarding headers in Cilium Envoy access logs
    • Use those observations to define the trust boundary for network policy, rate limiting, and audit logs
gateway-direct-public-ip.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: edge
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      hostname: "*.example.com"
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com

Accept the limits of DNS distribution

  • Multiple A/AAAA records offer candidates but do not choose a healthy node for every request like a central LB

    • Resolvers and clients can cache addresses beyond the configured TTL
    • Existing TCP connections remain tied to a failed node regardless of DNS changes
    • Retry and address-rotation behavior differs among clients

  • Use health-checked DNS or an external LB when the required RTO is shorter than simple DNS removal

    • Evaluate health detection, TTL, propagation, and client caching together for health-checked DNS
    • Prefer an external LB when consistent failover within seconds and connection-level health checks are required

  • Directly exposed control-plane nodes share server resources between management and user traffic

    • Move public IPs and Gateway listeners to dedicated infrastructure nodes when possible
    • Otherwise configure Envoy connection, request-rate, and resource limits and monitor API-server protection metrics

Verify every public IP independently

  • Pin every public IP so that DNS round robin cannot hide a single-node failure
kubectl get gateway -n edge public -o yaml
kubectl -n kube-system get pods -l k8s-app=cilium-envoy -o wide
sudo ss -lntp '( sport = :80 or sport = :443 )'
for ip in 203.0.113.11 203.0.113.12 203.0.113.13; do
  curl --fail-with-body \
    --connect-timeout 3 \
    --resolve app.example.com:443:$ip \
    https://app.example.com/healthz
done
  • Each pinned request verifies TLS, Route selection, and the backend path together
    • Stop one node and measure how long health-checked DNS takes to remove the failed address
    • Confirm that a recovered node has its listener and current Envoy configuration before DNS re-entry
    • Compare the packet capture with Envoy access logs to establish source-IP preservation

Recommended action

  • Confirm every per-node public-IP association and the destination observed by the VM before applying hostNetwork + node labels + public DNS
  • If Linux does not recognize a preserved public /32 as local, first apply the provider-required route or vNIC configuration
  • Add health-checked DNS or an external LB when the operational RTO is shorter than DNS failure removal

References