Receive Cilium Gateway API Traffic Through Per-Node Public IPs

Key takeaways

• The baseline associates a distinct public IP with each control-plane VM through provider SDN or virtual networking and publishes every public IP in DNS
• A public IP missing from eth0 does not prove NAT; first inspect the destination address of the packet that the VM actually receives
• A packet addressed to the fixed IP can reach Cilium's 0.0.0.0:80/443 hostNetwork listener, while a preserved public /32 can require additional Linux local-delivery configuration
• Manage provider public IPs in cloud infrastructure inventory and DNS, not in LB-IPAM-oriented Gateway.spec.addresses
• Multiple A/AAAA records do not select healthy nodes like a central LB, so the required RTO can demand health-checked DNS or an external LB

Separate per-node public IPs from guest addresses

• A provider public IP and a guest OS interface address can exist in different management layers
  • The provider control plane associates 203.0.113.11 with the vNIC of cp-a
  • The guest can show only its fixed IP, 10.0.0.11/24, on eth0
  • This state neither means that the association is absent nor proves that NAT is in use

• Treat the baseline data path as public IP → provider SDN or virtual switch → target VM vNIC and fixed IP → Cilium hostNetwork Envoy
  • Depending on the provider implementation, the VM can observe either the fixed IP or the public /32 as the final destination
  • Confirm address translation, routing, and virtual-NIC behavior in the documentation for the specific provider

Inspect the destination address inside the VM

• Start by running tcpdump -ni any while sending an external request to confirm that the packet reaches the target VM

sudo tcpdump -ni any 'tcp port 80 or tcp port 443'
ip -br address
ip route show
kubectl get nodes -o wide
sudo ss -lntp '( sport = :80 or sport = :443 )'

• When the destination is the node fixed IP, inspect the path to the wildcard hostNetwork listener

  • A SYN to an address such as 10.0.0.11:443 calls for checks of the local firewall and 0.0.0.0:443 listener
  • A SYN without a response points toward the listener, local firewall, or return path
  • No packet points toward the public-IP association, provider firewall, security group, or network ACL

• When the destination remains the public /32, separately verify that Linux treats it as a local address

ip addr
ip route get local 203.0.113.11
ip route show table local

• Enabling hostNetwork alone cannot receive a public destination that Linux does not recognize as local
  • Apply any required public /32, local route, or extra virtual NIC according to the provider documentation
  • Adding a provider-managed address to eth0 without that requirement can create an address conflict or incorrect return path
  • net.ipv4.ip_nonlocal_bind=1 permits binding to a nonlocal address but does not turn packets into local delivery, so it is not a general fix

Enable hostNetwork only on intended nodes

• Cilium hostNetwork exposes each Gateway listener on all interfaces of selected nodes through 0.0.0.0 or ::
  • It automatically disables the default LoadBalancer Service mode because the modes are mutually exclusive
  • Listener ports must not clash across Gateways on the selected nodes
  • TCPRoute and UDPRoute are not compatible with host network mode

kubectl label node cp-a cp-b cp-c gateway.cilium.io/public=true

gatewayAPI:
  enabled: true
  hostNetwork:
    enabled: true
    nodes:
      matchLabels:
        gateway.cilium.io/public: "true"

envoy:
  enabled: true
  securityContext:
    capabilities:
      keepCapNetBindService: true
      envoy:
        - NET_BIND_SERVICE

• Binding to 80/443, or any port at or below 1023, requires the NET_BIND_SERVICE capability
  • Add NET_BIND_SERVICE to the installed chart's capability defaults instead of replacing the existing list
  • The example shows only the key values for standalone Envoy DaemonSet mode
  • Embedded Envoy mode adds the capability under securityContext.capabilities.ciliumAgent

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --values values-direct-public-ip.yaml

kubectl -n kube-system rollout status ds/cilium-envoy

Separate Gateway addresses from provider inventory

• Gateway.status.addresses in host network mode can contain InternalIP or ExternalIP values reported by Kubernetes Nodes

  • A Node that reports only its fixed address as InternalIP can produce only private Gateway status addresses
  • Cilium sorts node addresses and publishes no more than 16 of them in Gateway status
  • An absent provider public IP does not by itself indicate failed Envoy programming

• Do not declare a provider public IP in LB-IPAM-oriented Gateway.spec.addresses

  • Cilium implements spec.addresses with LB IPAM to request a Service VIP
  • The field does not create a provider-SDN association or VM route
  • Keep per-node public-IP associations in cloud infrastructure inventory

• Publish the distinct public IP associated with every node in DNS

app.example.com.  60  IN  A  203.0.113.11
app.example.com.  60  IN  A  203.0.113.12
app.example.com.  60  IN  A  203.0.113.13

• Issue certificates for the Gateway listener hostname rather than the public IP
  • HTTP-01 requires the challenge Route to work through every DNS target
  • DNS-01 is less dependent on per-node HTTP reachability

Verify firewalls and source IP by observation

• Allow the actual listener ports, 80/443, only from required source ranges in both provider and host firewalls

  • Restrict Kubernetes API 6443 to administrative VPN and node CIDRs
  • Do not expose etcd, kubelet, or Cilium management ports to the Internet

• Do not assume that the provider network preserves the original source IP because the behavior can vary by implementation

  • Inspect the SYN source address in a packet capture on the VM
  • Inspect the downstream remote address and forwarding headers in Cilium Envoy access logs
  • Use those observations to define the trust boundary for network policy, rate limiting, and audit logs

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: edge
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      hostname: "*.example.com"
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com

Accept the limits of DNS distribution

• Multiple A/AAAA records offer candidates but do not choose a healthy node for every request like a central LB

  • Resolvers and clients can cache addresses beyond the configured TTL
  • Existing TCP connections remain tied to a failed node regardless of DNS changes
  • Retry and address-rotation behavior differs among clients

• Use health-checked DNS or an external LB when the required RTO is shorter than simple DNS removal

  • Evaluate health detection, TTL, propagation, and client caching together for health-checked DNS
  • Prefer an external LB when consistent failover within seconds and connection-level health checks are required

• Directly exposed control-plane nodes share server resources between management and user traffic

  • Move public IPs and Gateway listeners to dedicated infrastructure nodes when possible
  • Otherwise configure Envoy connection, request-rate, and resource limits and monitor API-server protection metrics

Verify every public IP independently

• Pin every public IP so that DNS round robin cannot hide a single-node failure

kubectl get gateway -n edge public -o yaml
kubectl -n kube-system get pods -l k8s-app=cilium-envoy -o wide
sudo ss -lntp '( sport = :80 or sport = :443 )'

for ip in 203.0.113.11 203.0.113.12 203.0.113.13; do
  curl --fail-with-body \
    --connect-timeout 3 \
    --resolve app.example.com:443:$ip \
    https://app.example.com/healthz
done

• Each pinned request verifies TLS, Route selection, and the backend path together
  • Stop one node and measure how long health-checked DNS takes to remove the failed address
  • Confirm that a recovered node has its listener and current Envoy configuration before DNS re-entry
  • Compare the packet capture with Envoy access logs to establish source-IP preservation

Recommended action

• Confirm every per-node public-IP association and the destination observed by the VM before applying hostNetwork + node labels + public DNS
• If Linux does not recognize a preserved public /32 as local, first apply the provider-required route or vNIC configuration
• Add health-checked DNS or an external LB when the operational RTO is shorter than DNS failure removal

References

• Cilium Gateway API host network mode
• Cilium bind to privileged port
• Cilium Gateway API addresses support
• Kubernetes Node status addresses