rke2spray and GitOps Operations Reference

Key facts

• rke2spray owns hosts, RKE2, cluster-foundation add-ons, and the Argo CD bootstrap
• The GitOps repository owns platform, data, observability, and storage resources after the root Application
• The single-cluster RKE2, CNI, topology, and network values are declared under inventory/
• GitOps verifies 15 Argo CD Application resources including the root bootstrap, and 301Gi of application PVC requests
• Telemetry follows two persistent queues before ClickHouse, then a ClickHouse-owned ZSTD(3) and SeaweedFS S3 lifecycle
• The NFS endpoint, external L4 load balancer, and absence of external backups define the current availability and recovery boundaries

arc42 coverage map

01GoalsScope and non-goals

02ConstraintsRKE2, NFS and external systems

03ContextRepositories and operators

04StrategyTwo-stage reconciliation

05Building blocksAnsible, RKE2 and Argo CD

06RuntimeBootstrap and recovery flows

07DeploymentSync waves and service exposure

08ConceptsOwnership, secrets and GitOps

09DecisionsRKE2 versus Kubespray

10QualityTests and readiness gates

11RisksNFS and recovery boundaries

12GlossaryAPI names and meanings

1. Goals and verification baseline

• This page was verified on 2026-08-30 against the current main code in both repositories
  • jongminchung/rke2spray is the cluster source
  • The target repository's bootstrap/, apps/, and tests/ directories define the GitOps contract
  • When prose differs from executable code, inventory, Playbooks, manifests, and verification scripts are authoritative

2. Architecture constraints

• The Kubernetes distribution is fixed at RKE2 v1.36.3+rke2r1, with Cilium 1.20.1 as its CNI
  • jamie-kr-gitops/ansible/rke2spray.rev selects the verified rke2spray commit
• ClickHouse Server and Keeper use the supported 26.3.25 LTS line
  • Merge concurrency and memory reservations are bounded for the three-replica observability workload
• Control-plane API and Gateway traffic depend on a stable external TCP load balancer outside both repositories
  • A server must leave LB backend pools before node removal begins
• Persistent workloads currently share one managed NFS endpoint
  • durable-rwo and replica spreading do not isolate an NFS service failure or data loss
• Plaintext secrets are not stored in Git
  • Host bootstrap values use Ansible Vault, while Kubernetes application values use the SOPS and age boundary

3. System context and repository ownership

C4 · Level 1 · System context

Pause motion

Platform operatorReviews desired state and runs guarded operations

jamie-kr-gitopsOwns environment inventory and Kubernetes desired state

rke2sprayInterprets Kubespray-shaped input as an RKE2 lifecycle

RKE2 platformRuns Kubernetes API, Argo CD, platform and data workloads

External systemsGitHub, DNS, TCP load balancer and managed NFS

• Concurrent ownership of the same Kubernetes resource is not supported
  • An RKE2 manifest owner maps to /var/lib/rancher/rke2/server/manifests/<owner>
  • native-addons-handoff verifies owner, basename, and checksum during an ownership transfer
  • An ownership-only ledger entry transfers ownership without deleting the Kubernetes resource
  • A prune ledger entry is removable only with the exact confirmation value

4. The solution strategy switches reconciliation engines at API readiness

• rke2spray reconciles hosts and RKE2 desired state before the Kubernetes API exists
  • It validates Kubespray-shaped inventory and translates it into the RKE2 server and agent lifecycle
• The bootstrap gate turns API creation and platform readiness into executable conditions
  • It checks nodes, Cilium, CoreDNS, Gateway API, cert-manager, and NFS CSI readiness
• Argo CD continuously reconciles application desired state after API creation
  • The root Application creates role-scoped child Applications and sync waves
• Both engines consume one environment release without concurrently owning the same resource
  • This boundary is the condition for safely combining Ansible IaC and GitOps in one repository

C4 · Level 2 · Containers and ownership

Pause motion

Inventory + VaultHosts, topology and protected bootstrap inputs

rke2sprayAnsible playbooks, roles, validation and lifecycle adapters

RKE2 APIServer, embedded etcd, Cilium and native add-ons

Argo CD bootstrapProjects, repository credentials and Root Application

GitOps applicationsPlatform, data, observability, storage and routes

5. rke2spray building blocks

Topology

Versions and networking

Storage and snapshots

• durable-rwo and shared-rwx express workload intent and currently share one NFS failure domain
• Retain and onDelete: retain restrict PV deletion and do not provide NFS data backups

RKE2 native add-on profile

6. rke2spray operations runtime

• These targets are defined by the repository-root Makefile

• Cluster access does not copy a long-lived kubeconfig into the environment repository

  • kubeconfig_localhost: false and kubectl_localhost: false keep Ansible-controller artifacts disabled
  • Read-only checks should run the bundled kubectl remotely through scripts/run_with_ssh_jump.py
  • Local integration tests use operations/with-cluster-kubeconfig.sh, which retrieves /etc/rancher/rke2/rke2.yaml into XDG_RUNTIME_DIR, opens a loopback SSH tunnel, verifies /readyz, and removes the credential on exit
  • operations/cluster-kubeconfig.md is the detailed issuance, validation, cleanup, and incident-response runbook

• cluster-reset, cluster-recover, cluster-remove-node, add-on pruning, and failure probes can destroy or interrupt state

  • Each target requires a bounded target, confirmation value, or opt-in environment variable
  • Server removal requires prior removal from external LB backends on ports 80, 443, 6443, and 9345
  • Off-host snapshot transfer and retention are outside the rke2spray contract

6. GitOps handoff runtime

Argo CD declaration

Bootstrap gates

• extra_playbooks/gitops_bootstrap.yml requires all of the following conditions

  • RKE2 API /readyz returns ok
  • Every RKE2 Node reports Ready
  • Cilium and CoreDNS rollouts are complete
  • Gateway API CRDs report Established and the Cilium GatewayClass reports Accepted
  • cert-manager and NFS CSI rollouts are complete
  • NFS migration-gate status, database status, StorageClass UID, and cluster UID match the live cluster

• Secrets move from Ansible Vault into Argo CD Secrets

  • Read-only SSH private key for the Git repository
  • GitHub known-hosts value
  • SOPS age private key
  • Neither repository stores plaintext private keys

Download source

Animated runtime view · two reconciliation engines

Pause motion

1. ReviewInventory and manifests
2. ConvergeAnsible + RKE2
3. Ready gateAPI, nodes and add-ons
4. ReconcileArgo CD sync waves
5. VerifyHealth and disruption contracts

7. Deployment view: GitOps applications

Application inventory

• Applications with pruning disabled do not use a finalizer
• Applications with pruning enabled use resources-finalizer.argocd.argoproj.io
• Every Application uses automated sync and selfHeal: true

Workload topology

• Application PVC requests total 301Gi and the deployment ceiling is 400Gi
• PostgreSQL, ClickHouse, Keeper, MongoDB, and SeaweedFS Master and Volume replicas spread across three distinct workers
• SeaweedFS uses placement 001, retaining two volume copies on different racks for newly written S3 chunks

7. Deployment view: services and public routes

ClusterIP services

• Data services in observability, postgresql, and seaweedfs may not use NodePort or LoadBalancer
• The ClickHouse cold S3 endpoint is private and distinct from the public SeaweedFS S3 API

Gateway API routes

8. Cross-cutting observability and storage concepts

Authenticated durable ingestion

Observability runtime · authenticated durable path

Pause motion

1. SourcesApplication OTLP, node logs and cluster metrics
2. OTel Gateway ×2Batch, retry and 1Gi PVC queue per replica
3. ClickStack ingest ×2Bearer auth and 1Gi PVC exporter queue
4. ClickHouse ×3ReplicatedMergeTree on hot durable-rwo volumes
5. SeaweedFS S3Private gateway ×3 and 001 two-copy cold volumes

• Every telemetry signal enters through the two-replica OTel Gateway
  • Agent, Cluster Collector, and application OTLP traffic share the same batch, infinite retry, and 1Gi PVC queue boundary
  • The Gateway authenticates to ClickStack ingestion with the protected HyperDX API credential
• ClickStack ingestion is a two-replica StatefulSet with a second persistent queue boundary
  • Both ClickHouse exporters use file_storage, queue size 8192, and infinite retry
  • A ClickHouse outage after Gateway delivery therefore remains recoverable from the ingestion PVC queue
• Resource identity preserves application semantics
  • Collectors add k8s.cluster.name=jamie-kr and their platform role
  • They do not overwrite an SDK-provided service.namespace
• NetworkPolicies limit OTLP and S3 ingress
  • Only the OTel Gateway and HyperDX can reach ClickStack ingestion ports
  • Only ClickHouse can reach the private S3 data port, while the Cluster Collector can reach its metrics port

Compression and cold-storage lifecycle

ClickHouse lifecycle · compression and cold tier

Pause motion

1. Day 0Ingest replicated telemetry into the hot volume
2. Hot windowQuery recent parts from local ClickHouse PVCs
3. Day 2Recompress eligible parts with ZSTD level 3
4. Day 3Move parts to the cold_s3 volume
5. Day 14Delete table parts and owned S3 objects

• All nine ClickStack telemetry tables use the hot_cold ClickHouse storage policy
  • New replicated parts start on the default hot volume backed by durable-rwo
  • Eligible parts are recompressed with CODEC(ZSTD(3)) after two days
  • Parts move to the cold_s3 volume after three days
  • ClickHouse deletes table parts and their owned S3 objects after 14 days
• ClickHouse is the single lifecycle owner
  • SeaweedFS does not apply an independent bucket TTL that could remove objects before ClickHouse metadata changes
  • The PostSync retention Job creates the private bucket idempotently before applying table settings and TTL expressions
• Cold storage reduces hot PVC pressure but is not a backup
  • ClickHouse replicas use separate S3 prefixes and still depend on the same SeaweedFS and NFS failure domain
  • Deletion, corruption, or loss of the shared storage layer is not recoverable from Git

9. Architecture decisions for telemetry retention

10. Quality requirements and verification contracts

rke2spray static verification

GitOps static verification

Live-cluster verification

• tests/integration/verify.sh defines the following completion state

  • All 15 Argo CD Applications report Synced and Healthy
  • Stateful and stateless Application prune and finalizer policies match
  • Three-replica workloads are Ready on three distinct workers
  • Services excluded from public exposure remain ClusterIP
  • No Pod is Pending and every Node root filesystem remains below 60% usage
  • durable-rwo uses nfs.csi.k8s.io, Immediate, Retain, and onDelete=retain
  • All target PVCs use durable-rwo and total 301Gi
  • ClickStack ingestion has two ready replicas, while the private S3 gateway has three ready replicas on distinct workers
  • SeaweedFS master and filer commands retain the 001 two-copy replication placement
  • Collector self-metrics include the otelcol_* family in ClickHouse
  • All nine telemetry tables use hot_cold, ZSTD(3), three-day cold movement, and fourteen-day deletion
  • The cold_s3 disk is visible in system.disks
  • Gateway HTTPRoute resources report Accepted=True and ResolvedRefs=True

• Verification that writes or deletes state requires an explicit opt-in variable

  • The NFS performance gate requires RUN_NFS_PERFORMANCE_GATE=true
  • The database gate requires RUN_NFS_DATABASE_GATE=true
  • Tri-signal telemetry generation requires RUN_TELEMETRY_E2E=true
  • Workload disruption checks require RUN_DISRUPTIVE_TESTS=true

• The tri-signal E2E gate proves the complete write path

  • A digest-pinned telemetrygen Pod emits logs, metrics, and traces with a unique service.name
  • Queries require all three signal families and k8s.cluster.name=jamie-kr to appear in ClickHouse
  • Generator Pods are removed after the test, while their small telemetry payload follows the normal retention lifecycle

11. Risks and recovery boundaries

12. Core glossary

Documentation update contract

• Update this page's updatedAt and verifiedAt when any of the following code changes
  • Version, topology, network, or storage inputs in inventory/
  • Public operator targets or required variables in Makefile
  • Readiness gates in gitops_bootstrap.yml
  • Applications, waves, chart versions, or prune policies in bootstrap/templates/applications.yaml
  • Replicas, PVCs, Services, or public routes under apps/
  • Completion assertions in tests/integration/verify.sh
  • Kubeconfig issuance or cleanup behavior in operations/cluster-kubeconfig.md and operations/with-cluster-kubeconfig.sh