How-torke2spray
Operate the cluster lifecycle
Check health, scale, upgrade, snapshot, recover, and remove nodes from an RKE2 cluster.
Core summary
- Every change begins with an Inventory review, Vault check, preflight, and current health check.
- Add workers through
scale.ymland add servers one at a time throughcluster.yml. - Upgrades sequence cordon, drain, release replacement, readiness verification, and uncordon.
- Recovery and removal require topology-specific prerequisites and exact confirmation.
- Do not treat CNI or cluster-CIDR changes as ordinary convergence of an existing cluster.
Follow the common execution sequence
Review Inventory and inline Vault diffs
Validate local Vault structure and remote topology
make inventory-vault-check make inventory-preflightPreserve the current cluster state
make cluster-healthConfirm targets and scope, then run the lifecycle
Verify completion with the same health checks
Select the operation entrypoint
| Task | Playbook or target | RKE2 behavior |
|---|---|---|
| Install or reconverge | cluster.yml / make cluster-converge | Configure servers and agents, then verify readiness |
| Add worker | scale.yml --limit=<worker> | Converge only the selected agent batch |
| Add server | cluster.yml --limit=<server> | Join one server at a time |
| Upgrade | upgrade-cluster.yml / make cluster-upgrade | Sequential drain and release replacement |
| Health | make cluster-health | Read-only service, API, Node, and etcd checks |
| Snapshot | make cluster-snapshot | Create an RKE2 embedded-etcd snapshot |
| Recovery | recover-control-plane.yml | Recover embedded etcd from a selected snapshot |
| Remove node | remove-node.yml | Drain, verify membership, and uninstall RKE2 |
| Reset all | reset.yml | Stop services and run the official uninstaller |
Compare Kubespray and rke2spray operations
| Operation or scenario | Kubespray baseline behavior | rke2spray behavior | Compatibility |
|---|---|---|---|
cluster.yml | Prepares hosts and the container runtime, installs etcd, runs kubeadm, installs the CNI, and applies cluster apps | Configures checksum-locked RKE2 servers and agents, uses bundled containerd, joins embedded-etcd members, and verifies services, the API, and Nodes | Adapter |
scale.yml | Adds Inventory-selected Kubernetes nodes through the Kubespray node lifecycle | Adds only selected kube_node workers as RKE2 agents; servers continue to use cluster.yml | Adapter |
upgrade-cluster.yml | Upgrades kubeadm-managed Kubernetes components and the selected runtime lifecycle | Replaces the checksum-locked RKE2 release one server at a time and agents in bounded batches | Adapter |
remove-node.yml with a reachable node | Drains the Node, removes an etcd member when applicable, and resets the target host | Drains the Node, verifies the exact embedded-etcd member, deletes the Node, and runs the official RKE2 uninstaller | Adapter |
remove-node.yml with an unreachable server | Cannot complete the target-host reset; the operator must move to control-plane recovery or repair host access | Does not pretend that target-host cleanup succeeded; use recover-control-plane.yml with the server fenced and removed from load balancer backends | Explicit recovery boundary |
recover-control-plane.yml with quorum retained | For non-kubeadm etcd, checks endpoint health, removes broken members, and reruns cluster.yml | Verifies exact peer membership, quorum, and learner absence; creates a protected snapshot, removes the exact dead member from a survivor, and joins the replacement through cluster.yml | Safety-expanded adapter |
recover-control-plane.yml after quorum loss | For non-kubeadm etcd, creates or accepts a snapshot, restores one etcd member, and reruns cluster.yml | Validates snapshot ownership, mode, and checksum; performs RKE2 cluster-reset, rejoins installed servers, and converges clean replacements through cluster.yml | RKE2 restore adapter |
reset.yml | Runs the Kubespray reset lifecycle for kubeadm, services, networking, and the selected runtime | Stops RKE2 services and runs the official RKE2 uninstaller | Adapter |
| Health and snapshot helpers | No equivalent stable root Playbook API in this compatibility contract | Keeps read-only health checks and RKE2 snapshot creation under extra_playbooks/ and Make targets | Kubespray extension point |
Compatibility scope
The shared Playbook name preserves the public operational intent, not the kubeadm task implementation. Consult Compatibility before assuming that an upstream variable or internal Role has an RKE2 equivalent.
Enforce destructive-operation boundaries
- Remove only one server at a time and leave at least two servers.
- Do not remove the bootstrap server without a separate lifecycle-identity review.
- With an external load balancer, first remove the server from the
80,443,6443, and9345backends. - Do not use
recover-control-plane.ymlfor an external datastore. - Never bypass confirmations for
reset,remove,recover, or AddOn pruning.
Operational boundary
Off-host snapshot copies and long-term retention belong to a separate backup system; creating a local snapshot alone does not complete disaster recovery.
Conclusion
- Preserve evidence from checks before and after each change instead of treating the command as the whole operation.
- When a prerequisite is unclear, check Compatibility first.