Automatic Kubelet Reservation Configuration

Bash script that dynamically configures system-reserved and kube-reserved on Kubernetes v1.32+ nodes.

Overview

The script automates kubelet reservation sizing. It:

Detects current node resources (vCPU, RAM, cgroup mode).
Calculates system and kube reservations using production proven formulas (GKE, EKS, OpenShift).
Adapts the result based on a desired pod density or a custom density factor.
Generates a full kubelet configuration file and preserves existing tweaks.
Applies the configuration with automatic validation, restart and backup/rotation logic.
Skips needless rewrites: if the generated config matches the live kubelet config, the script now exits early without creating extra backups or restarting kubelet.

Why this script?

Misconfigured reservations often lead to the fastest failure scenarios on Kubernetes:

Under-sized → OOM kills, eviction storms, NodeNotReady.
Over-sized → Large allocatable drop and wasted capacity.

This script applies the vendor reference formulas and enforces safe guardrails so you can keep nodes stable while maintaining usable capacity.

Prerequisites

Operating system

Ubuntu 20.04+ with systemd (cgroup v2 ready)
Linux kernel 5.x+

Kubernetes

Kubernetes v1.26 or newer (validated on v1.32)
containerd (recommended) or CRI-O
cgroupDriver: systemd

Dependencies

The script auto-installs its dependencies (bc, jq, yq v4) during the first run. No manual action is required.

Manual install is possible when internet access is restricted:

sudo apt update
sudo apt install -y bc jq

ARCH=$(uname -m)
case "$ARCH" in
  x86_64|amd64)   YQ_BIN=yq_linux_amd64 ;;
  arm64|aarch64)  YQ_BIN=yq_linux_arm64 ;;
  *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;;
esac

sudo wget -qO /usr/local/bin/yq "https://github.com/mikefarah/yq/releases/download/v4.44.3/${YQ_BIN}"
sudo chmod +x /usr/local/bin/yq
yq --version

Ubuntu packages ship yq v3 (Python) which is not compatible with this project. The script automatically replaces it with the mikefarah v4 binary and validates the SHA256 checksum.

Permissions

Run the script with root privileges:

sudo ./kubelet_auto_config.sh

Installation

Method 1 – direct download

curl -O https://github.com/MacFlurry/reserved-sys-kube/raw/main/kubelet_auto_config.sh
chmod +x kubelet_auto_config.sh
./kubelet_auto_config.sh --help

Method 2 – Git clone

git clone https://github.com/MacFlurry/reserved-sys-kube.git
cd reserved-sys-kube
chmod +x kubelet_auto_config.sh rollback-kubelet-config.sh remove-kubelet-auto-config.sh

Method 3 – Push to every node

NODES="node1 node2 node3"
for node in $NODES; do
  scp kubelet_auto_config.sh rollback-kubelet-config.sh remove-kubelet-auto-config.sh root@$node:/usr/local/bin/
  ssh root@$node "chmod +x /usr/local/bin/kubelet_auto_config.sh /usr/local/bin/rollback-kubelet-config.sh /usr/local/bin/remove-kubelet-auto-config.sh"
done

Usage

sudo ./kubelet_auto_config.sh [OPTIONS]

Key options:

Option	Description
`--profile <gke	eks
`--density-factor <float>`	Directly set the density multiplier (0.1 – 5.0).
`--target-pods <int>`	Ask the script to compute a density factor that satisfies a target pod count.
`--node-type <control-plane	worker
`--backup`	Preserve timestamped backups instead of rotating them only.
`--no-kubelet-restart`	Skip the kubelet restart (useful when invoked via systemd before kubelet starts).
`--dry-run`	Generate the config, display it, but do not apply it.
`--no-require-deps`	Continue even if dependencies cannot be installed (lab only).
`--wait-timeout <seconds>`	Custom kubelet restart timeout (default 60).

The script is idempotent: running it again overwrites the configuration with the latest calculation while preserving existing custom sections.

Available profiles

gke – Google reference formulas (balanced CPU/memory).
eks – Amazon reference formulas with extra CPU for AWS services.
conservative – Maximum isolation for noisy neighbors and best effort workloads.
minimal – Only safeguard resources for kubelet and core system daemons (useful for small workers or edge nodes).

Each profile can be combined with a density factor or target pod count to match your workload patterns.

Density factor

density-factor acts as an additional multiplier on top of the chosen profile. It enables three typical workflows:

Target pods: --target-pods 110 computes a safe multiplier to keep allocatable in check for that target.
Manual multiplier: --density-factor 1.25 increases every reservation by 25%.
Safety rails: The script prevents impossible combinations (e.g., reservations >= capacity) and fails fast with a clear message.

Recommendations:

Control-plane nodes: keep the density factor ≤ 1.0 to leave headroom for static pods.
High density workers: start around 1.2 and adjust using the post deployment validation commands.

Usage examples

# Minimal configuration on a worker
sudo ./kubelet_auto_config.sh --profile minimal

# Conservative profile with automatic pods target
sudo ./kubelet_auto_config.sh --profile conservative --target-pods 80

# Control-plane enforcement with backup
sudo ./kubelet_auto_config.sh --node-type control-plane --backup

# Dry run with explicit density
sudo ./kubelet_auto_config.sh --density-factor 1.4 --dry-run

Cluster deployment

Copy the script to every node (see installation section).
Execute it with the desired parameters.
Watch the logs with journalctl -u kubelet -f.
Confirm allocatable and taints using kubectl describe node <name>.

For large fleets consider using an Ansible playbook or a DaemonSet that wraps the script (samples are provided under ansible/ and daemonset/).

Optional: systemd automation

The systemd/ directory ships a templated unit and ready-to-use environment files so the kubelet reservations are recalculated automatically.

systemd/install-kubelet-auto-config.sh control-plane installs kubelet-auto-config@control-plane.service with --profile gke --backup --no-kubelet-restart (node role auto-detected).
systemd/install-kubelet-auto-config.sh worker installs kubelet-auto-config@worker.service with --profile gke --target-pods 110 --backup --no-kubelet-restart.
Each service is Type=oneshot, runs before kubelet, and uses --no-kubelet-restart so there is no recursive restart loop. Every time kubelet starts (boot, systemctl restart kubelet, or a kubelet crash/restart), systemd re-runs the script, reapplies the config, and then kubelet starts normally.

sudo ./systemd/install-kubelet-auto-config.sh control-plane   # default control-plane profile
sudo ./systemd/install-kubelet-auto-config.sh worker          # worker profile (target 110 pods ≈ density 1.2)
sudo systemctl status kubelet-auto-config@worker.service

To customize the arguments, edit /etc/systemd/system/kubelet-auto-config@<profile>.service.d/env.conf and reload systemd:

sudo vim /etc/systemd/system/kubelet-auto-config@worker.service.d/env.conf
sudo systemctl daemon-reload
sudo systemctl restart kubelet-auto-config@worker.service

The installer copies the service template from the repository, so you can ship it via cloud-init or automation by copying the systemd/ folder and running the installer once per node role.

Cloud-init step-by-step

Copy the scripts (via write_files or a tarball) to /usr/local/lib/kubelet-auto-config/.

Install the binaries:

install -m 0755 /usr/local/lib/kubelet-auto-config/kubelet_auto_config.sh /usr/local/bin/
install -m 0755 /usr/local/lib/kubelet-auto-config/rollback-kubelet-config.sh /usr/local/bin/
 install -m 0755 /usr/local/lib/kubelet-auto-config/remove-kubelet-auto-config.sh /usr/local/bin/

Run the installer once:
- Control-plane nodes: sudo /usr/local/lib/kubelet-auto-config/systemd/install-kubelet-auto-config.sh control-plane
- Workers: sudo /usr/local/lib/kubelet-auto-config/systemd/install-kubelet-auto-config.sh worker
Done – systemd now relaunches kubelet_auto_config.sh before kubelet starts and on every systemctl restart kubelet.

Post deployment validation

Run the following commands after every rollout:

kubectl get nodes
kubectl describe node <name> | grep -A3 Allocatable
journalctl -u kubelet -n 100
sudo systemd-cgls | grep -E "kubelet|kubepods"

Confirm that:

system-reserved and kube-reserved match the expected values.
The kubelet process runs inside kubelet.slice (the script configures a drop-in if necessary).
Pods remain schedulable and the cluster reaches Ready state.

Rollback

Use rollback-kubelet-config.sh to restore a previous version:

sudo ./rollback-kubelet-config.sh             # restore the latest rotating backup
sudo ./rollback-kubelet-config.sh --index 2   # restore `.last-success.2`
sudo ./rollback-kubelet-config.sh --dry-run   # preview

Features:

Rotating backups: /var/lib/kubelet/config.yaml.last-success.{0..3}
Permanent backups when --backup is passed: /var/lib/kubelet/config.yaml.backup.YYYYMMDD_HHMMSS
Safety checks ensure the selected file exists and is readable before copying.

Removal

Use remove-kubelet-auto-config.sh when you need to remove every component (scripts + systemd units) without touching the running kubelet:

sudo ./remove-kubelet-auto-config.sh             # cleanup with live changes
sudo ./remove-kubelet-auto-config.sh --dry-run   # preview actions

The script:

Disables every kubelet-auto-config@*.service unit and removes their drop-ins/env files.
Deletes /usr/local/bin/{kubelet_auto_config.sh,rollback-kubelet-config.sh,remove-kubelet-auto-config.sh} and /usr/local/lib/kubelet-auto-config/.
Cleans the lock file under /var/lock.
Leaves /var/lib/kubelet/config.yaml and your workloads untouched (no kubelet restart is attempted).

After the cleanup, you can optionally run rollback-kubelet-config.sh --index 1 during a maintenance window if you want to restore the pre-script kubelet config. Otherwise the last applied configuration keeps running until you change it manually.

FAQ

Q. Do I have to stop the kubelet manually?
No. The script restarts it and waits up to the configured timeout.

Q. Does it work on ARM nodes?
Yes. Calculations rely on bc and normalized integers to avoid arithmetic issues observed on ARM64.

Q. Can I version the generated config?
Yes, the script only touches /var/lib/kubelet/config.yaml and the backup files. Use any configuration management workflow you prefer.

Q. What about Windows nodes?
Not supported. The project targets Linux nodes with systemd.

Why is kube-reserved not enforced on control-plane nodes?

This script follows a conservative approach for control-plane nodes:

On workers: Both system-reserved AND kube-reserved are enforced
On control-planes: Only system-reserved is enforced (kube-reserved is disabled)

Why this choice?

Static pods architecture: Control-plane components (apiserver, etcd, etc.) run as static pods outside the kubelet.slice cgroup, so kube-reserved doesn't limit them anyway
Operational safety: Avoiding potential cgroup misconfiguration issues
Industry convention: GKE, EKS, and kubeadm follow this pattern
Minimal impact: The kubelet process itself uses only ~200-500MB, so enforcing limits provides little value

Is this a Kubernetes requirement?
No. Technically, you CAN enforce kube-reserved on control-planes. This is a conservative design choice, not a technical limitation.

See DESIGN_DECISIONS.md for detailed rationale.

Troubleshooting

Symptom	Resolution
`Checksum invalid for yq`	Download the release manually and place it in `/usr/local/bin/yq`. Offline environments often require an internal mirror.
`Reservations >= capacity`	Reduce the density factor or select the minimal profile. The node does not expose enough memory for the requested target.
`kubelet stuck in activating`	Ensure `/etc/kubernetes/bootstrap-kubelet.conf` and `kubelet.conf` exist. Re-run `kubeadm join` if they were removed.
`yq: -i can only be used with -y`	Remove the Python version of `yq`; install mikefarah v4+.

Monitoring and metrics

A full monitoring lab lives in tests/kubelet-alerting-lab/:

Helm deployment of kube-prometheus-stack.
Recording rules for reserving metrics (kubelet_system_reserved_memory, etc.).
Grafana dashboards and recommended alerts.

Launch it from the repo root:

cd tests/kubelet-alerting-lab
./deploy.sh   # see README for details

Security and best practices

Always keep at least one recent backup per node.
Run the script through a maintenance controller or an automation tool to avoid concurrent executions (the script uses flock for safety but planning ahead helps).
Track changes in Git: copy /var/lib/kubelet/config.yaml into your CMDB or repo after each rollout.
Combine with admission policies that cap pod density per node pool.

Additional resources

Contribution

Pull requests are welcome! Please:

Open an issue describing the bug or feature.
Run the Vagrant lab or the automated tests under tests/ when adding logic.
Keep shellcheck clean (./tests/quick_tests.sh).

Changelog and release notes

See CHANGELOG_v3.1.2.md for the latest stable release. Historical changelog files live under the changelog/ directory.

License

MIT License – see LICENSE.

Support

GitHub Issues: https://github.com/MacFlurry/reserved-sys-kube/issues

Credits

Developed and maintained by the Platform Engineering Team. Inspired by the sizing guidance from Google, Amazon, Red Hat, and the Kubernetes SIG Scalability group.

Name		Name	Last commit message	Last commit date
Latest commit History 90 Commits
ansible		ansible
changelog		changelog
daemonset		daemonset
systemd		systemd
tests		tests
.gitignore		.gitignore
CHANGELOG_v3.1.2.md		CHANGELOG_v3.1.2.md
DESIGN_DECISIONS.md		DESIGN_DECISIONS.md
README.md		README.md
kubelet_auto_config.sh		kubelet_auto_config.sh
remove-kubelet-auto-config.sh		remove-kubelet-auto-config.sh
rollback-kubelet-config.sh		rollback-kubelet-config.sh

Folders and files

Latest commit

History

Repository files navigation