dfmicro

Run MicroShift clusters inside rootful Podman containers. Each cluster gets its own network, loop-device backed LVM storage, and a kubeconfig. Workers can be added when multi-node operation is needed.

Verified on Linux (Fedora / RHEL). Best-effort support on macOS via rootful Podman machine.

Installation

Download the latest release for your platform from the releases page, extract, and place the binary on your PATH.

tar -xzf dfmicro_linux_amd64.tar.gz
install -m 0755 dfmicro ~/.local/bin/dfmicro

Or use the installer:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh | sh

Set VERSION to install a specific release, or INSTALL_DIR to change the destination directory:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh |
  VERSION=0.0.2 INSTALL_DIR="$HOME/bin" sh

List available releases without installing:

curl -fsSL https://raw.githubusercontent.com/leelavg/dfmicro/main/install.sh | sh -s -- --list

Or build from source:

git clone -b main https://github.com/leelavg/dfmicro
cd dfmicro
make build

Quick start

dfmicro ops sudoers create          # one-time: passwordless sudo for cluster tools on linux only
dfmicro cluster create              # create cluster with default name
dfmicro cluster kubeconfig          # outputs the cluster kubeconfig to stdout
kubectl get nodes                   # save above config and run cluster operations

dfmicro docs                        # for all the cli commands and args
dfmicro docs --examples             # samples for specific scenarios

Testing

Run the suite on a clean Linux host with rootful Podman and sudoers created against the built binary:

DFMICRO_BIN=./bin/dfmicro perl tests.pl --upto 3

See all test options with:

./tests.pl -h

Usage: perl tests.pl [options]

  --upto N             Run through level N (0-3) (default: 0)
  --etcd              Create clusters with etcd
  --list [MODE]        List "tests" or "cmds" (default: tests)
  --keep              Leave resources for inspection
  --pause             Wait for Enter before cleanup
  --fail-fast         Stop after the first failed check
  --cleanup           Remove suite resources without testing
  --timeout DURATION  Limit total runtime, for example 15m
  --help              Show this help

Test output and executed commands are written to /tmp/dfmicro-test/tests.log and /tmp/dfmicro-test/cmds.log.

Command reference

See internal/docs/cli.md for the full command reference, or run dfmicro docs to print it.

For development setup and design notes, see dev.md. For what is planned and what is done, see the devlog.

FAQ

How do I access cluster routes from my host?

IPs and Domain names mentioned are for default values, edit as per your cluster settings.

Routes use a cluster-specific base domain, such as metrics.apps.demo.dfmicro.io, and need DNS resolution to the cluster node IP. Find your cluster's node IP:

kubectl get nodes -o wide | grep 172.20

Choose your setup method (replace 172.20.0.11 with your actual node IP):

Linux with NetworkManager + dnsmasq:

echo 'address=/apps.demo.dfmicro.io/172.20.0.11' | sudo tee /etc/NetworkManager/dnsmasq.d/dfmicro.conf

sudo nmcli general reload dns-full
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux with standalone dnsmasq:

echo 'address=/apps.demo.dfmicro.io/172.20.0.11' | sudo tee /etc/dnsmasq.d/dfmicro.conf

sudo systemctl reload dnsmasq
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux with systemd-resolved:

echo 'DNS=172.20.0.11' | sudo tee /etc/systemd/resolved.conf.d/dfmicro.conf
echo 'Domains=~apps.demo.dfmicro.io' | sudo tee -a /etc/systemd/resolved.conf.d/dfmicro.conf
sudo systemctl restart systemd-resolved
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Linux / macOS:

echo '172.20.0.11 metrics.apps.demo.dfmicro.io' | sudo tee -a /etc/hosts
curl -k https://metrics.apps.demo.dfmicro.io/metrics

Or use curl --resolve (no /etc/hosts edit):

curl -k --resolve metrics.apps.demo.dfmicro.io:443:172.20.0.11 https://metrics.apps.demo.dfmicro.io/metrics

Cleanup:

sudo rm -f /etc/NetworkManager/dnsmasq.d/dfmicro.conf /etc/dnsmasq.d/dfmicro.conf
sudo nmcli general reload dns-full || sudo systemctl reload dnsmasq

Why dnsmasq instead of in-cluster DNS?

Podman's aardvark-dns only resolves container names, not in-cluster routes. It has no upstream config to forward to CoreDNS. The router on node IP `172.20.0.11` handles SNI dispatch, so dnsmasq just needs to answer `*.apps.demo.dfmicro.io` with that IP.

System reboots with "watchdog did not stop!" in journal

Symptom: Machine reboots hard during cluster creation, especially with multiple clusters. Journal shows:

kernel: watchdog: watchdog0: watchdog did not stop!
kernel: watchdog0: pretimeout event

Cause: Cluster startup causes heavy disk I/O. Real-time antivirus/EDR scanning processes (e.g., wdavdaemon from Microsoft Defender) scan this I/O, causing kernel stalls. On encrypted drives, the encryption/decryption overhead compounds the scanning load. Hardware watchdog (iTCO_wdt, 30-second heartbeat) assumes OS is frozen and forces hard reboot.

Fix: Exclude container storage and microshift processes from real-time scanning:

# Exclude container storage (all clusters)
sudo mdatp exclusion folder add --path "/var/lib/containers"

# Exclude cluster LVM backing files (optional, for extra safety)
dfmicro cluster config --name cluster-name | grep lvm-disk
# Example: lvm-disk: /home/user/.config/dfmicro/cluster-name/cluster-name.img
sudo mdatp exclusion folder add --path "/home/user/.config/dfmicro"

# Exclude microshift processes
sudo mdatp exclusion process add --name microshift
sudo mdatp exclusion process add --name microshift-etcd

# Restart mdatp
sudo systemctl restart mdatp

If using different scanning software (ClamAV, Sophos, etc.), add equivalent exclusions for /var/lib/containers and the cluster config directory.

dfmicro also mounts etcd to tmpfs to minimize disk I/O during startup.

Last resort: If exclusions don't work and reboots persist, blacklist the hardware watchdog entirely:

echo "blacklist iTCO_wdt" | sudo tee /etc/modprobe.d/disable-watchdog.conf
echo "blacklist intel_oc_wdt" | sudo tee -a /etc/modprobe.d/disable-watchdog.conf
sudo modprobe -r iTCO_wdt
sudo modprobe -r intel_oc_wdt

Disables hardware watchdog completely. wdavdaemon falls back to software watchdog. Safe for development.

Why not fixed in dfmicro: Real-time scanning is a host-level security feature. The exclusion must be configured on the host.

Contributing

Bug reports and suggestions are welcome via issues. This is a personal project with a focused scope and occasional history rewrites for now, so pull requests are not being accepted at this time.

Acknowledgements

None of this would exist without the incredible work at MicroShift.

Thanks to Anika and Tara for sourcing df shims and carrying out the proof of concept under guidance.

Thanks to all the folks whose shared knowledge is archived in OpenAI Luna, Claude Sonnet/Haiku and IBM Bob agents which made the timeline shorter.

License

Apache 2.0. See LICENSE.