kvm: Convert to internal driver #21625

nirs · 2025-09-23T19:14:21Z

Remove the docker-machine-driver-kvm2 wrapper and use the kvm driver as internal driver. This fixes all the build and test issues related to building and testing the docker-machine-driver-kvm2, and simplify the kvm driver runtime flow.

To avoid dependency on libvirt shared library on Linux, we build now with libvirt_dlopen tag. With this we don't need libvirt devel libraies during build, and in runtime we will fail if libvirt shared library is not installed.

Building minikube for arm64 with libvirt does not work yet. This build was already skipped for docker-machine-driver-kvm2 external driver. Since now the kvm driver is internal, the kvm driver was modified to build only for linux/amd64.

Example run

$ make
go build  -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0 -X k8s.io/minikube/pkg/version.gitCommitID="b25c816e2321bfd20593b48d1f9359079818c20b" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -o out/minikube k8s.io/minikube/cmd/minikube

$ ldd out//minikube 
	linux-vdso.so.1 (0x00007f2808c43000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f2808c1a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2808a28000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2808c45000)

$ rm -f ~/.minikube/bin/docker*

$ out/minikube start -d kvm
😄  minikube v1.37.0 on Fedora 42 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.34.0 on Docker 28.4.0 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ ls ~/.minikube/bin/

Testing

fedora-intergation-kvm.tar.gz

Fixes #21618
Fixes #21551
Fixes #21582
Fixes #21581
Fixes #21580

Based on #21644 for testing.

k8s-ci-robot · 2025-09-23T19:14:24Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-09-23T19:14:28Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign comradeprogrammer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nirs · 2025-09-23T20:43:54Z

/cc @afbjorklund
/cc @medyagh

nirs · 2025-09-23T20:44:29Z

/ok-to-test

Makefile

medyagh · 2025-09-23T21:10:43Z

/ok-to-test

nirs · 2025-09-23T21:55:21Z

Testing upgrade from version using external driver to version using builtin driver

Creating kvm cluster with master:

$ out/minikube start -d kvm
😄  minikube v1.37.0 on Fedora 42 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
💾  Downloading driver docker-machine-driver-kvm2:
    > docker-machine-driver-kvm2-...:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > docker-machine-driver-kvm2-...:  15.20 MiB / 15.20 MiB  100.00% 20.62 MiB
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.34.0 on Docker 28.4.0 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Switch to minikube version using builtin driver:

nsoffer@nsoffer ~/minikube (bootstrapper-path)$ git checkout kvm-driver-internal 
M	hack/benchmark/time-to-k8s/time-to-k8s-repo
Switched to branch 'kvm-driver-internal'
Your branch is up to date with 'origin/kvm-driver-internal'.
nsoffer@nsoffer ~/minikube (kvm-driver-internal)$ make
go build  -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0 -X k8s.io/minikube/pkg/version.gitCommitID="c311677bf8f9c6cb525eeb8b7add2a03a583aae4" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -o out/minikube k8s.io/minikube/cmd/minikube

Checking existing cluster:

$ out/minikube profile list
┌──────────┬────────┬─────────┬────────────────┬─────────┬────────┬───────┬────────────────┬────────────────────┐
│ PROFILE  │ DRIVER │ RUNTIME │       IP       │ VERSION │ STATUS │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├──────────┼────────┼─────────┼────────────────┼─────────┼────────┼───────┼────────────────┼────────────────────┤
│ minikube │ kvm2   │ docker  │ 192.168.39.227 │ v1.34.0 │ OK     │ 1     │ *              │ *                  │
└──────────┴────────┴─────────┴────────────────┴─────────┴────────┴───────┴────────────────┴────────────────────┘

Stopping the cluster

$ out/minikube stop
✋  Stopping node "minikube"  ...
🛑  1 node stopped.

Checking status:

$ out/minikube profile list
┌──────────┬────────┬─────────┬────────────────┬─────────┬─────────┬───────┬────────────────┬────────────────────┐
│ PROFILE  │ DRIVER │ RUNTIME │       IP       │ VERSION │ STATUS  │ NODES │ ACTIVE PROFILE │ ACTIVE KUBECONTEXT │
├──────────┼────────┼─────────┼────────────────┼─────────┼─────────┼───────┼────────────────┼────────────────────┤
│ minikube │ kvm2   │ docker  │ 192.168.39.227 │ v1.34.0 │ Stopped │ 1     │ *              │                    │
└──────────┴────────┴─────────┴────────────────┴─────────┴─────────┴───────┴────────────────┴────────────────────┘

Starting stopped cluster:

$ out/minikube start
😄  minikube v1.37.0 on Fedora 42 (kvm/amd64)
✨  Using the kvm2 driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔄  Restarting existing kvm2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.34.0 on Docker 28.4.0 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

medyagh · 2025-09-23T21:58:24Z

cross build failing https://storage.googleapis.com/minikube-builds/logs/21625/build.txt
you need to delete form hack/jenkins/minikube_cross_build_and_upload.sh
and Makefile

nirs · 2025-09-23T22:04:23Z

cross build failing https://storage.googleapis.com/minikube-builds/logs/21625/build.txt you need to delete form hack/jenkins/minikube_cross_build_and_upload.sh and Makefile

Should be fixed by https://github.com/kubernetes/minikube/compare/c311677bf8f9c6cb525eeb8b7add2a03a583aae4..32d5596fa0884e36b3f6d22bac7ae1c120219447

Makefile

nirs · 2025-09-27T13:24:50Z

Converting to draft for do some cleanups.

nirs · 2025-09-27T17:02:56Z

Trying to close and reopen to recover the stuck Jenkins Cross Build ...

nirs · 2025-09-27T19:59:26Z

/retest

k8s-ci-robot · 2025-09-27T20:00:05Z

@nirs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-minikube-integration	`1f56101`	link	true	`/test pull-minikube-integration`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

nirs · 2025-09-27T20:16:57Z

/retest-this-please

nirs · 2025-09-28T21:54:38Z

/retest-this-please

The DOCKER command was missing :Z for the ~/go directory, failing with: $ BUILD_IN_DOCKER=y make all ... GOOS="linux" GOARCH="arm64" \ go build -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 ... go: could not create module cache: stat /go/pkg/mod: permission denied make: *** [Makefile:279: out/minikube-linux-arm64] Error 1 make: *** [Makefile:276: out/minikube-linux-arm64] Error 2 With this change we can debug build issues on Fedora or other distros using selinux.

My editor is configured to remove trailing spaces automatically. Adding as a separate commit to minimize the actual change and allow commit this change separately from the kvm driver change.

It checked for HyperV() instead. This seems wrong since auxdriver is used only for hyperkit and kvm.

Add matchDrivers() helper for looking up --driver=name or --vm-driver=name.

Remove the docker-machine-driver-kvm2 wrapper and use the kvm driver as internal driver. To avoid dependency on libvirt shared library on Linux, we build now with libvirt_dlopen tag. With this we don't need libvirt devel libraies during build, and in runtime we will fail if libvirt shared library is not installed. With this change minikube can not be built for linux !amd64 since building libvirt go binding requires CGO, and it does not work by changing GOARCH.

The build fails on linux/arm64 because libvirt.org/go/libvirt is using CGO, and cross compiling CGO requires a C cross compiler. Setting GOARCH=arm64 is not enough. The issue is tracked in kubernetes#19959. Previously we built the kvm driver also on arm64 as part of the docker-machine-driver-kvm2 executable, but the build was skipped on arm64. Now that we build the driver as part of minikube, we cannot skip the entire build. Change the build tag so the libvirt bits are built only on amd64. To make this possible, the generic linux bits needed by the registry moved to pkg/drivers/kvm/driver.go, and a kvm_stub.go is used for unsupported architectures. In the registry Driver.Status(), move the arm64 check to front since there is no point in checking that libvirt is installed correctly if the driver is not supported yet.

minikube-pr-bot · 2025-10-03T18:11:18Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21625 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 43.8s    │ 43.9s                  │
│ enable ingress │ 16.0s    │ 18.2s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 45.4s 44.7s 46.0s 42.2s 40.7s
Times for minikube (PR 21625) start: 44.7s 42.7s 40.8s 48.0s 43.3s

Times for minikube (PR 21625) ingress: 15.7s 15.8s 15.7s 27.9s 15.8s
Times for minikube ingress: 16.9s 15.8s 15.9s 15.8s 15.8s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21625 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 23.1s    │ 22.5s                  │
│ enable ingress │ 11.0s    │ 11.2s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube ingress: 10.6s 10.6s 12.6s 10.6s 10.6s
Times for minikube (PR 21625) ingress: 10.6s 10.6s 10.7s 12.6s 11.6s

Times for minikube (PR 21625) start: 24.7s 22.5s 21.6s 21.7s 22.0s
Times for minikube start: 26.0s 24.2s 21.3s 22.7s 21.4s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21625 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.0s    │ 20.4s                  │
│ enable ingress │ 20.5s    │ 20.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 21625) start: 20.2s 19.8s 19.4s 22.7s 19.7s
Times for minikube start: 19.8s 20.0s 19.9s 20.5s 20.0s

Times for minikube ingress: 21.1s 21.1s 20.1s 20.1s 20.1s
Times for minikube (PR 21625) ingress: 21.1s 21.1s 21.1s 20.1s 20.1s

minikube-pr-bot · 2025-10-03T19:50:46Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Windows (1 failed)	TestErrorSpam/setup(gopogh)	Unknown

Besides the following environments also have failed tests:

Docker_Linux_crio: 56 failed (gopogh)
Docker_Linux_crio_arm64: 38 failed (gopogh)
KVM_Linux_crio: 2 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 23, 2025

k8s-ci-robot requested a review from ComradeProgrammer September 23, 2025 19:14

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 23, 2025

k8s-ci-robot requested a review from prezha September 23, 2025 19:14

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 23, 2025

nirs force-pushed the kvm-driver-internal branch 2 times, most recently from 061cec4 to b25c816 Compare September 23, 2025 20:31

k8s-ci-robot requested review from afbjorklund and medyagh September 23, 2025 20:43

nirs marked this pull request as ready for review September 23, 2025 20:43

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 23, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Sep 23, 2025

afbjorklund reviewed Sep 23, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

nirs force-pushed the kvm-driver-internal branch 2 times, most recently from bff2cfd to c311677 Compare September 23, 2025 21:09

nirs mentioned this pull request Sep 23, 2025

Require valid aux driver version #21594

Closed

nirs force-pushed the kvm-driver-internal branch from c311677 to 32d5596 Compare September 23, 2025 22:02

nirs commented Sep 23, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

nirs force-pushed the kvm-driver-internal branch 4 times, most recently from 74046b4 to 41720cd Compare September 24, 2025 01:04

nirs force-pushed the kvm-driver-internal branch from 03653be to 9a86213 Compare September 27, 2025 13:59

This comment has been minimized.

Sign in to view

nirs force-pushed the kvm-driver-internal branch 2 times, most recently from e67ffec to 1f56101 Compare September 27, 2025 15:09

nirs mentioned this pull request Sep 27, 2025

Prepare for making kvm driver internal #21644

Open

nirs closed this Sep 27, 2025

nirs reopened this Sep 27, 2025

This comment has been minimized.

Sign in to view

afbjorklund mentioned this pull request Sep 28, 2025

Internalize vendor/github.com/docker/machine #21647

Open

This comment has been minimized.

Sign in to view

nirs added 6 commits October 3, 2025 20:19

Remove trailing whitespace

dfbffb6

My editor is configured to remove trailing spaces automatically. Adding as a separate commit to minimize the actual change and allow commit this change separately from the kvm driver change.

test: Fix NeedsAuxDriver() to check for hyperkit

10789b2

It checked for HyperV() instead. This seems wrong since auxdriver is used only for hyperkit and kvm.

test: De-duplicate driver detection

ac416ec

Add matchDrivers() helper for looking up --driver=name or --vm-driver=name.

nirs force-pushed the kvm-driver-internal branch from 1f56101 to f5ddb06 Compare October 3, 2025 17:19

kvm: Convert to internal driver #21625

Are you sure you want to change the base?

kvm: Convert to internal driver #21625

Conversation

nirs commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example run

Testing

Uh oh!

k8s-ci-robot commented Sep 23, 2025

Uh oh!

k8s-ci-robot commented Sep 23, 2025

Uh oh!

nirs commented Sep 23, 2025

Uh oh!

nirs commented Sep 23, 2025

Uh oh!

Uh oh!

medyagh commented Sep 23, 2025

Uh oh!

nirs commented Sep 23, 2025

Testing upgrade from version using external driver to version using builtin driver

Uh oh!

medyagh commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented Sep 23, 2025

Uh oh!

Uh oh!

nirs commented Sep 27, 2025

Uh oh!

This comment has been minimized.

nirs commented Sep 27, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nirs commented Sep 27, 2025

Uh oh!

k8s-ci-robot commented Sep 27, 2025

Uh oh!

nirs commented Sep 27, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

nirs commented Sep 28, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

minikube-pr-bot commented Oct 3, 2025

Uh oh!

minikube-pr-bot commented Oct 3, 2025

Uh oh!

Uh oh!

nirs commented Sep 23, 2025 •

edited

Loading

medyagh commented Sep 23, 2025 •

edited

Loading