Skip to content

add options to tune concurrency, qps and burst #287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ajaysundark wants to merge 2 commits into
base: main
Choose a base branch
from
+38 −12
Open

add options to tune concurrency, qps and burst #287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48f3225
add options to tune concurrency, qps and burst
ajaysundark Jul 3, 2026
ba22b7f
update spacing
ajaysundark Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 27 additions & 7 deletions cmd/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ func main() {
var metricsCertDir string
var leaderElectionNamespace string
var enableNodeStateMetrics bool
var kubeAPIQPS float64

@Karthik-K-N Karthik-K-N Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that these vars are increasing, I think we can keep them in vars() block outside main for better code organization?

var kubeAPIBurst int
var nodeConcurrentReconciles int
var ruleConcurrentReconciles int

flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
Expand All @@ -84,6 +88,16 @@ func main() {
flag.StringVar(&leaderElectionNamespace, "leader-election-namespace", "", "The namespace where the leader election resource will be created.")
flag.BoolVar(&enableNodeStateMetrics, "enable-node-state-metrics", false,
"Enable aggregate node state metrics on node updates)")
flag.Float64Var(&kubeAPIQPS, "kube-api-qps", 20,
"Maximum queries per second to the API server from this client. "+
"Raise together with --kube-api-burst on large clusters.")
flag.IntVar(&kubeAPIBurst, "kube-api-burst", 30,
"Maximum burst for throttle between requests to the API server.")

@Karthik-K-N Karthik-K-N Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this makes it simpler "maximum number of queries that should be allowed in one burst"

@Karthik-K-N Karthik-K-N Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any specific reason for keeping 20 and 30 or is it just to get started.

@ajaysundark ajaysundark Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the current defaults set in controller runtime. I think we could adjust this after our experiments. Maybe I could add a comment to clarify these random numbers here

@Karthik-K-N Karthik-K-N Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, That would help, I looked at the default and saw its 5 and 10 https://github.com/kubernetes/client-go/blob/f16383b964b3519812bac4daf8f48fc5a529ae0f/rest/config.go#L118-L127, I am I looking at wrong place?

@ajaysundark ajaysundark Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm.. I referred kcm default config here: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager.

looks like the controller-runtime had these hardcoded as defaults until recently: kubernetes-sigs/controller-runtime@ab40409

@vitorfloriano vitorfloriano Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the magic numbers, could we have them as constants?

flag.IntVar(&nodeConcurrentReconciles, "node-concurrent-reconciles", 1,
"Maximum number of Node objects reconciled concurrently. "+
"Raise on large clusters to reduce readiness-taint latency during node join/condition updates.")
flag.IntVar(&ruleConcurrentReconciles, "rule-concurrent-reconciles", 1,
"Maximum number of NodeReadinessRule objects reconciled concurrently.")

opts := zap.Options{
Development: true,
Expand All @@ -107,7 +121,11 @@ func main() {
}(),
}

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
restConfig := ctrl.GetConfigOrDie()
restConfig.QPS = float32(kubeAPIQPS)

@Karthik-K-N Karthik-K-N Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't we declare it directly as float32?

restConfig.Burst = kubeAPIBurst

mgr, err := ctrl.NewManager(restConfig, ctrl.Options{
Scheme: scheme,
Metrics: metricsServerOptions,
HealthProbeBindAddress: probeAddr,
Expand All @@ -132,15 +150,17 @@ func main() {

// Create reconcilers linked to the main controller
ruleReconciler := &controller.RuleReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Controller: readinessController,
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Controller: readinessController,
MaxConcurrentReconciles: ruleConcurrentReconciles,
}

nodeReconciler := &controller.NodeReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Controller: readinessController,
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Controller: readinessController,
MaxConcurrentReconciles: nodeConcurrentReconciles,
}

// Setup controllers with manager
Expand Down
8 changes: 6 additions & 2 deletions internal/controller/node_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/builder"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/predicate"

Expand All @@ -38,14 +39,17 @@ import (
// NodeReconciler reconciles a Node object.
type NodeReconciler struct {
client.Client
Scheme *runtime.Scheme
Controller *RuleReadinessController
Scheme *runtime.Scheme
Controller *RuleReadinessController
MaxConcurrentReconciles int // caps how many nodes are reconciled concurrently
}

// SetupWithManager sets up the controller with the Manager.
func (r *NodeReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager) error {
concurrency := max(r.MaxConcurrentReconciles, 1)
return ctrl.NewControllerManagedBy(mgr).
Named("node").
WithOptions(controller.Options{MaxConcurrentReconciles: concurrency}).
For(&corev1.Node{}, builder.WithPredicates(predicate.Funcs{
CreateFunc: func(e event.CreateEvent) bool {
log := ctrl.LoggerFrom(ctx)
Expand Down
8 changes: 5 additions & 3 deletions internal/controller/nodereadinessrule_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ type RuleReadinessController struct {
// RuleReconciler handles NodeReadinessRule reconciliation.
type RuleReconciler struct {
client.Client
Scheme *runtime.Scheme
Controller *RuleReadinessController
Scheme *runtime.Scheme
Controller *RuleReadinessController
MaxConcurrentReconciles int // caps how many rules are reconciled concurrently
}

// NewRuleReadinessController creates a new controller.
Expand All @@ -82,9 +83,10 @@ func NewRuleReadinessController(mgr ctrl.Manager, clientset kubernetes.Interface

// SetupWithManager sets up the controller with the Manager.
func (r *RuleReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager) error {
concurrency := max(r.MaxConcurrentReconciles, 1)
return ctrl.NewControllerManagedBy(mgr).
Named("nodereadiness-controller").
WithOptions(controller.Options{MaxConcurrentReconciles: 1}).
WithOptions(controller.Options{MaxConcurrentReconciles: concurrency}).
For(&readinessv1alpha1.NodeReadinessRule{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
Complete(r)
}
Expand Down
Loading