Skip to content

Consider removing userdata (startup) script from VM when the machine has been bootstrapped successfully #990

Open
Open
Consider removing userdata (startup) script from VM when the machine has been bootstrapped successfully#990
Labels
area/robustnessRobustness, reliability, resilience relatedkind/enhancementEnhancement, improvement, extensionlifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.priority/3Priority (lower number equals higher priority)

Description

@oliver-goetz
oliver-goetz
opened on Apr 28, 2025

How to categorize this issue?

/area robustness
/kind enhancement
/priority 3

What would you like to be added:
The userdata script added to VMs by MCM usually contains the operating system config of type provision and some OS extensions specific enhancements. This content is meant to be used once when the VM is bootstrapped.
Thus, removing the script from the VM when the machine has been bootstrapped successfully avoids issues which might occur when the script runs multiple times.

Why is this needed:
The userdata script turned out to be a startup script at many cloud providers which runs every time the VM is booting. Since the script is only created when the VM is bootstrapped its content is outdated pretty soon.
When the VM restarts after a while the userdata script might overwrite parts of the config with outdated data.

While we created certain measurements to avoid such issues (see gardener/gardener#11208) there is still a risk of corrupted configs, e.g. VMs created before the mentioned fix has been applied are still writing outdated config data to the disk when they are rebooted.

At the moment we could still get rid of those machines by rolling them. However, this won't be possible for machines where in-place updates are enabled.

Activity

  1. oliver-goetz
    added
    kind/enhancementEnhancement, improvement, extension
    on Apr 28, 2025
  2. ghost added
    area/robustnessRobustness, reliability, resilience related
    on Apr 28, 2025
  3. ghost added
    priority/3Priority (lower number equals higher priority)
    on Apr 28, 2025

  4. unmarshall commented on May 21, 2025

    @unmarshall
    unmarshall
    on May 21, 2025
    Member

    Hello @oliver-goetz,

    We discussed this within the MCM team and we have a few questions.

    1. User data script is meant to be used once when initializing the VM to make it into a k8s Node. The script therefore should be idempotent to restarts of the VM. Do you see an issue in making the script idempotent? I see that you have already raised Create a wrapper for OperatingSystemConfig provisioning bash script to ensure that it is executed only once gardener#11208 to ensure that there are no side effects if the script is invoked again. Is that not sufficient?
    2. MCM does not know anything about what is contained in User data (see mcm-aws-provider). When creating a VM it sets the UserData field of RunInstancesInput (in case of aws provider). Similarly relevant field is set for other providers as well. Ideally from its current responsibility set it should not know about what is contained in UserData and should also not manage the lifecycle of user data. Can you clarify the reasons that you think that MCM should now remove this script upon successful bootstrapping?

    From our perspective the responsibility for identifying that bootstrapping has been successfully completed, followed by subsequent removal of the init script should be the responsibility of GNA or any other component running on the VM which keeps tap on the bootstrapping process.

    If you wish to discuss this further, then we are happy to schedule a meeting to learn more from you in case we have gaps in our understanding.

  5. oliver-goetz commented on Sep 24, 2025

    @oliver-goetz
    oliver-goetz
    on Sep 24, 2025
    MemberAuthor

    I lost track of this issue 😅

    Yes, there is gardener/gardener#11208 which prevents the provisioning OSC from being executed multiple times. While this fixes most of the issues we had, there are still a couple of drawbacks.
    Most importantly, the provisioning OSC is envisioned to be applied only when the node is bootstrapped. It should not be applied later, because this would tamper the configuration of the node. Afaik we were not fully aware that the userdata script runs every time the node is booting (this might be even different in the different landscapes) until we were facing some issues with gardener-node-agent .

    These are the drawback I still see.

    • The wrapper is a utility function only, so OS extensions have to include it into their implementation.
    • There could be more bugs in the bootstrap script in the future (introduced in Gardener and its OS extensions)
    • VMs could life for a very long time (the fix has been implemented in January 2025 and we still find nodes which run an older version of the userdata script). When there are more nodes using in-place updates the number of long living nodes and their lifetime will increase further.

    So why should this be implemented in MCM?

    • MCM is the only component which is touching VMs. GNA does not have access to the hyperscaler API to update its underlying VM and we do not plan to grant such permissions to it.
    • The userdata script is somehow injected into the startup process of a VM, so GNA cannot disable it on OS level.

    Let's discuss this in a meeting 🙂

  6. gardener-ci-robot commented on Dec 23, 2025

    @gardener-ci-robot
    gardener-ci-robot
    on Dec 23, 2025
    Contributor

    The Gardener project currently lacks enough active contributors to adequately respond to all issues.
    This bot triages issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as fresh with /remove-lifecycle stale
    • Mark this issue as rotten with /lifecycle rotten
    • Close this issue with /close

    /lifecycle stale

  7. ghost added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Dec 23, 2025
  8. aaronfern
    removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jan 6, 2026

  9. gardener-ci-robot commented on Apr 6, 2026

    @gardener-ci-robot
    gardener-ci-robot
    on Apr 6, 2026
    Contributor

    The Gardener project currently lacks enough active contributors to adequately respond to all issues.
    This bot triages issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as active with /lifecycle active
    • Mark this issue as frozen with /lifecycle frozen
    • Mark this issue as fresh with /remove-lifecycle stale
    • Mark this issue as rotten with /lifecycle rotten
    • Close this issue with /close

    /lifecycle stale

  10. gardener-prow
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Apr 6, 2026
  11. aaronfern
    removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on May 12, 2026

  12. gardener-ci-robot commented on Oct 18, 2026

    @gardener-ci-robot
    gardener-ci-robot
    on Oct 18, 2026
    Contributor

    The Gardener project currently lacks enough active contributors to adequately respond to all issues.
    This bot triages issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as active with /lifecycle active
    • Mark this issue as frozen with /lifecycle frozen
    • Mark this issue as fresh with /remove-lifecycle stale
    • Mark this issue as rotten with /lifecycle rotten
    • Close this issue with /close

    /lifecycle stale

  13. gardener-prow
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/robustnessRobustness, reliability, resilience relatedkind/enhancementEnhancement, improvement, extensionlifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.priority/3Priority (lower number equals higher priority)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions