|
| 1 | +--- |
| 2 | +title: "Guide: Node preparation for swap and upgrading to v26" |
| 3 | +description: "Upgrade procedure when upgrading to v26 which has swap enabled by default." |
| 4 | +menu: |
| 5 | + main: |
| 6 | + parent: "installation" |
| 7 | + weight: 69 |
| 8 | +--- |
| 9 | + |
| 10 | +Swap allows for infrequently accessed data to be moved from memory to disk. Enabling swap reduces the memory required to operate Materialize and improves cost efficiency. Upgrades to v26 and later have swap enabled by default. |
| 11 | + |
| 12 | +## Upgrading to v26 with swap requires node preparation |
| 13 | +We've added new labels to the node selectors for clusterd pods to enable smooth upgrades. As a result, if you are running v25.2.12 or earlier, your existing nodes will not match these selectors and won't be selected to run the pods. Before upgrading to v26, you must prepare your nodes by adding the required labels. |
| 14 | + |
| 15 | +## Preparing for the upgrade using terraform |
| 16 | +v0.6.1 of the Materialize terraform modules can handle much of the preparation work for you. If using our terraform modules, please follow the instructions provided in the respective upgrade notes: |
| 17 | +- [AWS](https://github.com/MaterializeInc/terraform-aws-materialize?tab=readme-ov-file#v061) |
| 18 | +- [GCP](https://github.com/MaterializeInc/terraform-google-materialize?tab=readme-ov-file#v061) |
| 19 | +- [Azure](https://github.com/MaterializeInc/terraform-azurerm-materialize?tab=readme-ov-file#v061) |
| 20 | + |
| 21 | +## Preparing for the upgrade without terraform |
| 22 | + |
| 23 | +1. Label existing scratchfs/lgalloc node groups |
| 24 | + |
| 25 | + If using lgalloc on scratchfs volumes, you must add the additional `"materialize.cloud/scratch-fs": "true"` label to your existing node groups and nodes running Materialize workloads. |
| 26 | + |
| 27 | + Adding this label to the node group (or nodepool) configuration will apply the label to newly spawned nodes, but depending on your cloud provider may not apply the label to existing nodes. |
| 28 | + |
| 29 | + If not automatically applied, you may need to use `kubectl label` to apply the change to existing nodes. |
| 30 | + |
| 31 | +1. Modify existing scratchfs/lgalloc disk setup daemonset selector labels |
| 32 | + |
| 33 | + If using our [ephemeral-storage-setup image](https://github.com/MaterializeInc/ephemeral-storage-setup-image/) as a daemonset to configure scratchfs LVM volumes for lgalloc, you must add the additional `"materialize.cloud/scratch-fs": "true"` label to multiple places: |
| 34 | + * `spec.selector.matchLabels` |
| 35 | + * `spec.template.metadata.labels` |
| 36 | + * (if using `nodeAffinity`) `spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms` |
| 37 | + * (if using `nodeSelector`) `spec.template.spec.nodeSelector` |
| 38 | + |
| 39 | + You **must** use at least one of `nodeAffinity` or `nodeSelector`. |
| 40 | + |
| 41 | + It is recommended to rename this daemonset to make it clear that it is only for the legacy scratchfs/lgalloc nodes (for example, change the name `disk-setup` to `disk-setup-scratchfs`). |
| 42 | + |
| 43 | +1. Create a new node group for swap |
| 44 | + |
| 45 | + 1. Create a new node group (or ec2nodeclass and nodepool if using Karpenter in AWS) using an instance type with local NVMe disks. If in GCP, the disks must be in `raw` mode. |
| 46 | + |
| 47 | + 1. Label the node group with `"materialize.cloud/swap": "true"`. |
| 48 | + |
| 49 | + 1. If using AWS Bottlerocket AMIs (highly recommended if running in AWS), set the following in the userdata to configure the disks for swap, and enable swap in the kubelet: |
| 50 | + |
| 51 | + ```toml |
| 52 | + [settings.oci-defaults.resource-limits.max-open-files] |
| 53 | + soft-limit = 1048576 |
| 54 | + hard-limit = 1048576 |
| 55 | + |
| 56 | + [settings.bootstrap-containers.diskstrap] |
| 57 | + source = "docker.io/materialize/ephemeral-storage-setup-image:v0.4.0" |
| 58 | + mode = "once" |
| 59 | + essential = "true" |
| 60 | + # ["swap", "--cloud-provider", "aws", "--bottlerocket-enable-swap"] |
| 61 | + user-data = "WyJzd2FwIiwgIi0tY2xvdWQtcHJvdmlkZXIiLCAiYXdzIiwgIi0tYm90dGxlcm9ja2V0LWVuYWJsZS1zd2FwIl0=" |
| 62 | + |
| 63 | + [kernel.sysctl] |
| 64 | + "vm.swappiness" = "100" |
| 65 | + "vm.min_free_kbytes" = "1048576" |
| 66 | + "vm.watermark_scale_factor" = "100" |
| 67 | + ``` |
| 68 | + |
| 69 | + 1. If not using AWS or not using Bottlerocket AMIs, and your node group supports it (Azure does not as of 2025-11-05), add a startup taint. This taint will be removed after the disk is configured for swap. |
| 70 | + |
| 71 | + ```yaml |
| 72 | + taints: |
| 73 | + - key: startup-taint.cluster-autoscaler.kubernetes.io/disk-unconfigured |
| 74 | + value: "true" |
| 75 | + effect: NoSchedule |
| 76 | + ``` |
| 77 | + |
| 78 | +1. Create a new disk-setup-swap daemonset |
| 79 | + |
| 80 | + If using Bottlerocket AMIs in AWS, you may skip this step, as you should have configured swap using userdata previously. |
| 81 | + |
| 82 | + Create a new daemonset using our [ephemeral-storage-setup image](https://github.com/MaterializeInc/ephemeral-storage-setup-image/) to configure the disks for swap and to enable swap in the kubelet. |
| 83 | + |
| 84 | + The arguments to the init container in this daemonset need to be configured for swap. See the examples in the linked git repository for more details. |
| 85 | + |
| 86 | + This daemonset should run only on the new swap nodes, so we need to ensure it has the `"materialize.cloud/swap": "true"` label in several places: |
| 87 | + |
| 88 | + * `spec.selector.matchLabels` |
| 89 | + * `spec.template.metadata.labels` |
| 90 | + * (if using `nodeAffinity`) `spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms` |
| 91 | + * (if using `nodeSelector`) `spec.template.spec.nodeSelector` |
| 92 | + |
| 93 | + You **must** use at least one of `nodeAffinity` or `nodeSelector`. |
| 94 | + |
| 95 | + It is recommended to name this daemonset to clearly indicate that it is for configuring swap (ie: `disk-setup-swap`), as opposed to other disk configurations. |
| 96 | + |
| 97 | +1. (Optional) Configure environmentd to also use swap |
| 98 | + |
| 99 | + Swap is enabled by default for clusterd, but not for environmentd. If you'd like to enable swap for environmentd, add `"materialize.cloud/swap": "true"` to the `environmentd.node_selector` helm value. |
| 100 | + |
| 101 | +1. Upgrade the Materialize operator helm chart to v26 |
| 102 | + |
| 103 | + The cluster size definitions for existing Materialize instances will not be changed at this point, but any newly created Materialize instances, or upgraded Materialize instances will pick up the new sizes. |
| 104 | + |
| 105 | + Do not create any new Materialize instances at versions less than v26, or perform any rollouts to existing Materialize instances to versions less than v26. |
| 106 | + |
| 107 | +1. Upgrade existing Materialize instances to v26 |
| 108 | + |
| 109 | + The new v26 pods should go to the new swap nodes. |
| 110 | + |
| 111 | + You can verify that swap is enabled and working by `exec`ing into a clusterd pod and running `cat /sys/fs/cgroup/memory.swap.max`. If you get a number greater than 0, swap is enabled and the pod is allowed to use it. |
| 112 | + |
| 113 | +1. (Optional) Delete old scratchfs/lgalloc node groups and disk-setup-scratchfs daemonset |
| 114 | + |
| 115 | + If you no longer have anything running on the old scratchfs/lgalloc nodes, you may delete their node group and the disk-setup-scratchfs daemonset. |
| 116 | + |
| 117 | +## How to disable swap |
| 118 | +If you wish to opt out of swap and retain the old behavior, you may set `operator.clusters.swap_enabled: false` in your helm values. |
0 commit comments