Skip to content

Commit e8bd0cd

Browse files
sudharshanibmsudharshanibm
committed
Add terraform for provisioning s390x build cluster on ibmcloud
Signed-off-by: Sudharshan Muralidharan <[email protected]>
1 parent 1225e4a commit e8bd0cd

File tree

23 files changed

+1402
-0
lines changed

23 files changed

+1402
-0
lines changed

‎infra/ibmcloud/terraform/k8s-s390x-build-cluster/README.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# _TF: IBM K8s s390x Build Cluster_
2+
These terraform resources define a IBM Cloud project containing a s390xVS cluster intended to serve as a "build cluster" for prow.k8s.io.
3+
4+
---
5+
## Initial Setup
6+
7+
### Supporting infrastructure
8+
9+
#### Deploy k8s-infra-setup resources
10+
11+
- this covers things like Resource Group, s390x Virtual Server Instances, Virtual Private Cloud, IBM Cloud Secret Manager Secrets, etc.
12+
- Once the deployment successfully completes, the `secrets_manager_id` will be generated and should be used in the subsequent steps.
13+
14+
---
15+
#### Deploy k8s-s390x-build-cluster resources
16+
17+
**1. Navigate to the correct directory**
18+
<br> You need to be in the `k8s-s390x-build-cluster` directory to run the automation.
19+
20+
**2. Export COS Secrets**
21+
<br> Export `access_key` and `secret_key` as environment variables.
22+
```
23+
export AWS_ACCESS_KEY_ID="<HMAC_ACCESS_KEY_ID>"
24+
export AWS_SECRET_ACCESS_KEY="<HMAC_SECRET_ACCESS_KEY>"
25+
```
26+
27+
**3. Initialize Terraform**
28+
<br> Execute the following command to initialize Terraform in your project directory. This command will download the necessary provider plugins and prepare the working environment.
29+
```
30+
terraform init -reconfigure
31+
```
32+
33+
**4. Check the `variables.tf` file**
34+
<br> Open the `variables.tf` file to review all the available variables. This file lists all customizable inputs for your Terraform configuration.
35+
36+
`ibmcloud_api_key`, `secrets_manager_id` are the only required variables that you must set in order to proceed. You can set this key either by adding it to your `var.tfvars` file or by exporting it as an environment variable.
37+
38+
**Option 1:** Set in `var.tfvars` file
39+
Create `var.tfvars` file and set the following variables in `var.tfvars` file:
40+
```
41+
ibmcloud_api_key = "<YOUR_API_KEY>"
42+
secrets_manager_id = "<SECRETS_MANAGER_ID>"
43+
```
44+
Tip: To get the secrets_manager_id (GUID) for IBM Cloud Secrets Manager instance:
45+
```
46+
ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
47+
jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid'
48+
```
49+
**Option 2:** Export as an environment variable
50+
Alternatively, you can export above as an environment variable before running Terraform:
51+
```
52+
export TF_VAR_ibmcloud_api_key="<YOUR_API_KEY>"
53+
export TF_VAR_secrets_manager_id=$(ibmcloud resource service-instances --service-name secrets-manager --output JSON | \
54+
jq -r '.[] | select(.name | contains("k8s-s390x")) | .guid')
55+
```
56+
57+
**5. Run Terraform Apply**
58+
<br> After setting the necessary variables (particularly the API_KEY), execute the following command to apply the Terraform configuration and provision the infrastructure:
59+
```
60+
terraform apply -var-file var.tfvars
61+
```
62+
Terraform will display a plan of the actions it will take, and you'll be prompted to confirm the execution. Type `yes` to proceed.
63+
64+
**6. Get Output Information**
65+
<br> Once the infrastructure has been provisioned, use the terraform output command to list details about the provisioned resources.
66+
```
67+
terraform output
68+
```
69+
70+
**7. Set up the Kubernetes cluster using ansible**
71+
Clone the repository `https://github.com/kubernetes-sigs/provider-ibmcloud-test-infra` and change the directory to `kubetest2-tf/data/k8s-ansible`:
72+
```
73+
cd kubetest2-tf/data/k8s-ansible
74+
```
75+
76+
**8. Install ansible on the deployer VM**
77+
```
78+
dnf install ansible -y
79+
```
80+
81+
**9. Update the fields under `group_vars/all` to include the Kubernetes version to install**
82+
<br> The following lines will update the version to the latest stable release of Kubernetes. You can modify it accordingly to set up the CI (alpha) version.
83+
```
84+
K8S_VERSION=$(curl -Ls https://dl.k8s.io/release/stable.txt)
85+
LOADBALANCER_EP=<mention the loadbalancer endpoint obtained from terraform output>
86+
sed -i \
87+
-e "s/^directory: .*/directory: release/" \
88+
-e "s/build_version: .*/build_version: $K8S_VERSION/" \
89+
-e "s/release_marker: .*/release_marker: $K8S_VERSION/" \
90+
-e "s/loadbalancer: .*/loadbalancer: $LOADBALANCER_EP/" group_vars/all
91+
```
92+
93+
**10. Update the fields under `examples/k8s-build-cluster/hosts.yml` to contain IP addresses of the VMs to set up Kubernetes**
94+
```
95+
For example:
96+
97+
[bastion]
98+
56.77.34.6
99+
100+
[masters]
101+
192.168.100.3
102+
192.168.100.4
103+
104+
[workers]
105+
192.168.100.5
106+
192.168.100.6
107+
192.168.100.7
108+
109+
[workers:vars]
110+
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
111+
112+
[masters:vars]
113+
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -i <path/to/private-key> -q [email protected]" -i <path/to/private-key>'
114+
```
115+
116+
**11. Update the fields under `group_vars/bastion_configuration` to contain the information of the private network.**
117+
```
118+
For example:
119+
120+
bastion_private_gateway: 192.168.100.1
121+
bastion_private_ip: 192.168.100.2
122+
```
123+
124+
**12. Trigger the installation using ansible**
125+
```
126+
ansible-playbook -v -i examples/k8s-build-cluster/hosts.yml install-k8s-ha.yaml -e @group_vars/bastion_configuration --extra-vars @group_vars/all
127+
```

‎infra/ibmcloud/terraform/k8s-s390x-build-cluster/bastion.tf‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
locals {
18+
bastion_nodes = {
19+
"primary" = {
20+
profile = var.bastion_profile
21+
boot_volume = {
22+
size = var.bastion_boot_volume_size
23+
}
24+
}
25+
}
26+
}
27+
28+
resource "ibm_is_instance" "bastion" {
29+
for_each = local.bastion_nodes
30+
name = "bastion-s390x-${each.key}"
31+
vpc = data.ibm_is_vpc.vpc.id
32+
zone = var.zone
33+
profile = each.value.profile
34+
image = data.ibm_is_image.os_image.id
35+
keys = [ibm_is_ssh_key.k8s_ssh_key.id]
36+
resource_group = data.ibm_resource_group.resource_group.id
37+
38+
primary_network_interface {
39+
name = "public-nic-${each.key}"
40+
subnet = data.ibm_is_subnet.subnet.id
41+
security_groups = [data.ibm_is_security_group.bastion.id]
42+
}
43+
44+
boot_volume {
45+
name = "boot-vol-bastion-${each.key}"
46+
size = each.value.boot_volume.size
47+
}
48+
49+
user_data = <<-EOF
50+
#cloud-config
51+
package_update: true
52+
package_upgrade: true
53+
packages:
54+
- tcpdump
55+
- net-tools
56+
- iptables-persistent
57+
write_files:
58+
- path: /etc/ssh/sshd_config.d/99-bastion.conf
59+
content: |
60+
AllowTcpForwarding yes
61+
GatewayPorts yes
62+
PermitTunnel yes
63+
PermitRootLogin prohibit-password
64+
PasswordAuthentication no
65+
ClientAliveInterval 120
66+
ClientAliveCountMax 3
67+
MaxSessions 50
68+
MaxStartups 50:30:100
69+
- path: /etc/systemd/network/10-eth1.network
70+
content: |
71+
[Match]
72+
Name=eth1
73+
[Network]
74+
Address=${data.ibm_is_subnet.subnet.ipv4_cidr_block}
75+
DNS=8.8.8.8
76+
DNS=8.8.4.4
77+
runcmd:
78+
- [sysctl, -w, net.ipv4.ip_forward=1]
79+
- [echo, "net.ipv4.ip_forward = 1", ">>", /etc/sysctl.conf]
80+
- [iptables, -t, nat, -A, POSTROUTING, -o, eth0, -j, MASQUERADE]
81+
- [iptables, -A, FORWARD, -i, eth1, -o, eth0, -j, ACCEPT]
82+
- [iptables, -A, FORWARD, -i, eth0, -o, eth1, -m, state, --state, RELATED,ESTABLISHED, -j, ACCEPT]
83+
- [netfilter-persistent, save]
84+
- [systemctl, restart, systemd-networkd]
85+
- [systemctl, restart, sshd]
86+
- [hostnamectl, set-hostname, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com"]
87+
- [echo, "bastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com", ">", /etc/hostname]
88+
- [sed, -i, "s/^127.0.1.1.*/127.0.1.1\tbastion-s390x-${each.key}.s390x-vpc.cloud.ibm.com/", /etc/hosts]
89+
- [touch, /var/lib/cloud/instance/bastion-setup-success]
90+
EOF
91+
}
92+
93+
resource "ibm_is_floating_ip" "bastion_fip" {
94+
for_each = ibm_is_instance.bastion
95+
name = "bastion-fip-${each.key}"
96+
target = each.value.primary_network_interface[0].id
97+
resource_group = data.ibm_resource_group.resource_group.id
98+
}

‎infra/ibmcloud/terraform/k8s-s390x-build-cluster/data.tf‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
data "ibm_resource_group" "resource_group" {
17+
name = "rg-build-cluster"
18+
}
19+
20+
data "ibm_is_vpc" "vpc" {
21+
name = "k8s-s390x-vpc"
22+
}
23+
24+
data "ibm_is_subnet" "subnet" {
25+
name = "k8s-s390x-subnet"
26+
}
27+
28+
data "ibm_is_security_group" "bastion" {
29+
name = "k8s-vpc-s390x-bastion-sg"
30+
vpc = data.ibm_is_vpc.vpc.id
31+
}
32+
33+
data "ibm_is_security_group" "control_plane_sg" {
34+
name = "k8s-vpc-s390x-control-plane-sg"
35+
vpc = data.ibm_is_vpc.vpc.id
36+
}
37+
38+
data "ibm_is_security_group" "worker_sg" {
39+
name = "k8s-vpc-s390x-worker-sg"
40+
vpc = data.ibm_is_vpc.vpc.id
41+
}
42+
43+
data "ibm_sm_arbitrary_secret" "ssh_private_key" {
44+
instance_id = var.secrets_manager_id
45+
region = var.region
46+
name = "zvsi-ssh-private-key"
47+
secret_group_name = "default"
48+
}
49+
50+
data "ibm_sm_arbitrary_secret" "ssh_public_key" {
51+
instance_id = var.secrets_manager_id
52+
region = var.region
53+
name = "zvsi-ssh-public-key"
54+
secret_group_name = "default"
55+
}
56+
data "ibm_is_image" "os_image" {
57+
name = var.image_name
58+
}

‎infra/ibmcloud/terraform/k8s-s390x-build-cluster/load_balancer.tf‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
resource "ibm_is_lb" "public" {
17+
name = "k8s-s390x-ci"
18+
type = "public"
19+
subnets = [data.ibm_is_subnet.subnet.id]
20+
resource_group = data.ibm_resource_group.resource_group.id
21+
security_groups = [data.ibm_is_security_group.control_plane_sg.id]
22+
}
23+
24+
resource "ibm_is_lb_pool" "k8s_api_pool" {
25+
name = "k8s-api-server-pool"
26+
lb = ibm_is_lb.public.id
27+
protocol = "tcp"
28+
algorithm = "round_robin"
29+
health_delay = 5
30+
health_retries = 2
31+
health_timeout = 2
32+
health_type = "tcp"
33+
health_monitor_url = "/"
34+
health_monitor_port = var.api_server_port
35+
}
36+
37+
resource "ibm_is_lb_listener" "k8s_api_listener" {
38+
lb = ibm_is_lb.public.id
39+
protocol = "tcp"
40+
port = var.api_server_port
41+
default_pool = ibm_is_lb_pool.k8s_api_pool.pool_id
42+
}
43+
44+
resource "ibm_is_lb_pool_member" "k8s_api_members" {
45+
for_each = ibm_is_instance.control_plane
46+
47+
lb = ibm_is_lb.public.id
48+
pool = ibm_is_lb_pool.k8s_api_pool.pool_id
49+
port = var.api_server_port
50+
target_address = each.value.primary_network_interface[0].primary_ipv4_address
51+
}

0 commit comments

Comments
 (0)