CI/CD

Rules grouping support, with tree dragging functionality (#1680) #16

Workflow file for this run

	# ---------------------------------------------------------------------------------------------------------------------
	# This workflow handles all CI/CD related tasks and can be re-used in custom projects
	# (https://github.com/openremote/custom-project) or forks of this repo, it handles:
	#
	# - Running tests on push/release
	# - Distributing openremote/manager images to docker hub (DOCKERHUB_USER and DOCKERHUB_PASSWORD must be set) on push/release
	# - Deploying to hosts on push/release and/or manual trigger and/or when certain openremote/manager docker tags are updated
	#
	# By default this workflow will just run tests on push/release; distribution (pushing of openremote/manager docker image)
	# and deployment behaviour is configured via the .ci_cd/ci_cd.json file (see .ci_cd/README.md for more details).
	#
	# DEPLOYMENTS
	#
	# When a deployment is requested the repo is checked for a 'deployment/Dockerfile' and if that exists then also
	# 'deployment/build.gradle' must exist with an installDist task that prepares the deployment image in the
	# 'deployment/build' directory; if this condition is not met the deployment will fail.
	#
	# Secrets, inputs and environment variables are combined for consumption by the '.ci_cd/deploy.sh' bash script, they are
	# combined in the following priority:
	#
	# - '.ci_cd/env/.env'
	# - '.ci_cd/env/{ENVIRONMENT}.env'
	# - secrets
	# - inputs (from manual trigger using workflow_dispatch)
	#
	# The above variables are output to 'temp/env' file for each requested deployment. If a secret called 'SSH_PASSWORD' is
	# found that is output to 'ssh.env' file (so it is not copied to the host), if a secret called 'SSH_KEY' is
	# found that is output to 'ssh.key' file (so it is not copied to the host). Sensitive credentials should always be stored
	# in github secrets so they are encrypted, do not store within repo files (even private repos as these are not encrypted).
	#
	# SEE https://github.com/openremote/openremote/tree/master/.ci_cd for details of standard variables and handling.
	#
	# MANUAL TRIGGER
	#
	# This workflow can be triggered by push, release, manually or schedule (from any repo other than openremote/openremote)
	# if triggered manually then the following inputs are available and these override any set in env files and/or secrets:
	#
	# Inputs:
	#
	# - ENVIRONMENT - Which environment to deploy (equivalent to deploy/environment in '.ci_cd/ci_cd.json')
	# - MANAGER_TAG - Which manager docker tag to deploy (equivalent to deploy/managerTags in '.ci_cd/ci_cd.json')
	# leave empty to build a manager image from the repo (must be an openremote repo or have
	# an openremote submodule).
	# - CLEAN_INSTALL - Should the .ci_cd/host_init/clean.sh script be run during deployment (warning this will delete
	# assets, rules, etc.)
	# - COMMIT - Which branch/SHA should be checked out for the deployment (defaults to trigger commit)
	# - OR_HOSTNAME - FQDN of host to deploy to (e.g. demo.openremote.app)
	# - SSH_USER - Set/override the SSH user to use for SSH/SCP commands
	# - SSH_PASSWORD - Set/override the SSH password to use for SSH/SCP commands (SSH key should be preferred)
	# - SSH_PORT - Set/override the SSH port to use for SSH/SCP commands
	# - OR_ADMIN_PASSWORD - The admin password to set for clean installs
	# ---------------------------------------------------------------------------------------------------------------------


	name: CI/CD

	on:
	# Push on master excluding tags
	push:
	branches:
	- 'master'
	tags:
	- '[0-9]+.[0-9]+.[0-9]+'

	# PR
	pull_request:

	# Manual trigger
	workflow_dispatch:
	inputs:
	ENVIRONMENT:
	description: 'Environment to use (if any)'
	MANAGER_TAG:
	description: 'Manager docker tag to pull'
	CLEAN_INSTALL:
	description: 'Delete data before starting'
	type: boolean
	COMMIT:
	description: 'Repo branch or commit SHA to checkout'
	OR_HOSTNAME:
	description: 'Host to deploy to (e.g. demo.openremote.app)'
	OR_ADMIN_PASSWORD:
	description: 'Admin password override'

	workflow_call:
	inputs:
	INPUTS:
	type: string
	secrets:
	SECRETS:
	required: false

	jobs:

	build:
	name: CI/CD
	runs-on: ubuntu-latest

	steps:

	- name: Cancel previous runs
	uses: styfle/cancel-workflow-action@85880fa0301c86cca9da44039ee3bb12d3bedbfa # 0.12.1
	with:
	access_token: ${{ github.token }}

	- name: Get inputs and secrets
	id: inputs-and-secrets
	shell: python
	run: \|
	import os
	import json

	# Overlay all inputs and secrets onto this jobs outputs
	callerInputs = os.getenv("CALLER_INPUTS")
	inputs = os.getenv("INPUTS")
	secrets = os.getenv("SECRETS")
	eventName = os.getenv("GITHUB_EVENT_NAME")

	if inputs is not None and inputs != '':
	inputs = json.loads(inputs)
	if secrets is not None and secrets != '':
	secrets = json.loads(secrets)
	if eventName == 'workflow_call' and callerInputs is not None and callerInputs != 'null':
	os.system(f"echo 'Processing caller inputs'")
	inputs = json.loads(callerInputs)

	if inputs is not None and 'INPUTS' in inputs:
	os.system("echo 'Processing inputs from caller'")
	inputs = json.loads(inputs['INPUTS'])
	if 'SECRETS' in secrets:
	os.system("echo 'Processing secrets from caller'")
	secrets = json.loads(secrets['SECRETS'])

	# Iterate over secrets then inputs and assign them as outputs on this step

	if secrets is not None and secrets != 'null':
	for key, value in secrets.items():
	os.system(f"echo 'Outputting secret: {key}'")
	lines = len(value.split("\n"))
	if lines > 1:
	os.system(f"echo '{key}<<EEOOFF\n{value}\nEEOOFF' >> $GITHUB_OUTPUT")
	else:
	os.system(f"echo '{key}={value}' >> $GITHUB_OUTPUT")

	if inputs is not None and inputs != 'null':
	for key, value in inputs.items():
	os.system(f"echo 'Outputting input: {key}'")
	lines = len(value.split("\n"))
	if lines > 1:
	os.system(f"echo '{key}<<EEOOFF\n{value}\nEEOOFF' >> $GITHUB_OUTPUT")
	else:
	os.system(f"echo '{key}={value}' >> $GITHUB_OUTPUT")

	env:
	CALLER_INPUTS: ${{ toJSON(inputs) }}
	SECRETS: ${{ toJSON(secrets) }}
	INPUTS: ${{ toJSON(github.event.inputs) }}

	- name: Public IP
	id: ip-address
	shell: bash
	run: \|
	set -e
	PUBLIC_IPV4=$(curl -sf v4.ident.me 2>/dev/null \|\| true)
	PUBLIC_IPV6=$(curl -sf v6.ident.me 2>/dev/null \|\| true)
	echo "ipv4=$PUBLIC_IPV4" >> $GITHUB_OUTPUT
	echo "ipv6=$PUBLIC_IPV6" >> $GITHUB_OUTPUT

	- name: Checkout
	uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
	with:
	# This will only be available when run by workflow_dispatch otherwise will checkout branch/commit that triggered the workflow
	ref: ${{ steps.inputs-and-secrets.outputs.COMMIT }}
	submodules: recursive
	fetch-depth: 200

	# Check which files have changed to only run appropriate tests and checks
	- name: Backend files changed
	id: backend-files-changed
	if: github.event_name == 'pull_request'
	uses: tj-actions/changed-files@48d8f15b2aaa3d255ca5af3eba4870f807ce6b3c # v45
	with:
	files_ignore: \|
	.ci_cd/**
	deployment/**
	ui/**
	*/.md
	*/.yml
	yarn.lock

	# Check which files have changed to only run appropriate tests and checks
	- name: UI files changed
	id: ui-files-changed
	if: github.event_name == 'pull_request'
	uses: tj-actions/changed-files@48d8f15b2aaa3d255ca5af3eba4870f807ce6b3c # v45
	with:
	files: \|
	ui/**
	files_ignore: \|
	*/.md
	*/.yml

	- name: Set skip backend manager tests
	id: skip-backend-tests
	if: steps.backend-files-changed.outputs.any_modified != 'true'
	run: echo "value=true" >> $GITHUB_OUTPUT

	- name: Set skip UI tests
	id: skip-ui-tests
	if: steps.ui-files-changed.outputs.any_modified != 'true'
	run: echo "value=true" >> $GITHUB_OUTPUT

	- name: Check if main repo
	id: is_main_repo
	run: \|
	if [ -f manager/src/main/java/org/openremote/manager/Main.java ]; then
	echo "value=true" >> $GITHUB_OUTPUT
	else
	if [ -e 'openremote' ]; then
	echo "::error::openremote submodule no longer supported, see custom project template!"
	exit 1
	fi

	echo "value=false" >> $GITHUB_OUTPUT
	echo "repository=$(sed -E 's#(.+)/.+#\1/openremote#' <<< $GITHUB_REPOSITORY)" >> $GITHUB_OUTPUT

	version=$(cat gradle.properties \| grep openremoteVersion \| sed -E 's#.+=(.+)#\1#' \| xargs)

	if [ -z "$version" ]; then
	echo "openremoteVersion must be set in gradle.properties"
	exit 1
	fi

	echo "ref=$version" >> $GITHUB_OUTPUT
	fi

	- name: Checkout main repo
	uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
	if: ${{ steps.is_main_repo.outputs.value == 'false' }}
	with:
	repository: ${{ steps.is_main_repo.outputs.repository }}
	path: openremote
	ref: ${{ steps.is_main_repo.outputs.ref }}
	sparse-checkout: \|
	.ci_cd
	profile
	sparse-checkout-cone-mode: false

	- name: Check deployment build.gradle
	id: check_deployment_gradle
	uses: andstor/file-existence-action@076e0072799f4942c8bc574a82233e1e4d13e9d6 # v3
	with:
	files: "deployment/build.gradle"

	- name: Check deployment dockerfile
	id: check_deployment_dockerfile
	uses: andstor/file-existence-action@076e0072799f4942c8bc574a82233e1e4d13e9d6 # v3
	with:
	files: "deployment/Dockerfile"

	- name: Check ci_cd existence
	id: check_cicd_json
	uses: andstor/file-existence-action@076e0072799f4942c8bc574a82233e1e4d13e9d6 # v3
	with:
	files: ".ci_cd/ci_cd.json"

	- name: Get version details
	id: get-version-details
	run: \|
	set -o pipefail # exit build with error when pipes fail
	./gradlew currentVersion \| tee /tmp/currentVersion
	versionWithQualifier=$(grep "Project version" /tmp/currentVersion \| sed 's#Project version: ##')
	version=$(sed -E 's#-.+##' <<< $versionWithQualifier)
	isRelease=false
	if [ "$version" == "$versionWithQualifier" ]; then
	isRelease=true
	fi
	echo "version=$version" >> $GITHUB_OUTPUT
	echo "version_with_qualifier=$versionWithQualifier" >> $GITHUB_OUTPUT
	echo "is_release=$isRelease" >> $GITHUB_OUTPUT

	- name: Process ci_cd.json file
	if: ${{ steps.check_cicd_json.outputs.files_exists == 'true' && (github.ref_type == 'tag' \|\| (github.event_name != 'workflow_dispatch' && github.event_name != 'pull_request')) }}
	id: ci-cd-output
	shell: python
	run: \|
	import json
	import os

	eventName = os.getenv('GITHUB_EVENT_NAME')
	refName = os.getenv('GITHUB_REF_NAME')
	refType = os.getenv('GITHUB_REF_TYPE')
	isMainRepo = os.getenv('IS_MAIN_REPO') == 'true'
	isRelease = os.getenv('IS_RELEASE') == 'true'
	managerVersion = os.getenv('MANAGER_VERSION')
	version = os.getenv('VERSION')
	deploys = None
	dockerPublishTags = None
	mavenPublishTag = None
	npmPublishTag = None
	deployEnvironment = None
	eventConfig = ()

	f = open(".ci_cd/ci_cd.json")
	data = json.load(f)
	f.close()

	if isRelease:
	eventName = "release"

	if data is not None and eventName in data:
	eventConfig = data[eventName]

	if eventName == "push" and refType == "branch" and refName in eventConfig:
	eventConfig = eventConfig[refName]

	if eventConfig is not None:
	deploys = eventConfig['deploy'] if 'deploy' in eventConfig else None
	if 'distribute' in eventConfig:
	if 'docker' in eventConfig['distribute']:
	dockerPublishTags = eventConfig['distribute']['docker']
	if 'maven' in eventConfig['distribute']:
	mavenPublishTag = eventConfig['distribute']['maven']
	if 'npm' in eventConfig['distribute']:
	npmPublishTag = eventConfig['distribute']['npm']

	if isMainRepo and ((eventName == "release" and isRelease) or (eventName != "release" and not isRelease)):

	if dockerPublishTags is not None:
	if "$version" in dockerPublishTags:
	firstDockerTag = version
	dockerPublishTags = dockerPublishTags.replace("$version", version)
	else:
	dockerPublishTags = dockerPublishTags.replace("$version", version)
	firstDockerTag = dockerPublishTags.split(",")[0]

	os.system(f"echo 'firstDockerTag={firstDockerTag}' >> $GITHUB_OUTPUT")
	os.system(f" echo 'Manager tags to push to docker hub: {dockerPublishTags}'")
	dockerPublishTags = " ".join(map(lambda t: f"-t openremote/manager:{t.strip()}", dockerPublishTags.split(",")))
	os.system(f"echo 'dockerTags={dockerPublishTags}' >> $GITHUB_OUTPUT")

	if mavenPublishTag is not None:
	mavenPublishTag = mavenPublishTag.replace("$version", version)
	os.system(f" echo 'Maven publish version: {mavenPublishTag}'")
	os.system(f"echo 'mavenTag={mavenPublishTag}' >> $GITHUB_OUTPUT")

	if npmPublishTag is not None:
	npmPublishTag = npmPublishTag.replace("$version", version)
	os.system(f" echo 'npm publish version: {npmPublishTag}'")
	os.system(f"echo 'npmTag={npmPublishTag}' >> $GITHUB_OUTPUT")

	deployStr = None
	if deploys is not None:
	if not isinstance(deploys, list):
	deploys = [deploys]

	deployStr = ""
	managerTagDefault = '#ref'

	for deploy in deploys:
	if 'environment' in deploy:
	deployStr += deploy['environment']
	deployStr += ":"
	if 'managerTag' in deploy:
	if isMainRepo:
	deployStr += deploy['managerTag']
	else:
	# Custom projects manager docker image must match the openremoteVersion from gradle.properties
	deployStr += managerVersion
	else:
	if isMainRepo:
	os.system("echo 'Manager tag not specified so using commit SHA'")
	deployStr += managerTagDefault
	else:
	# Custom projects manager docker image must match the openremoteVersion from gradle.properties
	deployStr += managerVersion

	deployStr += ";"

	deployStr = deployStr.rstrip(";")

	if deployStr is not None and len(deployStr) > 0:
	print(f"Deployments to deploy: {deployStr}")
	os.system(f"echo 'deploys={deployStr}' >> $GITHUB_OUTPUT")
	env:
	IS_MAIN_REPO: ${{ steps.is_main_repo.outputs.value }}
	MANAGER_VERSION: ${{ steps.is_main_repo.outputs.ref }}
	IS_RELEASE: ${{ steps.get-version-details.outputs.is_release }}
	VERSION: ${{ steps.get-version-details.outputs.version }}

	- name: Sanitize deployments value
	id: deployments
	shell: python
	run: \|
	import os

	isMainRepo = os.getenv('IS_MAIN_REPO') == 'true'
	deployments = os.getenv('DEPLOYMENTS')
	eventName = os.getenv('GITHUB_EVENT_NAME')
	refType = os.getenv('GITHUB_REF_TYPE')
	inputTag = os.getenv('INPUT_MANAGER_TAG')
	inputEnv = os.getenv('INPUT_ENVIRONMENT')
	managerVersion = os.getenv('MANAGER_VERSION')

	if eventName == 'workflow_dispatch' and refType != 'tag':
	if isMainRepo:
	tag=inputTag
	if not inputTag:
	tag='#ref'
	else:
	tag=managerVersion
	deployments=f'{inputEnv}:{tag}'

	if not isMainRepo:
	if "-SNAPSHOT" in deployments:
	os.system("SNAPSHOT manager docker images not currently supported for custom project deployment")
	sys.exit(1)

	os.system(f"echo 'value={deployments}' >> $GITHUB_OUTPUT")
	env:
	IS_MAIN_REPO: ${{ steps.is_main_repo.outputs.value }}
	MANAGER_VERSION: ${{ steps.is_main_repo.outputs.ref }}
	DEPLOYMENTS: ${{ steps.ci-cd-output.outputs.deploys }}
	INPUT_ENVIRONMENT: ${{ steps.inputs-and-secrets.outputs.ENVIRONMENT }}
	INPUT_MANAGER_TAG: ${{steps.inputs-and-secrets.outputs.MANAGER_TAG }}

	- name: Define backend test command
	id: test-backend-command
	if: ${{ steps.skip-backend-tests.outputs.value != 'true' }}
	run: echo "value=./gradlew -p test test" >> $GITHUB_OUTPUT

	- name: Define UI test command
	id: test-ui-command
	if: ${{ steps.skip-ui-tests.outputs.value != 'true' }}
	run: echo ""

	- name: Define manager docker build command
	id: manager-docker-command
	if: ${{ steps.is_main_repo.outputs.value == 'true' }}
	shell: bash
	run: \|
	buildPath="manager/build/install/manager"
	commitSha=$(git rev-parse HEAD)
	commitShaShort=$(git rev-parse --short HEAD)

	if [ -n "$MANAGER_TAGS" ] \|\| [[ "$DEPLOYMENTS" == "#ref" ]] \|\| [ -n "$TEST_UI_CMD" ]; then
	if [ -n "$MANAGER_TAGS" ]; then
	command="docker build --push --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 $MANAGER_TAGS $buildPath"
	echo "pushRequired=true" >> $GITHUB_OUTPUT
	echo "managerDockerImage=openremote/manager:$FIRST_MANAGER_TAG" >> $GITHUB_OUTPUT
	else
	command="docker build --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 -t openremote/manager:$commitShaShort $buildPath"
	echo "managerDockerImage=openremote/manager:$commitShaShort" >> $GITHUB_OUTPUT
	fi
	echo "value=$command" >> $GITHUB_OUTPUT
	fi
	echo "buildPath=$buildPath" >> $GITHUB_OUTPUT
	echo "refTag=$commitShaShort" >> $GITHUB_OUTPUT
	env:
	FIRST_MANAGER_TAG: ${{ steps.ci-cd-output.outputs.firstDockerTag }}
	MANAGER_TAGS: ${{ steps.ci-cd-output.outputs.dockerTags }}
	DEPLOYMENTS: ${{ steps.deployments.outputs.value }}
	TEST_UI_CMD: ${{ steps.test-ui-command.outputs.value }}

	- name: Define maven publish command
	id: maven-publish-command
	if: ${{ steps.ci-cd-output.outputs.mavenTag != '' }}
	shell: bash
	run: \|
	command="./gradlew publishToSonatype closeAndReleaseSonatypeStagingRepository --no-parallel -PsigningKey=$MAVEN_SIGNING_KEY -PsigningPassword=$MAVEN_SIGNING_PASSWORD -PpublishUsername=$MAVEN_USERNAME -PpublishPassword=$MAVEN_PASSWORD"
	echo "value=$command" >> $GITHUB_OUTPUT
	env:
	MAVEN_TAG: ${{ steps.ci-cd-output.outputs.mavenTag }}
	MAVEN_SIGNING_PASSWORD: ${{ steps.inputs-and-secrets.outputs._TEMP_MAVEN_SIGNING_PASSWORD }}
	MAVEN_SIGNING_KEY: ${{ steps.inputs-and-secrets.outputs._TEMP_MAVEN_SIGNING_KEY }}
	MAVEN_USERNAME: ${{ steps.inputs-and-secrets.outputs._TEMP_MAVEN_USERNAME }}
	MAVEN_PASSWORD: ${{ steps.inputs-and-secrets.outputs._TEMP_MAVEN_PASSWORD }}

	- name: Define deployment docker build command
	id: deployment-docker-command
	shell: bash
	run: \|
	if [ "$DEPLOYMENT_DOCKERFILE_EXISTS" == 'true' ]; then
	if [ "$DEPLOYMENT_GRADLE_EXISTS" != 'true' ]; then
	echo "Deployment must have a build.gradle file to prepare the deployment files in the deployment/build dir"
	exit 1
	fi

	buildPath="deployment/build"
	commitSha=$(git rev-parse HEAD)
	commitShaShort=$(git rev-parse --short HEAD)
	echo "buildPath=$buildPath" >> $GITHUB_OUTPUT
	echo "refTag=$commitShaShort" >> $GITHUB_OUTPUT

	if [ -n "$DEPLOYMENTS" ]; then
	command="docker build --build-arg GIT_COMMIT=$commitSha --platform linux/amd64,linux/aarch64 -t openremote/deployment:$commitShaShort $buildPath"
	echo "value=$command" >> $GITHUB_OUTPUT
	fi
	fi
	env:
	DEPLOYMENTS: ${{ steps.deployments.outputs.value }}
	DEPLOYMENT_DOCKERFILE_EXISTS: ${{ steps.check_deployment_dockerfile.outputs.files_exists }}
	DEPLOYMENT_GRADLE_EXISTS: ${{ steps.check_deployment_gradle.outputs.files_exists }}

	- name: Define installDist command
	id: install-command
	shell: bash
	run: \|
	if [ -n "$MANAGER_DOCKER_CMD" ]; then
	echo "value=./gradlew installDist" >> $GITHUB_OUTPUT
	elif [ -n "$DEPLOYMENT_DOCKER_CMD" ]; then
	echo "value=./gradlew -p deployment installDist" >> $GITHUB_OUTPUT
	fi
	env:
	MANAGER_DOCKER_CMD: ${{ steps.manager-docker-command.outputs.value }}
	DEPLOYMENT_DOCKER_CMD: ${{ steps.deployment-docker-command.outputs.value }}

	- name: Login to DockerHub
	if: ${{ steps.manager-docker-command.outputs.pushRequired == 'true' }}
	uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
	with:
	username: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_USER }}
	password: ${{ steps.inputs-and-secrets.outputs._TEMP_DOCKERHUB_PASSWORD }}

	- name: set up QEMU
	if: ${{ steps.manager-docker-command.outputs.value != '' \|\| steps.deployment-docker-command.outputs.value != '' }}
	uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3
	with:
	platforms: linux/amd64,linux/aarch64

	- name: install buildx
	if: ${{ steps.manager-docker-command.outputs.value != '' \|\| steps.deployment-docker-command.outputs.value != '' }}
	id: buildx
	uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3
	with:
	version: latest
	install: true

	- name: Set up JDK 21 and gradle cache
	id: java
	if: ${{ steps.install-command.outputs.value != '' \|\| steps.test-backend-command.outputs.value != '' }}
	uses: actions/setup-java@3a4f6e1af504cf6a31855fa899c6aa5355ba6c12 # v4
	with:
	distribution: 'temurin'
	java-version: '21'
	cache: 'gradle'

	- name: Get yarn cache directory path
	id: yarn-cache-dir-path
	run: echo "dir=$(yarn config get cacheFolder)" >> $GITHUB_OUTPUT

	- name: Yarn cache
	uses: actions/cache@d4323d4df104b026a6aa633fdb11d772146be0bf # v4
	if: steps.skip-cicd.outputs.value != 'true'
	id: yarn-cache # use this to check for `cache-hit` (`steps.yarn-cache.outputs.cache-hit != 'true'`)
	with:
	path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
	key: ${{ runner.os }}-yarn---${{ hashFiles('**/yarn.lock') }}
	restore-keys: \|
	${{ runner.os }}-yarn---

	- name: Output info
	if: steps.skip-cicd.outputs.value != 'true'
	run: \|
	echo "************************************************************"
	echo "************ INFO *****************"
	echo "************************************************************"
	echo 'Trigger event: ${{ github.event_name }}'
	echo 'Is main repo: ${{ steps.is_main_repo.outputs.value == 'true' }}'
	echo 'Has deployment dockerfile: ${{ steps.check_deployment_dockerfile.outputs.files_exists == 'true' }}'
	echo 'OpenRemote version: ${{ steps.is_main_repo.outputs.ref }}'
	echo 'Manager commit SHA: ${{ steps.manager-docker-command.outputs.refTag }}'
	echo 'Deployment commit SHA: ${{ steps.deployment-docker-command.outputs.refTag }}'
	echo 'Deployments: ${{ steps.deployments.outputs.value }}'
	echo 'Test backend command: ${{ steps.test-backend-command.outputs.value }}'
	echo 'Test UI command: ${{ steps.test-ui-command.outputs.value }}'
	echo 'Manager docker build command: ${{ steps.manager-docker-command.outputs.value }}'
	echo 'Manager docker image: ${{ steps.manager-docker-command.outputs.managerDockerImage }}'
	echo 'Maven publish command: ${{ steps.maven-publish-command.outputs.value }}'
	echo 'Deployment docker build command: ${{ steps.deployment-docker-command.outputs.value }}'
	echo 'InstallDist command: ${{ steps.install-command.outputs.value }}'
	echo "Java version: $(java --version)"
	echo "Yarn version: $(yarn -v)"
	echo "Node version: $(node -v)"
	echo "************************************************************"
	echo "************************************************************"

	- name: Pull docker images
	if: ${{ steps.test-backend-command.outputs.value != '' \|\| steps.test-ui-command.outputs.value != '' }}
	run: \|
	# Only need keycloak and postgres services for backend testing
	if [ -z $TEST_UI_COMMAND ]; then
	composeProfile='profile/dev-testing.yml'
	else
	composeProfile='profile/dev-ui.yml'
	fi

	# Pull the images
	docker compose -f $composeProfile pull
	env:
	TEST_UI_COMMAND: ${{ steps.test-ui-command.outputs.value }}

	# Use docker layer caching to speed up image building
	- uses: jpribyl/action-docker-layer-caching@c632825d12ec837065f49726ea27ddd40bcc7894 # v0.1.1
	if: ${{ steps.manager-docker-command.outputs.value != '' \|\| steps.deployment-docker-command.outputs.value != '' }}
	# Ignore the failure of a step and avoid terminating the job.
	continue-on-error: true

	- name: Update version in package.json files
	id: update-package-json-files
	run: \|
	VERSION=$(awk '{print tolower($0)}' <<< $VERSION)

	QUALIFIER=$(sed -E 's/.+-([a-z]+).*/\1/g' <<< $VERSION)
	if [ "$QUALIFIER" == "$VERSION" ]; then
	QUALIFIER=""
	fi

	if [ "$QUALIFIER" == "" ]; then
	TAG="latest"
	else
	if [ "$QUALIFIER" == "snapshot" ]; then
	# Add a timestamp to snapshot versions
	VERSION="$VERSION.$(date +'%Y%m%d%H%M%S')"
	else
	# Separate qualifiers from numbers of alpha, beta, rc versions
	VERSION=$(sed "s/$QUALIFIER/$QUALIFIER./g" <<< $VERSION)
	fi
	TAG="$QUALIFIER"
	fi

	find ui -maxdepth 3 -name package.json \| xargs -I{} sed -i -E "s#\"version\": \".+\",#\"version\": \"$VERSION\",#" {}

	echo "tag=$TAG" >> $GITHUB_OUTPUT
	echo "version=$VERSION" >> $GITHUB_OUTPUT
	env:
	VERSION: ${{ steps.get-version-details.outputs.version_with_qualifier }}

	- name: Run backend tests
	id: run-backend-tests
	if: ${{ steps.test-backend-command.outputs.value != '' }}
	run: \|
	composeProfile='profile/dev-testing.yml'

	# Make temp dir and set mask to 777 as docker seems to run as root
	mkdir -p tmp
	chmod 777 tmp

	# Define cleanup command
	echo "cleanup=docker compose -f $composeProfile down" >> $GITHUB_OUTPUT

	# Start the stack
	echo "docker compose -f ${composeProfile} up -d --no-build"
	docker compose -f ${composeProfile} up -d --no-build

	# Run the tests
	${{ steps.test-backend-command.outputs.value }}
	timeout-minutes: 20
	#continue-on-error: true

	- name: Archive backend test results
	if: always()
	uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4
	with:
	name: backend-test-results
	path: test/build/reports/tests

	- name: Cleanup backend tests
	if: ${{ steps.test-backend-command.outputs.value != '' && steps.test-ui-command.outputs.value != '' }}
	run: ${{ steps.run-backend-tests.outputs.cleanup }}

	- name: Run frontend tests
	if: steps.test-ui-command.outputs.value != ''
	run: \|
	composeProfile='profile/dev-ui.yml'

	if [ $IS_MAIN_REPO == 'false' ]; then
	composeProfile="openremote/$composeProfile"
	fi

	# Start the stack
	MANAGER_VERSION=$MANAGER_TAG docker compose -f $composeProfile up -d --no-build

	# Run the tests
	${{ steps.test-ui-command.outputs.value }}
	env:
	IS_MAIN_REPO: ${{ steps.is_main_repo.outputs.value }}
	MANAGER_TAG: ${{ steps.manager-docker-command.outputs.refTag }}
	timeout-minutes: 20
	continue-on-error: true

	- name: Run install dist
	if: steps.install-command.outputs.value != ''
	shell: python
	run: \|
	import json
	import os
	import sys
	import subprocess

	inputsAndSecrets = json.loads(os.getenv("INPUTS_AND_SECRETS"))

	# Output inputs and secrets as environment variables for build
	for key, value in inputsAndSecrets.items():
	if "." in key:
	continue

	# Look for temp and env prefixed keys
	if key.startswith("_"):
	if key.startswith("_TEMP_"):
	key = key.replace("_TEMP_", "")
	else:
	continue

	os.system(f"echo 'Setting environment variable {key}...'")
	os.putenv(key, value)

	buildCmd = os.getenv("CMD")
	result = subprocess.run(f"{buildCmd}", shell=True)

	if result.returncode != 0:
	os.system("echo 'installDist failed'")
	sys.exit(result.returncode)

	env:
	CMD: ${{ steps.install-command.outputs.value }}
	INPUTS_AND_SECRETS: ${{ toJSON(steps.inputs-and-secrets.outputs) }}
	timeout-minutes: 20

	- name: Run manager docker build command
	if: steps.manager-docker-command.outputs.value != ''
	run: \|
	${{ steps.manager-docker-command.outputs.value }}

	- name: Scan manager docker image
	if: steps.manager-docker-command.outputs.pushRequired == 'true'
	uses: anchore/scan-action@3343887d815d7b07465f6fdcd395bd66508d486a # v3
	id: manager-anchore-scan
	with:
	image: ${{ steps.manager-docker-command.outputs.managerDockerImage }}
	fail-build: false
	severity-cutoff: critical

	- name: Upload Anchore scan SARIF report
	if: \|
	!cancelled() &&
	steps.manager-docker-command.outputs.pushRequired == 'true'
	uses: github/codeql-action/upload-sarif@e488e3c8239c26bf8e6704904a8cb59be658d450 # v3
	with:
	sarif_file: ${{ steps.manager-anchore-scan.outputs.sarif }}

	- name: Inspect Anchore scan SARIF report
	if: \|
	!cancelled() &&
	steps.manager-docker-command.outputs.pushRequired == 'true'
	run: cat ${{ steps.manager-anchore-scan.outputs.sarif }}

	- name: Run maven publish command
	if: steps.maven-publish-command.outputs.value != ''
	run: \|
	${{ steps.maven-publish-command.outputs.value }}

	- name: Run npm publish command
	if: ${{ steps.ci-cd-output.outputs.npmTag != '' }}
	run: \|
	echo 'npmRegistries:' >> .yarnrc.yml
	echo ' "https://registry.yarnpkg.com":' >> .yarnrc.yml
	echo " npmAuthToken: $NPM_AUTH_TOKEN" >> .yarnrc.yml

	yarn workspaces foreach --all --no-private --topological npm publish --tag $TAG
	env:
	NPM_AUTH_TOKEN: ${{ steps.inputs-and-secrets.outputs._TEMP_NPM_AUTH_TOKEN }}
	TAG: ${{ steps.update-package-json-files.outputs.tag }}

	- name: Run deployment docker command
	if: steps.deployment-docker-command.outputs.value != ''
	run: \|
	${{ steps.deployment-docker-command.outputs.value }}

	- name: Do deployments
	if: steps.deployments.outputs.value != ''
	shell: python
	run: \|
	import json
	import os
	import sys
	import subprocess

	deployments = os.getenv("DEPLOYMENTS")
	deployments = deployments.split(";")
	managerRef = os.getenv("MANAGER_REF")
	deploymentRef = os.getenv("DEPLOYMENT_REF")
	isMainRepo = os.getenv("IS_MAIN_REPO") == 'true'
	inputsAndSecrets = json.loads(os.getenv("INPUTS_AND_SECRETS"))
	ipv4 = os.getenv("IPV4")
	ipv6 = os.getenv("IPV6")

	failure = False

	# Determine deploy script to use
	deployScript = ".ci_cd/deploy.sh"

	if not os.path.exists(deployScript) and not isMainRepo:
	deployScript = "openremote/.ci_cd/deploy.sh"

	if not os.path.exists(deployScript):
	os.system(f"Deploy script not found '{deployScript}'")
	sys.exit(1)

	for deployment in deployments:
	dep = deployment.split(":")
	env = dep[0]
	managerTag = dep[1]
	managerTagFound = True

	os.putenv("MANAGER_TAG", managerTag)
	os.putenv("ENVIRONMENT", env)

	# Clean stale ssh credentials and temp files
	os.system("rm temp.env 2>/dev/null")
	os.system("rm ssh.key 2>/dev/null")
	os.system("rm -r temp 2>/dev/null")
	os.system("mkdir temp")

	# ------------------------------------------------------
	# Output env variables to temp env file for POSIX shell
	# ------------------------------------------------------

	# Output inputs and secrets (spacial handling for SSH_KEY and some other variables)
	# _$ENV_ prefixed keys are output last (to override any non env specific keys)
	environment = (env if env else "").upper()
	prefix = "_" + environment + "_"

	for key, value in inputsAndSecrets.items():
	if "." in key:
	continue

	envFile = "temp/env"

	# Look for temp and env prefixed keys
	if key.startswith("_"):
	if key.startswith("_TEMP_"):
	key = key.replace("_TEMP_", "")
	envFile = "temp.env"
	elif key.startswith(prefix):
	key = key.replace(prefix, "")
	else:
	continue

	if key == "github_token":
	continue
	else:
	os.system(f"echo 'Secret found {key}...'")
	if key == "SSH_KEY":
	os.system(f"echo \"{value}\" > ssh.key")
	else:
	lines = len(value.split("\n"))
	if lines > 1:
	os.system(f"echo '{key}='\"'\"'' >> {envFile}")
	os.system(f"echo '{value}'\"'\"'' >> {envFile}")
	else:
	os.system(f"echo '{key}='\"'\"'{value}'\"'\"'' >> {envFile}")

	# Output new line
	os.system(f"echo '\n' >> {envFile}")

	# Output env file if exists
	if os.path.exists(".ci_cd/env/.env"):
	os.system(f"echo 'Outputting .ci_cd/env/.env to temp/env'")
	os.system("cat .ci_cd/env/.env >> temp/env")
	# Output new line
	os.system(f"echo '\n' >> {envFile}")

	# Output environment specific env file if exists
	if env is not None and env != '' and os.path.exists(f".ci_cd/env/{env}.env"):
	os.system(f"echo 'Outputting .ci_cd/env/{env}.env to temp/env'")
	os.system(f"cat .ci_cd/env/{env}.env >> temp/env")
	# Output new line
	os.system(f"echo '\n' >> {envFile}")

	# Set CIDR environment variable
	if ipv4 is not None and ipv4 != '':
	os.putenv("CIDR", ipv4 + '/32')
	elif ipv6 is not None and ipv6 != '':
	os.putenv("CIDR", ipv6 + '/64')

	# Execute deploy script
	os.system(f"echo 'Executing deploy script for deployment: managerTag={managerTag} deploymentTag={deploymentRef} environment={env}'")
	# Uncomment this in combination with the SSH debug step afterwards to debug deployment script
	#sys.exit(0)
	result = subprocess.run(f"bash {deployScript}", shell=True)

	if result.returncode != 0:
	os.system(f"echo 'Deployment failed: managerTag={managerTag} deploymentTag={deploymentRef} environment={env}'")
	failure = True
	continue

	if failure == True:
	os.system("echo 'One or more deployments failed'")
	sys.exit(1)
	env:
	IS_MAIN_REPO: ${{ steps.is_main_repo.outputs.value }}
	DEPLOYMENTS: ${{ steps.deployments.outputs.value }}
	MANAGER_DOCKER_BUILD_PATH: ${{ steps.manager-docker-command.outputs.buildPath }}
	DEPLOYMENT_DOCKER_BUILD_PATH: ${{ steps.deployment-docker-command.outputs.buildPath }}
	MANAGER_REF: ${{ steps.manager-docker-command.outputs.refTag }}
	DEPLOYMENT_REF: ${{ steps.deployment-docker-command.outputs.refTag }}
	INPUTS_AND_SECRETS: ${{ toJSON(steps.inputs-and-secrets.outputs) }}
	IPV4: ${{ steps.ip-address.outputs.ipv4 }}
	IPV6: ${{ steps.ip-address.outputs.ipv6 }}

	# - name: Setup upterm session
	# uses: lhotari/action-upterm@b0357f23233f5ea6d58947c0c402e0631bab7334 # v1
	# with:
	# ## limits ssh access and adds the ssh public keys of the listed GitHub users
	# limit-access-to-actor: true

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rules grouping support, with tree dragging functionality (#1680) #16

Workflow file

Rules grouping support, with tree dragging functionality (#1680) #16

Uh oh!

Workflow file for this run