Reduce the scope of the legacy gcp secrets

[fr4nc1sc0-r4m0n]

What does this PR do?

This PR aims to reduce the scope of the legacy kv/ci-shared/observability-ingest/cloud/gcp secret only for the integration-test-matrix pipeline.
This is part of the global process about removing this secret usage.

Why is it important?

It allows us to simplify the pre-command script and to limit the usage of a legacy secret which will be removed in the future.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

• This is part of: https://github.com/elastic/observability-robots/issues/2800

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[v1v]

Can you confirm if the name of the pipeline is correct? See

elastic-agent/catalog-info.yaml

Lines 360 to 371 in 5546bd7

	spec:
	type: buildkite-pipeline
	owner: group:ingest-fp
	system: platform-ingest
	implementation:
	apiVersion: buildkite.elastic.dev/v1
	kind: Pipeline
	metadata:
	name: buildkite-elastic-agent-integration-matrix
	description: Runs elastic-agent integration tests for all supported platforms
	spec:
	pipeline_file: ".buildkite/pipeline.integration-test-matrix.yml"

I can see BUILDKITE_PIPELINE_SLUG="buildkite-elastic-agent-integration-matrix" in the builds:

• https://buildkite.com/elastic/buildkite-elastic-agent-integration-matrix/builds/76

[fr4nc1sc0-r4m0n]

You are absolutely right! Thanks

[v1v]

IIUC, API_KEY_TOKEN is used in the its, so we will need to create a new conditional for the matrix and the GCP

[v1v]

we don't need this conditional here, I think the conditionals are safe here:

• if variable exists
• if file exists

Regarldess of the BK pipeline, let's keep it simple

[fr4nc1sc0-r4m0n]

I thought the same but found this error:

• https://buildkite.com/elastic/elastic-agent/builds/29152#019a10f5-1f13-4e1d-80ef-b1edac2f88e4

The thing is that I don't see the set -u option set which is the reason for this error so I preferred to check the pipeline slug as we did in the pre-command.

[v1v]

Change:

• "$GOOGLE_APPLICATION_CREDENTIALS" → "${GOOGLE_APPLICATION_CREDENTIALS:-}"
• "$TEST_INTEG_AUTH_GCP_SERVICE_TOKEN_FILE" → "${TEST_INTEG_AUTH_GCP_SERVICE_TOKEN_FILE:-}"

This ensures the pre-exit won't fail when these environment variables are not set and set -u is set.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 9a5b5d9

Failed CI Steps

• x86_64:sudo: install-uninstall

History

• 💛 Build #29240 was flaky 2607890
• 💔 Build #29188 failed 6536db8
• 💔 Build #29152 failed 3df2b36

cc @fr4nc1sc0-r4m0n