internal/testrunner/script: add script testing package

[efd6]

This adds script testing for data streams. This is the MVP, future versions can be generalised to operate on input packages.

The current implementation supports:

• pipeline testing
• system testing
• package upgrade testing (only to latest, not to arbitrary versions)
• shared stack
• independent stack
• docker services

Not supported:

• test coverage
• report output configuration
• k8s services
• tf services (ish, these can be shimmed via docker)

Depends on a v3.5.1 of package-spec (currently shimmed in via a go.mod replace directive, to be removed)

Please take a look.

[efd6]

/test

[efd6]

This does not appear to run in the CI, but it's entirely unclear how to achieve that.

[mrodm]

In order to be executed in the CI, it needs to be updated this shell script that generates dynamically all the Buildkite steps to be executed:

https://github.com/elastic/elastic-package/blob/d5f73ab15af1dcf3547ed789f8ad4631257f4197/.buildkite/pipeline.trigger.integration.tests.sh

Depending on how it is required to launch these steps, all packages in one step or each package in its own CI step, it would be needed to do different modifications:

• all packages in one step:
  • similar to with-kind or other CI step (Buildkite link)
  • it should be added a new element (Makefile target name) to this list:
    

    elastic-package/.buildkite/pipeline.trigger.integration.tests.sh

    Lines 42 to 47 in d5f73ab

    	CHECK_PACKAGES_TESTS=(
    	test-check-packages-other
    	test-check-packages-with-kind
    	test-check-packages-with-custom-agent
    	test-check-packages-benchmarks
    	)

• each package in its own CI step:
  • it would be needed to duplicate this code for the new Makefile target name
  • for instance, for packages under parallel folder:
    

    elastic-package/.buildkite/pipeline.trigger.integration.tests.sh

    Lines 117 to 138 in d5f73ab

    	pushd test/packages/parallel > /dev/null
    	while IFS= read -r -d '' package ; do
    	package_name=$(basename "${package}")
    	echo " - label: \":go: Integration test: ${package_name}\""
    	echo " key: \"integration-parallel-${package_name}-agent\""
    	echo " command: ./.buildkite/scripts/integration_tests.sh -t test-check-packages-parallel -p ${package_name}"
    	echo " env:"
    	echo " UPLOAD_SAFE_LOGS: 1"
    	echo " agents:"
    	echo " provider: \"gcp\""
    	echo " image: \"${UBUNTU_X86_64_AGENT_IMAGE}\""
    	echo " plugins:"
    	echo " # See https://github.com/elastic/oblt-infra/blob/main/conf/resources/repos/integrations/01-gcp-buildkite-oidc.tf"
    	echo " # This plugin authenticates to Google Cloud using the OIDC token."
    	echo " - elastic/oblt-google-auth#v1.2.0:"
    	echo " lifetime: 10800 # seconds"
    	echo " project-id: \"elastic-observability-ci\""
    	echo " project-number: \"911195782929\""
    	echo " artifact_paths:"
    	echo " - build/test-results/*.xml"
    	echo " - build/test-coverage/coverage-*.xml" # these files should not be used to compute the final coverage of elastic-package
    	done < <(find . -maxdepth 1 -mindepth 1 -type d -print0)

[efd6]

The logic in this function is rudimentary and only intended to allow an MVP to be demonstrated. In order to be able to render reports and redirect to file output this needs enhancement.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: fdc0951

Failed CI Steps

• :go: Integration test (with logstash): ti_misp
• :go: Integration test: mongodb

History

• 💔 Build #6472 failed 7f4815f
• 💔 Build #6471 failed c5e6b61
• 💔 Build #6470 failed 61abadf
• 💔 Build #6465 failed 460341d

cc @efd6