Skip to content

[WIP] Add a common utility function which makes debugging test failures #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[WIP] Add a common utility function which makes debugging test failures #161

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d8dfb16
Add a common utility function which makes debugging test failures
Kami May 15, 2019
122ba6d
Use consistent line breaks.
Kami May 15, 2019
1f96aea
Pull in improvement from master.
Kami May 15, 2019
0c1ef09
Add function that echos stderr
blag May 15, 2019
13da9f6
Fix some typos and Bashisms
blag May 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions cli/test_pack_python3.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
load '../test_helpers/bats-support/load'
load '../test_helpers/bats-assert/load'
load '../test_helpers/common'

setup() {
if [[ ! -d /opt/stackstorm/packs/examples ]]; then
Expand Down Expand Up @@ -42,20 +43,15 @@ skip_tests_if_python3_is_not_available_or_if_already_running_under_python3() {
@test "packs.setup_virtualenv without python3 flags works and defaults to Python 2" {
skip_tests_if_python3_is_not_available_or_if_already_running_under_python3

SETUP_VENV_RESULTS=$(st2 run packs.setup_virtualenv packs=examples -j)
SETUP_VENV_RESULTS=$(run_command_and_log_output st2 run packs.setup_virtualenv packs=examples -j)
run eval "echo '$SETUP_VENV_RESULTS' | jq -r '.result.result'"
assert_success

assert_output "Successfully set up virtualenv for the following packs: examples"
Copy link
Member

@arm4b arm4b May 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, in this scenario avoiding another run and mutating output will help to preserve original command results. And so we'll do assert against the original output, step with jq filtering here is overcomplication:

-	run eval "echo '$SETUP_VENV_RESULTS' | jq -r '.result.result'"
-	assert_success
-	assert_output "Successfully set up virtualenv for the following packs: examples"
+	assert_output --partial "Successfully set up virtualenv for the following packs: examples"

Copy link
Contributor

@blag blag May 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That change is fine, but in addition to checking for a specific string, we should also check that the output is valid JSON.

One of the commands in one of the BATS tests returned a JSON array concatenated to a JSON dict:

[
  "example_element_1",
  "example_element_2"
]
{
  "example_key_1": "example_value_1",
  "example_key_2": "example_value_2"
}

Both of those objects individually are valid JSON, but simply concatenating them together produces invalid JSON.

So we do still need some way to validate that the produced output is valid JSON. This might be good to do in a custom assert_ function, and could even simply wrap jq.

But however we check for JSON validity, this simplification is a good one.


run eval "echo '$SETUP_VENV_RESULTS' | jq -r '.status'"
assert_success

assert_output "succeeded"

run st2-register-content --register-pack /opt/stackstorm/packs/examples/ --register-all
assert_success

run /opt/stackstorm/virtualenvs/examples/bin/python --version
assert_output --partial "Python 2.7"

Expand All @@ -66,7 +62,7 @@ skip_tests_if_python3_is_not_available_or_if_already_running_under_python3() {
@test "packs.setup_virtualenv with python3 flag works" {
skip_tests_if_python3_is_not_available_or_if_already_running_under_python3

SETUP_VENV_RESULTS=$(st2 run packs.setup_virtualenv packs=examples python3=true -j)
SETUP_VENV_RESULTS=$(run_command_and_log_output st2 run packs.setup_virtualenv packs=examples python3=true -j)
run eval "echo '$SETUP_VENV_RESULTS' | jq -r '.result.result'"
assert_success
Copy link
Member

@arm4b arm4b May 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this specific case, if we really need jq, this construct with run | jq + assert should be replaced in favor of creating custom assert_output_jq function that will help us to avoid overriding original test results/outputs.

Copy link
Contributor

@blag blag May 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just read this comment after coming to the same conclusion. 👍


Expand All @@ -81,7 +77,7 @@ skip_tests_if_python3_is_not_available_or_if_already_running_under_python3() {

assert_output --partial "Python 3."

RESULT=$(st2 run examples.python_runner_print_python_version -j)
RESULT=$(run_command_and_log_output st2 run examples.python_runner_print_python_version -j)
Copy link
Member

@arm4b arm4b May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
RESULT=$(run_command_and_log_output st2 run examples.python_runner_print_python_version -j)
run st2 run examples.python_runner_print_python_version -j

assert_success

run eval "echo '$RESULT' | jq -r '.result.stdout'"
Expand All @@ -99,7 +95,7 @@ skip_tests_if_python3_is_not_available_or_if_already_running_under_python3() {

run eval "echo '$RESULT' | jq -r '.result.stdout'"
assert_success
assert_output --regexp ".*PYTHONPATH: /usr/(local/)?lib/python3.*"
assert_output --regexp ".*PYTHONPATH: .*/usr/(local/)?lib/python3.*"
}

@test "python3 imports work correctly" {
Expand All @@ -108,7 +104,7 @@ skip_tests_if_python3_is_not_available_or_if_already_running_under_python3() {
run st2 pack install python3_test --python3 -j
assert_success

RESULT=$(st2 run python3_test.test_stdlib_import -j)
RESULT=$(run_command_and_log_output st2 run python3_test.test_stdlib_import -j)
assert_success

run eval "echo '$RESULT' | jq -r '.result.result'"
Expand Down
69 changes: 69 additions & 0 deletions test_helpers/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Copyright 2019 Extreme Networks, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Common bash utility functions used by test code

run_command_and_log_output() {
# Utility function which runs a bash command and logs the command output (stdout) and exit
# code.
#
# This comes in handy in scenarios where you want to save command stdout in a variable, but you
# also want to output the command stdout to make troubleshooting / debugging in test failures
# easier.
#
# Example usage:
#
# RESULT=$(run_command_and_log_output st2 run pack install packs=examples)
#
# 1. Run the command and capture the outputs
eval "$({ stderr=$({ stdout=$($@); exit_code=$?; } 2>&1; declare -p stdout exit_code >&2); declare -p stderr ; } 2>&1)"

# 2. Log the output to stderr
>&2 echo "=========="
>&2 echo "Ran command: ${@}"
>&2 echo "stdout: ${stdout}"
>&2 echo "stderr: ${stderr}"
>&2 echo "exit code: ${exit_code}"
>&2 echo "=========="

# 3. Return original command value
echo "${stdout}"
}

run_command_and_log_output_get_stderr() {
# Utility function which runs a bash command and logs the command output (stdout) and exit
# code.
#
# This comes in handy in scenarios where you want to save command stdout in a variable, but you
# also want to output the command stdout to make troubleshooting / debugging in test failures
# easier.
#
# Example usage:
#
# RESULT=$(run_command_and_log_output_get_stderr st2 run pack install packs=examples)
#
# 1. Run the command and capture the outputs
eval "$({ stderr=$({ stdout=$($@); exit_code=$?; } 2>&1; declare -p stdout exit_code >&2); declare -p stderr ; } 2>&1)"

# 2. Log the output to stderr
>&2 echo "=========="
>&2 echo "Ran command: ${@}"
>&2 echo "stdout: ${stdout}"
>&2 echo "stderr: ${stderr}"
>&2 echo "exit code: ${exit_code}"
>&2 echo "=========="

# 3. Return original command error
echo "${stderr}"
}