add command line flag for system test result dumps

[efd6]

Please take a look.

[mrodm]

Adding just one flag it forces to always set a prefix to the user, but probably they often need just a default value for that.

What about adding two flags to manage this feature?

• One flag to enable or not dumping the documents (--dump-docs) without defining any path or prefix.
  • The default filename could be the same one used when the environment variable is set.
  • Should this file be saved in the same location where elastic-package runs ? Take into account that this could change with the parameter -C.

    elastic-package -C test/packages/parallel/nginx test system -v

• Another flag to set a custom prefix (--dump-docs-prefix) or even it could be set the whole path (--dump-docs-output) leaving to the user the choice to where save the output file. I think I'd prefer the latter.

# enable dumping documents and using the default filename for the path
elastic-package test system -v --dump-docs-

# enable dumping documents and using a custom path
elastic-package test system -v --dump-docs --dump-docs-output /path/to/dump.json

Doing it like this, it would be similar on how coverage is set in elastic-package:

• --test-coverage to enable the feature,
• --coverage-format to choose which format is required.

[efd6]

I think this comes down to a philosophical position; I think having a default location for things like this is a significant misfeature. IMO actions should require explicit requests. Including things like this in the config add complexity to the user experience and is also IMO a misfeature.

I'm not entirely sure what the gain is for users in adding this additional complexity.

[mrodm]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix. That's why I would say that the most common usage would be (not requiring to set any prefix there):

elastic-package test system -v --dump-docs

Moreover, doing it like this, it would enable this feature and it could be coded to be equivalent to set the ELASTIC_PACKAGE_TEST_DUMP_SCENARIO_DOCS environment variable. So users setting the environment variable and setting that flag can expect the same exact behavior of elastic-package.

For the other flag I proposed, I thought there could be users that require some specific known file, for instance they are running tests with different packages and they want those files to be easily identified. That's my reasoning to add that new --dump-docs-output or --dump-docs-prefix flag, so they can decide how to save those documents.

# allow a full path
elastic-package test system -v --dump-docs --dump-docs-output /path/store/documents.json

# or as you proposed adding a prefix and save them in the same location where elastic-package runs
elastic-package test system -v --dump-docs --dump-docs-prefix "package-test"

The concern I have of writing those new documents in the same location as elastic-package runs is that this could lead to users committing and pushing those files in the repository inadvertently.

[efd6]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix.

This is certainly not what I'd want; doing this means that I cannot script obtaining the data.

In terms of work, specifying a location is easier than digging through logs to find the name that the program chose. In my ideal world, the output would not be to a file whose name was under the control of the program at all. IMO this should be entirely under the control of the user, but I made a concession to the current behaviour on the basis that it was still relatively easy to obtain the files if the prefix was known.

[mrodm]

Ok, got it.

Could it be changed what the value would indicate for this flag? Instead of setting a prefix for files, this flag could set the path to a folder.

I'm trying to be consistent with other commands/parameters used in elastic-package. For instance, in these commands --output is a path to a folder:

• dump stack logs: elastic-package stack dump --output <path>
• dump assets: elastic-package dump agent-policies --output <path>

That folder should be checked (and created if needed) by elastic-package. And it would be the folder where the JSON files are stored.
Files stored in that folder could follow the same naming patterns as it was set when setting the environment variable:

    dumpPath = filepath.Join(dumpFolderBase, fmt.Sprintf("elastic-package-test-docs-dump-%s.json", time.Now().Format("20060102150405")))

So, elastic-package would be like this:

elastic-package test system -v --dump-logs "docs_folder_package/"

Probably, doing it like this, the parameter could be renamed as --dump-logs-to.

[efd6]

I think I'd also prefer this flag to specify a folder instead of a prefix.

Which behaviour would you prefer:

1. mkdir -p ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

2. mkdir ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

(where ${prefix_slug} is determined without user input and $(data) is the current timestamp suffix logic).

even in the unix world, scripts create directories, by calling mkdir.

In those cases they mkdir not mkdir -p. This is the distinction that I was making.

[jsoriano]

I think I'd also prefer this flag to specify a folder instead of a prefix.

Which behaviour would you prefer.

1. mkdir -p ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

2. mkdir ${outdir}
   echo ${result} > ${outdir}/${prefix_slug}-$(date).ndjson
   

(where ${prefix_slug} is determined without user input and $(data) is the current timestamp suffix logic).

I prefer the first behavior, as this is more convenient for users.

This is also what elastic-package does at the moment in most of the cases, so it would be consistent with current behaviors:

.../elastic-package$ git grep Mkdir | wc -l
65
.../elastic-package$ git grep MkdirAll | wc -l
60
.../elastic-package$ git grep MkdirTemp | wc -l
4

Though it is also true that maybe scripts don't always try to create parent directories, so for this case both options would work for me if you prefer the second behavior.

[efd6]

I would prefer the second; explicit is better than implicit.

[mrodm]

Ok!
So then, IIUC with that option, the command would be like:

elastic-package test system --dump-docs <folder>

And elastic-package would create the (last) folder and use the same prefix (prefix_slug as mentioned above) for the filename as set with the environment variable (elastic-package-test-docs-dump-%s.json). Is that right ?

[efd6]

That's correct, but only if there is a path to the location where the directory would be created.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1125166

History

• 💚 Build #6275 succeeded eb7be7f
• 💔 Build #6274 failed 9456b6f
• 💚 Build #6266 succeeded a9fe9ba
• 💔 Build #6265 failed cd7a5e3

cc @efd6

[teresaromero]

just for understanding, why we change the behavior, so now the error is not thrown but joined to the result of validateTestScenario. Does it make sense to run r.validateTestScenario if this fails?

[efd6]

Why would it be relevant to the success of the test if some aspect of the dump fails? As an example, if the dump fails because of an fs error, do we now not care that the validation is not valid?

[teresaromero]

that is the question i asked in order to understand the change. before, the test was failing if dump failed, isn't it? i am just asking to understand, lacking context here.

[efd6]

Yes. I think that behaviour was unhelpful. (see this for context)