Multi PHP + Multi Environment + Check all commits #212

javiercasares · 2024-03-16T18:02:19Z

This is the Cloudfest Hackathon 2024 PR.

This PR solves:

Multi PHP. There is a new env option where you can set all the different PHP versions you want to test.
Multi Environment. You can set a label to differentiate, for example, your “shared”, “vps”, or “cloud” setups.
Check all commits. Until now, only the latest commit can be checked. With this new option, we can check the latest 10 commits so none of the commits are missing if they are approved without a lot of time between them.

The test-reporter will be upgraded to comply with these changes.

javiercasares · 2024-03-16T18:02:34Z

FYI: WordPress/phpunit-test-reporter#109

- add the label - test dir is a dir (a tmp one) - no cert validation by default - simple WP by default

- support for labels - read latest commits

javiercasares · 2024-03-17T14:21:06Z

The code is not very beauty, but it should work...

this is a first step to include some documentation inside the project, and not only in the handbook page.

.env.default

docs/1-setup.md

functions.php

prepare.php

report.php

prepare.php

Co-authored-by: Matthias Pfefferle <[email protected]>

- Descriptions in 80 chars columns - Some Yoda fixes - Fix the "tab" vs "space" - Check some "comments" that should not be there (they were because some errors on the developing era)

- Better comments / PHPDoc - Improved code quality (do the same, in a better way) - reduce duplicated code (that's something we didn't care at Cloudfest) - updated the Runner Compatibity with PHP 7.2 (as WordPress is)

- Improved readability - default .env updated from source - other fixes

- better documentation - details on each value at the configuration file - scripts so you can run (and update) the software automatically.

javiercasares · 2024-09-29T07:30:00Z

I applied the suggestion @pfefferle did on WordCamp Europe, and reduce a lot of duplicated code that we couldn't do at the Cloudfest Hackathon, mostly on the parts when testing multiple PHP versions.

I did some improvements on the documentation, with some scripts so the software (from the main branch) is updated and maintained automatically.

I've been doing some tests in my local machine, and on my production server (where I was doing the former tests) and looks like everything is working.

On my side, I think everything is in order, and we can approve this and put it available for everyone.

benniledl · 2024-10-02T12:10:20Z

Running tests in parallel?

Running the PHPUnit test suite across 7 different PHP versions is taking a long time in my environment. Is there a way to run the tests in parallel to speed things up?

Preparing a new environment for every version necessary?

Also, is it necessary to create a completely new environment for each PHP version? So far, I've been running the prepare.php file for the first PHP version, running the tests then report.php then using a script to search and replace the PHP executable in the configuration files before rerunning the tests and reporting again. This approach seemed to work.

Copying the environment + Running in parallel

I could see that using the same environment for multiple php versions in parallel could cause problems, is it maybe possible to just copy one environment to a different directory and then replace the php version in the config? Since preparing an environment also takes a long time for me.

WesRapyd · 2025-03-13T09:45:38Z

What is the status of this pull ???

kittenkamala · 2025-08-07T21:37:53Z

I would like to have someone with access to hosting envs test this before we merge if possible

desrosj · 2025-08-13T13:28:08Z

I'm going to put some time towards getting this tested and merged over the course of the next few weeks.

At Bluehost, we currently have a fork that we used to make changes directly and get our tests back up and running. However, it's a bit of a pain to maintain those small differences while keeping the fork in sync with changes here. Some of the changes actually unintentionally accomplish multi-php but just through the GitHub Action.

This PR is quite a bear, though. I am going to try and pull all of the unrelated documentation changes into a separate pull request, and all other unrelated functional changes into another in order to make this more focused.

kittenkamala · 2025-08-13T19:51:07Z

I'm going to put some time towards getting this tested and merged over the course of the next few weeks.

At Bluehost, we currently have a fork that we used to make changes directly and get our tests back up and running. However, it's a bit of a pain to maintain those small differences while keeping the fork in sync with changes here. Some of the changes actually unintentionally accomplish multi-php but just through the GitHub Action.

This PR is quite a bear, though. I am going to try and pull all of the unrelated documentation changes into a separate pull request, and all other unrelated functional changes into another in order to make this more focused.

Thank you so much, @desrosj !!

Co-Authored-By: Javier Casares <[email protected]>

This wraps all string values in double quotes (`"`) to prevent word splitting. This is being split off from #212 since this specific change was unrelated to accomplishing the goal of that pull request. This also applies the change for all variables expecting a string type instead of just some. Co-Authored-By: Javier Casares <[email protected]>

desrosj

I've been extracting out some of the changes in this pull request that are unrelated to achieving multi PHP/multi environment/multi commit testing and tracking.

There are some merge conflicts that need to be resolved after (and if) #262, #256, and #255 are merged, but for now I've left some feedback on the parts specific to the goals of the PR.

desrosj · 2024-09-04T18:42:28Z

docs/1-setup.md

+
+`8.0+/bin/php8.0,8.1+/bin/php8.1`
+
+Use as much versions as you want, but it will take more time. The idea is to put all the versions offered to users.


Suggested change

Use as much versions as you want, but it will take more time. The idea is to put all the versions offered to users.

Use as many versions as you'd like, but keep in mind that it will take more time. Ideally all versions offered to users are tested.

desrosj · 2025-08-12T19:52:43Z

report.php

+	$WPT_PREPARE_DIR_MULTI = $WPT_PREPARE_DIR . '-' . crc32( $php_multi['version'] );
+	$WPT_TEST_DIR_MULTI = $WPT_TEST_DIR . '-' . crc32( $php_multi['version'] );


Is there a reason why the version needs to be obfuscated? $php_multi['version] should only be set to 8.1, 8.0, etc. Including the version in the directory name in plain text could be very helpful for debugging purposes.

What about something like this instead?

Suggested change

$WPT_PREPARE_DIR_MULTI = $WPT_PREPARE_DIR . '-' . crc32( $php_multi['version'] );

$WPT_TEST_DIR_MULTI = $WPT_TEST_DIR . '-' . crc32( $php_multi['version'] );

$WPT_PREPARE_DIR_MULTI = $WPT_PREPARE_DIR . '-' . str_replace( '.', '-', $php_multi['version'] );

$WPT_TEST_DIR_MULTI = $WPT_TEST_DIR . '-' . str_replace( '.', '-', $php_multi['version'] );

It occurs multiple times, so I think a helper function in functions.php may be good.

function directory_name_from_php_version( $version ) { return str_replace( '.', '-', $version] ); }

desrosj · 2025-08-12T20:00:37Z

.env.default

 # (Optionally) define the PHP executable to be called
 export WPT_PHP_EXECUTABLE=${WPT_PHP_EXECUTABLE-php}

+# (Optionally) array of versions
+# like: "8.1=/bin/php8.1;8.2=/bin/php8.2;8.3=/bin/php8.3"
+export WPT_PHP_EXECUTABLE_MULTI=""
+


Is a new environment variable necessary? Could WPT_PHP_EXECUTABLE be made to accept single or array values instead?

The default can still remain as php, and the code below responsible for parsing the value can check for the presence of = and/or ; characters to process as an array.

desrosj · 2025-08-26T06:55:04Z

cleanup.php

-$WPT_SSH_OPTIONS     = trim( getenv( 'WPT_SSH_OPTIONS' ) ) ? : '-o StrictHostKeyChecking=no';
-$WPT_TEST_DIR        = trim( getenv( 'WPT_TEST_DIR' ) );
-$WPT_RM_TEST_DIR_CMD = trim( getenv( 'WPT_RM_TEST_DIR_CMD' ) ) ? : 'rm -r ' . $WPT_TEST_DIR;
+$WPT_PREPARE_DIR          = trim( getenv( 'WPT_PREPARE_DIR' ) ) ? : '/tmp/wp-test-runner';


Applying these new, default values has been extracted out into #262.

desrosj · 2025-08-26T07:01:04Z

commits.json

@@ -0,0 +1,5 @@
+{


It's not quite clear what each of these represent based on their names. Have the executed_commits been successfully tested and reported? If the runner fails to be configured and run the tests is it included in that list?

Are pending_commits in a queue to be tested? Something else? Are testing_commit being tested right now? Will this never be more than one?

Also, not sure if this file is here for testing purposes, but it should probably be added to the .gitignore file.

desrosj · 2025-08-26T07:02:53Z

prepare.php

+		$commit_sha = null;
+		$commits_file = __DIR__ . '/commits.json';
+
+		if ( file_exists( $commits_file ) ) {


If the file does not exist and $WPT_COMMITS, the runner should copy the example file to ensure the same hashes are not retested.

desrosj · 2025-08-26T07:08:13Z

.env.default

+
+# Check all commits
+# 0 = latest
+# 1 = all


All is a bit misleading here because the API caps the results per page at 100. The default is 30.

I would change this to an on or off value. If it's on, the runner will query for the last 30 commits and add them to the queue if they have not yet been tested. If it's off, then only the most recent commit when the runner starts should be tested.

Regardless of this value, though, I think that the commits.json file should still be used to track what has been tested. This will help avoid re-running the suite for a commit that has already been tested and reported.

javiercasares added 6 commits March 16, 2024 17:16

environment for multi-php

775ce38

prepare for multi-php

80b179a

test for multi-php

ee4ac7b

cleanup for multi-php

78c5e61

report for multi-php

331ecc3

temporary fix

ef8a5de

javiercasares mentioned this pull request Mar 16, 2024

Support multiple PHP versions, improve display WordPress/phpunit-test-reporter#109

Merged

4 tasks

javiercasares self-assigned this Mar 16, 2024

javiercasares added 8 commits March 17, 2024 08:15

some changes

58034b0

- add the label - test dir is a dir (a tmp one) - no cert validation by default - simple WP by default

env file with commints checking

9205042

possible future files

8b49cb2

some changes

09bf9a6

- support for labels - read latest commits

testing version

fea79f5

fixes

0878108

reorder the commits to run

109f52d

move the commit to the executed part

07e45fc

javiercasares marked this pull request as ready for review March 17, 2024 14:21

javiercasares changed the title ~~Multi PHP~~ Multi PHP + Multi Environment + Check all commits Mar 17, 2024

This was linked to issues Mar 17, 2024

Update Runner to Support Multiple Report Submissions From Hosts #103

Open

Process all commits #205

Open

javiercasares added 3 commits March 17, 2024 17:04

checks that there are not duplicated tests to run

d9106c9

Merge branch 'WordPress:master' into multi-php

5702840

add some documentation

2e3b29d

this is a first step to include some documentation inside the project, and not only in the handbook page.

pfefferle reviewed Jun 13, 2024

View reviewed changes

This was referenced Sep 25, 2024

Configure GitHub Actions test matrix #235

Closed

Add multi-php testing to the test matrix. newfold-labs/wp-test-runner-bluehost#1

Merged

javiercasares and others added 2 commits September 28, 2024 09:53

Update .env.default

d3eedad

Co-authored-by: Matthias Pfefferle <[email protected]>

Update docs/1-setup.md

64d44e8

Co-authored-by: Matthias Pfefferle <[email protected]>

javiercasares added 7 commits September 28, 2024 12:04

TLS validation active by default

770200c

Yoda and some organization and fixes

805d78d

- Descriptions in 80 chars columns - Some Yoda fixes - Fix the "tab" vs "space" - Check some "comments" that should not be there (they were because some errors on the developing era)

Improving and normalizing code.

070c9d5

- Better comments / PHPDoc - Improved code quality (do the same, in a better way) - reduce duplicated code (that's something we didn't care at Cloudfest) - updated the Runner Compatibity with PHP 7.2 (as WordPress is)

80 columns

234a0e3

readme changes

398b809

- Improved readability - default .env updated from source - other fixes

documentation will be on the readme

e1abdad

Documentation

592e6c7

- better documentation - details on each value at the configuration file - scripts so you can run (and update) the software automatically.

javiercasares requested review from Crixu, desrosj and pfefferle September 29, 2024 07:01

kittenkamala added the ContributorDay label May 7, 2025

Crixu approved these changes May 8, 2025

View reviewed changes

kittenkamala added needs review needs testing labels Aug 7, 2025

kittenkamala approved these changes Aug 7, 2025

View reviewed changes

desrosj added a commit that referenced this pull request Aug 19, 2025

Extract unrelated documentation changes from #212

b3a76f1

Co-Authored-By: Javier Casares <[email protected]>

This was referenced Aug 19, 2025

Use double quotes to prevent word splitting #254

Merged

Change the default value of WPT_FLAVOR #255

Merged

Documentation updates #256

Draft

kittenkamala mentioned this pull request Aug 26, 2025

Update Runner to Support Multiple Report Submissions From Hosts #103

Open

desrosj mentioned this pull request Aug 26, 2025

Create helper function for parsing runner configuration environment variables #262

Merged

desrosj requested changes Aug 26, 2025

View reviewed changes

kittenkamala mentioned this pull request Sep 17, 2025

cicd pipeline for testing code changes #274

Open


		`8.0+/bin/php8.0,8.1+/bin/php8.1`

		Use as much versions as you want, but it will take more time. The idea is to put all the versions offered to users.

	Use as much versions as you want, but it will take more time. The idea is to put all the versions offered to users.
	Use as many versions as you'd like, but keep in mind that it will take more time. Ideally all versions offered to users are tested.

		$WPT_PREPARE_DIR_MULTI = $WPT_PREPARE_DIR . '-' . crc32( $php_multi['version'] );
		$WPT_TEST_DIR_MULTI = $WPT_TEST_DIR . '-' . crc32( $php_multi['version'] );

Multi PHP + Multi Environment + Check all commits #212

Are you sure you want to change the base?

Multi PHP + Multi Environment + Check all commits #212

Uh oh!

Conversation

javiercasares commented Mar 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

javiercasares commented Mar 16, 2024

Uh oh!

javiercasares commented Mar 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javiercasares commented Sep 29, 2024

Uh oh!

benniledl commented Oct 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Running tests in parallel?

Preparing a new environment for every version necessary?

Copying the environment + Running in parallel

Uh oh!

WesRapyd commented Mar 13, 2025

Uh oh!

kittenkamala commented Aug 7, 2025

Uh oh!

desrosj commented Aug 13, 2025

Uh oh!

kittenkamala commented Aug 13, 2025

Uh oh!

desrosj left a comment

Choose a reason for hiding this comment

Uh oh!

desrosj Sep 4, 2024

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

desrosj Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

javiercasares commented Mar 16, 2024 •

edited

Loading

benniledl commented Oct 2, 2024 •

edited

Loading