The profile java25-test-classpath is automatically enabled if the JVM used to initialize Maven is Java 25

[jamezp]

The profile java25-test-classpath is enabled if the JVM that launched Maven is 25 or greater. This adds additional class path entries which may not be usable if the maven-surefire-plugin or maven-failsafe-plugin sets the jvm setting to an older version.

The profile looks like:

<profile>
    <id>java25-test-classpath</id>
    <activation>
        <jdk>[25,26)</jdk>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-test</id>
                        <configuration>
                            <classesDirectory>${project.build.outputDirectory}/META-INF/versions/25
                            </classesDirectory>
                            <additionalClasspathElements>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/24
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/23
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/22
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/21
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/20
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/19
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/18
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/17
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/16
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/15
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/14
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/13
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/12
                                </additionalClasspathElement>
                                <additionalClasspathElement>${project.build.outputDirectory}
                                </additionalClasspathElement>
                            </additionalClasspathElements>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

This works when you use the build-javaXX pattern because that profile overrides these settings, example for Java 17.

<profile>
    <id>java17-test</id>
    <activation>
        <jdk>[18,)</jdk>
        <property>
            <name>java17.home</name>
        </property>
        <file>
            <exists>${basedir}/build-test-java17</exists>
        </file>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <executions>
                    <execution>
                        <id>java17-test</id>
                        <phase>test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <jvm>${java17.home}/bin/java</jvm>
                            <reportsDirectory>${project.build.directory}/surefire-reports/java17</reportsDirectory>
                            <classesDirectory>${project.build.directory}/classes/META-INF/versions/17
                            </classesDirectory>
                            <additionalClasspathElements>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/16
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/15
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/14
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/13
                                </additionalClasspathElement>
                                <additionalClasspathElement>
                                    ${project.build.directory}/classes/META-INF/versions/12
                                </additionalClasspathElement>
                                <additionalClasspathElement>${project.build.outputDirectory}
                                </additionalClasspathElement>
                            </additionalClasspathElements>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

An example PR that's failing due to this jberet/jsr352#721.

I'm not sure what the best option is to deal with this. A workaround is to either disable the profile or override the classesDirectory configuration option. However, then we lose testing the newly compiled source.

[dmlloyd]

If you're using the JBoss Parent POM, then you should not be using the <jvm> option on the default surefire config. The defined multi-JVM testing mechanism should be used instead; that's what it exists for.

[jamezp]

The problem is for multi-module projects, you have to add the marker files to very module you want to test with. I assume that is what you mean here is adding the build-test-javaXX marker files to enable the javaXX-test profile.

It also potentially enables much longer testing for large projects if you have the javaXX.home properties set in something like your settings.xml. That can be worked around though as that is a local user problem :)

[dmlloyd]

Yes, the usability of the marker files isn't great, but they are the only non-inheritable profile activation mechanism that exists in Maven AFAIK.

The point of the mechanism is to test with many JDKs. In general, tests should be fast and efficient. If you do not want to test many JDKs, you can either not create the marker files, or skip tests altogether, or remove the javaXX.home properties from your settings (or rehome them under a profile that you manually activate from the command line, perhaps).

If this issue is a very oblique way of requesting a way to avoid multi-release testing, we can definitely try to come up with a switchable mechanism to do that, but as for the text of this issue as it stands, I'd say "not a bug"/"working as intended" or something along those lines.

[jamezp]

I definitely don't think it's a bug. I mostly added this as something to keep in mind in case we can think of another way to do this.

In some multi-module projects I've started using a property test.java.home to set the jvm value and I use it in a GitHub workflow like:

- name: Build and Test on ${{ matrix.os }} - Java 25
  run: ${{ env.MVN_CMD }} clean install
- name: Build and Test on ${{ matrix.os }} - Java 26
  run: ${{ env.MVN_CMD }} verify '-Dtest.java.home=${{env.JAVA_HOME_26_X64}}${{env.JAVA_HOME_26_ARM64}}' '-Dtest.report.dir=java26'
- name: Build and Test on ${{ matrix.os }} - Java 21
  run: ${{ env.MVN_CMD }} verify '-Dtest.java.home=${{env.JAVA_HOME_21_X64}}${{env.JAVA_HOME_21_ARM64}}' '-Dtest.report.dir=java21'
- name: Build and Test on ${{ matrix.os }} - Java 17
  run: ${{ env.MVN_CMD }} verify '-Dtest.java.home=${{env.JAVA_HOME_17_X64}}${{env.JAVA_HOME_17_ARM64}}' '-Dtest.report.dir=java17'

To get this to work on the workflow it's easy enough to disable the profile with -P-javaXX-test-classpath, which isn't a huge deal.

I just personally find it easier to test this way vs creating multiple marker files. In the case where I saw this, there are 45 modules which is a lot of files :)

[dmlloyd]

You're creating quite a bit more CI churn that way (repeating cost); it seems like a lot of hoops to jump through to avoid creating marker files (one time cost), which can probably be shell scripted pretty easily.

[jamezp]

Yeah, I thought there was another reason I'd done it like that, but maybe not. The more I think about it though, the more I realize you're right that the marker files are likely the best solution.

I remember one concern I had was making sure any new module would also be tested. However, that's something that could pretty easily be caught in a review. It's also probably a good idea for projects to use some kind of standard option for this as it makes it easier to test for projects like this. For example, those defined in the build.yml.

[dmlloyd]

Even if you forget the marker files, you're still going to execute the default test execution, so you'll still have some degree of coverage at least.

[jamezp]

I remember what the issue is now. It doesn't work if there are different executions defined. That's why I started using this pattern for cases where tests are run in different executions. As an example:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <executions>
        <execution>
            <id>default-test</id>
            <configuration>
                <systemPropertyVariables>
                    <jboss.home>${project.build.testOutputDirectory}/fake-modules</jboss.home>
                </systemPropertyVariables>
                <groups>system-property</groups>
            </configuration>
        </execution>
        <execution>
            <id>env-var</id>
            <phase>test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
                <environmentVariables>
                    <JBOSS_HOME>${project.build.testOutputDirectory}/fake-modules</JBOSS_HOME>
                </environmentVariables>
                <groups>env-var</groups>
            </configuration>
        </execution>
    </executions>
</plugin>

That fails with the marker files because it attempts to execute all the tests:

[INFO] --- surefire:3.5.5:test (java17-test) @ wildfly-junit-api ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.wildfly.testing.junit.JBossHomeParameterTestCase
[ERROR] Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.044 s <<< FAILURE! -- in org.wildfly.testing.junit.JBossHomeParameterTestCase
[ERROR] org.wildfly.testing.junit.JBossHomeParameterTestCase.fileSet(File) -- Time elapsed: 0.017 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Could not resolve the jboss.home system property or JBOSS_HOME environment variable.
	at org.wildfly.testing.junit.condition.JBossHomeParameterResolver.resolveParameter(JBossHomeParameterResolver.java:42)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[ERROR] org.wildfly.testing.junit.JBossHomeParameterTestCase.pathSet(Path) -- Time elapsed: 0.005 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Could not resolve the jboss.home system property or JBOSS_HOME environment variable.
	at org.wildfly.testing.junit.condition.JBossHomeParameterResolver.resolveParameter(JBossHomeParameterResolver.java:42)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[ERROR] org.wildfly.testing.junit.JBossHomeParameterTestCase.stringSet(String) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Could not resolve the jboss.home system property or JBOSS_HOME environment variable.
	at org.wildfly.testing.junit.condition.JBossHomeParameterResolver.resolveParameter(JBossHomeParameterResolver.java:42)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[INFO] Running org.wildfly.testing.junit.RequiresModuleTestCase
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.wildfly.testing.junit.RequiresModuleTestCase
[ERROR] org.wildfly.testing.junit.RequiresModuleTestCase -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Could not resolve the jboss.home system property or JBOSS_HOME environment variable.
	at org.wildfly.testing.junit.condition.JBossHomeParameterResolver.resolveParameter(JBossHomeParameterResolver.java:42)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[dmlloyd]

Why do you need more special executions in this case? Seems like an X/Y issue; could you solve this using JUnit parameterization somehow maybe?

[jamezp]

In this specific scenario I likely could. However, not in every case. I was working on changing this and realized it was one of the reasons I did it like this for other projects.