Order for DependsOnGroups has changed after TestNg 7.4.0

[martinaldrin]

TestNG Version

7.5 not working
7.4 works

Expected behavior

s3
s1
s2

Actual behavior

s1
s2
s3

Is the issue reproducible on runner?

• Maven
• Eclipse
• IntelliJ

Test case sample

Here is an example for BeforeSuite, but the same problem exists for all configuration methods. The same issue also for dependsOnMethod

    @BeforeSuite(groups = { "g1" })
    public void s3() {
        System.out.println("s3");
    }

    @BeforeSuite(groups = { "g2" }, dependsOnGroups = "g1")
    public void s2() {
        System.out.println("s2");
    }

    @BeforeSuite(groups = { "g2" }, dependsOnGroups = "g1")
    public void s1() {
        System.out.println("s1");
    }

    @Test()
    public void test3() {
        System.out.println("test3");
    }

I have tried to find the commit where the fault was introduced, but my IDE did not like to switch gradle version and I'm new at gradle.

[martinaldrin]

After some more investigation I found the commit

#2503

[krmahadevan]

@martinaldrin - I didnt quite understand the issue here from the sample that you shared. How is the test being run? What does the suite xml file look like ?

Also can you please add the expected and the actual output as well ?

[martinaldrin]

The problems is that the sorting on depends on doesn't not work on master.
Regardless if I add depends on or not the methods are executed in alphabetical order. DependsOn only works for test annotation, not for any of the configuration methods

[krmahadevan]

@martinaldrin - The configurations are defined as being part of groups. So dependsOnGroups would work only when the group selection is enabled from the suite xml file.

So if you are running from within the IDE, then you are just running the test class without any groups. And hence the dependsOnGroups attribute is going to be ignored.

[martinaldrin]

But that is a non backward compatible change. similar to what we raised for beforeGroups.
Why should the ordering be ignored just because I don't add a filter?

[Elisedlund-ericsson]

@krmahadevan the issue is not just dependsOnGroups. its also fails for dependsOnMethods

like the following should fail:

@BeforeSuite
public void s3() {
    System.out.println("s3");
}

@BeforeSuite(dependsOnMethods= "s3")
public void s2() {
    System.out.println("s2");
}

@BeforeSuite(dependsOnMethods= "s2")
public void s1() {
    System.out.println("s1");
}

Should result in execution order: s3 s2 s1 but does not on since the commit #2503

basically dependsOnMethods and dependsOnGroups does nothing for all configuration methods right now, It has worked in all released TestNG versions

replacing "BeforeSuite"(or any other config method) with "Test" seems to work so it only an issue for configuration methods.

[krmahadevan]

But that is a non backward compatible change. similar to what we raised for beforeGroups.

The below test runs fine when executed against 6.14.3 and which uses the test class sample that you shared in this defect. So not sure I quite understand the backward compatibility being broken here.

import java.io.File;
import static org.assertj.core.api.Assertions.assertThat;
import org.testng.TestNG;
import org.testng.annotations.Test;

public class IssueTest {

  @Test
  public void testMethod() {
    TestNG testng = new TestNG();
    testng.setTestClasses(new Class[]{SampleTestClass.class});
    testng.run();
    File file = new File(
        TestNG.class.getProtectionDomain().getCodeSource().getLocation().getFile());
    assertThat(file.getName()).isEqualTo("testng-6.14.3.jar");
    String[] expected = new String[]{"s3", "s1", "s2", "test3"};
    assertThat(SampleTestClass.logs).containsExactly(expected);
  }
}

[martinaldrin]

on master it will run in following order "s1, s2, s32 instead of "s3, s1, s2"
6.14 and up 7.4 works, the fault is introduced in #2503

[martinaldrin]

Sorry if we confused you. the problem seems to only affect DependsOnGroups, not DependsOnMethod.
and the problem seems to be related to the added if statement in MethodHelper

 if (XmlTest.isGroupBasedExecution(xmlTest)) {
......
}

[krmahadevan]

@martinaldrin -

on master it will run in following order "s1, s2, s32 instead of "s3, s1, s2"

Yes, because when you run from within the IDE, you basically are not using groups.

The javadocs for dependsOnGroups states the below

  /**
   * The list of groups this method depends on. Every method member of one of these groups is
   * guaranteed to have been invoked before this method. Furthermore, if any of these methods was
   * not a SUCCESS, this test method will not be run and will be flagged as a SKIP.
   *
   * @return the value
   */

So when we are running from within the IDE, group filtering is not getting applied.

@cbeust - Am back with the same question again. I have always been under the assumption that anything that involves or is associated with groups is meant to be applicable only when running with group filtering enabled. From within IDE, thats not the case.

What should be the behaviour for dependsOnGroups attribute, when no group filtering is enabled. TestNG seems to still honour this attribute even when no groups are chosen until now (7.4.0) which IMO is a bug that has always been there. But I would like to understand what is the expected behaviour here so that we can move forward with this.

@juherr - WDYT ?

[martinaldrin]

Hi @krmahadevan
In my view, suite xml group filtering is just for include/exclude groups, in similar way as method include/exclude works.

The javadoc does not say anything about filtering in suite xml is required.

And in this case, you did not skip them, you just ignored the sorting of the configuration methods that have dependsOnGroups.

[krmahadevan]

@martinaldrin -

In my view, suite xml group filtering is just for include/exclude groups, in similar way as method include/exclude works.
The javadoc does not say anything about filtering in suite xml is required.

Just to ensure that we sort of the difference of opinions is why I am asking for inputs from @cbeust who has built TestNG so that the confusions are sorted out.

The javadoc does not say anything about filtering in suite xml is required.

The way i see this is that it is applicable only when groups are enabled. including or excluding groups is exactly that.

Lets say there are 3 methods as below :

@Test public void foo() {}

@Test (groups="a") public void bar() {}

@Test(groups="b") public void fooBar() {}

The way that I am seeing this is that,

• if a user does not mention the <groups> tag at all and just runs this test class, then all the 3 methods should be executed
• if a user uses includes or excludes tags, in the suite file, then he/she should be able to select only bar and foobar but foo() should not be chosen, because its NOT part of any groups and to me there's a distinction between being part of a group and not being part of any group. Not being part of any group IMO is not equivalent to I am part of all groups.

This is why I need this to be clarified once and for all and we add tests so that this discrepancy in terms of understanding the behaviour goes away.

There were no tests which iterates the behaviour that you are suggesting, else my earlier PR would have broken some changes. Which is what is making me wonder, are we looking at undocumented behaviour ?