Feature/UI tests by NikolasFunction · Pull Request #342 · jenkinsci/junit-plugin

NikolasFunction · 2022-01-20T22:28:23Z

Added ui tests for the junit plugin.
https://issues.jenkins.io/browse/JENKINS-67275

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

…ses)

# Conflicts: # pom.xml

jtnord · 2022-02-17T23:08:55Z

What is the reasoning to do this here?
UI testing will be brittle with changes in Jenkins core so the Jenkins tools BOM will likely be failing regularly for weeklies, and or versions held back from the BOM for some lines.

jglick · 2022-02-17T23:30:29Z

Note that this will complicate infrastructure somewhat. jenkinsci/plugin-compat-tester#249

uhafner · 2022-02-18T07:59:24Z

What is the reasoning to do this here?

I think UI tests belong to a plugin like unit and integration tests (the idea is from @olivergondza, see jenkinsci/acceptance-test-harness#556 for details). After contributing to ATH for several years now I made the observation that nobody is actually looking at UI tests that are located somewhere else. So it makes sense to have them in the plugin. Then we see if a new feature breaks an existing feature (on the UI side) or if an upgrade to a more recent core version breaks the UI.

Additionally, JUnit plugin uses integration tests with HTMLUnit which does not support modern JS frameworks (see #272, #286) . So it makes sense to remove those tests in favor of real UI tests.

UI testing will be brittle with changes in Jenkins core so the Jenkins tools BOM will likely be failing regularly for weeklies, and or versions held back from the BOM for some lines.

Why should that be a problem? If the tests will break then something in the UI is wrong. That means we should fix it.

uhafner · 2022-02-18T08:04:34Z

Note that this will complicate infrastructure somewhat. jenkinsci/plugin-compat-tester#249

I haven't looked in detail at those changes but I would assume that we would have two alternatives: a single module plugin and a multi-module plugin. So the ifWarnings blocks should be actually ifMultiModule blocks.

Would it be possible to just use the plugin module in those tests and ignore the other ones?

timja · 2022-02-18T08:14:09Z

As Ulli said the critical reason for this is modern JS frameworks don't work with HTMLUnit 😢

jglick · 2022-02-18T15:11:02Z

Would it be possible to just use the plugin module in those tests and ignore the other ones?

Yes. But this means that PCT (as run from bom) will not verify that changes to core and plugin deps do not break the UI of the junit plugin. Basically the trick of including ATH-based tests inside a plugin repo is not currently supported by other tooling.

timja · 2022-02-18T16:13:41Z

Basically the trick of including ATH-based tests inside a plugin repo is not currently supported by other tooling.

Probably belongs in ATH?

uhafner

See comments in #355 (review).

GitHub is unable to render this PR in my browser.

Main review comments

Use the correct AssertJ assertions
Don't duplicate login in page objects (2 constructors)

It would be also helpful if you can include one additional SmokeTest that exercises the most important UI elements in one row so that this test can be used during PR builds. (The other UI tests can be then used only as additional tests).

jtnord

I forgot to finish my review :(

jtnord · 2022-02-17T23:10:52Z

@@ -190,15 +189,5 @@ public String getDisplayName() {
            return Collections.unmodifiableSet(context);
        }

-        public FormValidation doCheckHealthScaleFactor(@QueryParameter double value) {


Why is this removed if the change is only adding UI tets?

jtnord · 2022-02-17T23:11:55Z

-Test\ Description=Testbeschreibung
-Test\ Duration=Testdauer
-Test\ Result=Testergebnis
+# The MIT License


Something seems odd here...

jtnord · 2022-02-17T23:12:26Z

-
-Error\ Details=Fehlerdetails
-Stack\ Trace=Stacktrace
+# The MIT License


jtnord · 2022-02-17T23:12:39Z

-Fail=Fehlgeschlagen
-Skip=Ausgelassen
-Total=Summe
+# The MIT License


uhafner · 2022-04-23T11:27:53Z

@NikolasFunction @Tjuri Are you still planning to update this PR? The longer you are waiting the more merge conflicts will occur...

NikolasFunction and others added 30 commits December 11, 2021 20:26

Added ui tests top hierarchy

0b99a5a

Added UI tests structure

f74b5eb

tests are running now

21feff3

edit pom

f9fa8bf

fist progress ui tests

9a99cbb

build summary tests work

887e421

Added page object files plus dummy test methods (both for a few useca…

2f2fef0

…ses)

Added two TODOs

1f077e8

Fixed a bug, organized imports

3f5f942

Added error stacktrace and message, added title failures

221c1a0

merge stashed changes

8539c95

added test utils class

3e5d283

fixed test in build summary

db36b91

added todo for check extraction of test result text

8054bbb

Fixes in junitBuildDetail

9958c4a

Added tests, fixed a few things, deleted unused README

c226c3f

Merge remote-tracking branch 'origin/develop' into develop

0db2210

Replaced hamcrest with assertj, fixed copy bug

4d65bfd

Fixed testing bug

5bc7a8b

Fixed Testing Bugs

c373ba3

Added build configuration UI tests. In progress.

5580459

added junit publisher tests. in progress

25a1b69

added test for retain long standard output error

f146352

Moved files, removed hamcrest testing, fixed openPage, fixed test

271bf79

Moved files, removed hamcrest testing, fixed openPage, fixed test

104debe

added test for direct url navigation to test report

587624b

Refactored BuildDetails, fixed tests, implemented tests

082d67c

Implemented tests, fixed tests, added testfiles

ede0865

cleaned up tests.

995a01c

rename and restructure tests

f111cef

uhafner added 9 commits February 11, 2022 13:36

Move ui-tests workflow to correct folder.

40bb597

Update selection of links due to removed CSS classes in jenkinsci#345.

8cf406a

Fix some paths due to new table layout changes.

0e2b337

Merge remote-tracking branch 'upstream/master' into ui-tests

d9d506e

# Conflicts: # pom.xml

Merge with master.

ae69d52

Use JDK 11 in build.

c085a11

Update steps to latest versions.

4a6c035

Fix Jenkinsfile configuration for JDK11.

0f8f584

Add caching to Java setup step.

ede4252

uhafner added the test A PR that adds to testing - used by Release Drafter label Feb 16, 2022

uhafner self-requested a review February 16, 2022 18:29

uhafner added 2 commits February 16, 2022 20:08

Fix broken yml file.

f4ed8b7

Add missing checkout.

ba363f9

Use docker agent in CI.

ac49da6

uhafner mentioned this pull request Feb 20, 2022

UI tests of #342 without structural folder changes #355

Closed

Start test ProjectSummaryTest.

dad7865

uhafner requested changes Feb 21, 2022

View reviewed changes

jtnord reviewed Feb 21, 2022

View reviewed changes

Tjuri added 2 commits March 26, 2022 19:22

fixed seperation of text title and text for failures in build summary

d91569f

Fixed assertion in multiple test classes. Few still missing

0c85320

uhafner mentioned this pull request Apr 24, 2022

Bump versions of JS UI plugins to latest version jenkinsci/bom#571

Merged

6 tasks

basil mentioned this pull request Apr 30, 2022

Update build toolchain on 5.0.2 line jenkinsci/echarts-api-plugin#228

Closed

Conversation

NikolasFunction commented Jan 20, 2022 • edited by jglick Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtnord commented Feb 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jglick commented Feb 17, 2022

Uh oh!

uhafner commented Feb 18, 2022

Uh oh!

uhafner commented Feb 18, 2022

Uh oh!

timja commented Feb 18, 2022

Uh oh!

jglick commented Feb 18, 2022

Uh oh!

timja commented Feb 18, 2022

Uh oh!

uhafner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtnord left a comment

Choose a reason for hiding this comment

Uh oh!

jtnord Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

jtnord Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

jtnord Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

jtnord Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

uhafner commented Apr 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

NikolasFunction commented Jan 20, 2022 •

edited by jglick

Loading

jtnord commented Feb 17, 2022 •

edited

Loading

uhafner left a comment •

edited

Loading