ci(WPB-21987): shard playwright tests across multiple runners

[markbrockhoff]

WPB-21987 [WEB/QA] POC - Use sharding to run e2e tests in parallel in CI

Pull Request

Summary

• Build and run Testservice as part of CI
• Shard playwright tests across multiple runners
• Include regression tests when running tests
• Fix script reporting test results as github comment to work with nested test suites

---

Security Checklist (required)

• External inputs are validated & sanitized on client and/or server where applicable.
• API responses are validated; unexpected shapes are handled safely (fallbacks or errors).
• No unsafe HTML is rendered; if unavoidable, sanitization is applied and documented where it happens.
• Injection risks (XSS/SQL/command) are prevented via safe APIs and/or escaping.

Standards Acknowledgement (required)

• I have read and this PR upholds our Coding Standards and Tech Radar Choices.

---

Notes for reviewers

• I updated the node setup within the pipeline to respect the .nvmrc so we'll always use the same node version in CI as we do locally

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.71%. Comparing base (20b716e) to head (4280c02).
⚠️ Report is 1 commits behind head on dev.

Additional details and impacted files

@@           Coverage Diff           @@
##              dev   #19807   +/-   ##
=======================================
  Coverage   43.71%   43.71%           
=======================================
  Files        1298     1298           
  Lines       32707    32707           
  Branches     7259     7259           
=======================================
+ Hits        14297    14298    +1     
  Misses      16693    16693           
+ Partials     1717     1716    -1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

🔗 Download Full Report Artifact

🧪 Playwright Test Summary

• ✅ Passed: 84
• ❌ Failed: 17
• ⏭ Skipped: 5
• 🔁 Flaky: 2
• 📊 Total: 108
• ⏱ Total Runtime: 502.2s (~ 8 min 22 sec)

Failed Tests:

❌ I want to see collapsed view when app is narrow (tags: TC-48, regression)

Location: specs/Accessibility/Accessibility.spec.ts:128
Duration: 26910ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="do-history-confirm"]')


   at pageManager/webapp/pages/infoHistory.page.ts:42

  40 |
  41 |   async clickConfirmButton() {
> 42 |     await this.continueButton.click();
     |                               ^
  43 |   }
  44 | }
  45 |
    at HistoryInfoPage.clickConfirmButton (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/infoHistory.page.ts:42:31)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Accessibility/Accessibility.spec.ts:135:31

❌ I should not lose a drafted message when switching between conversations in collapsed view (tags: TC-51, regression)

Location: specs/Accessibility/Accessibility.spec.ts:141
Duration: 27093ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="do-history-confirm"]')


   at pageManager/webapp/pages/infoHistory.page.ts:42

  40 |
  41 |   async clickConfirmButton() {
> 42 |     await this.continueButton.click();
     |                               ^
  43 |   }
  44 | }
  45 |
    at HistoryInfoPage.clickConfirmButton (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/infoHistory.page.ts:42:31)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Accessibility/Accessibility.spec.ts:150:33

❌ I should not be able to change email of user managed by SCIM (tags: TC-60, regression)

Location: specs/AccountSettingsSpecs/accountSettings.spec.ts:87
Duration: 88581ms

Errors:

TimeoutError: locator.waitFor: Timeout 60000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="go-preferences"]') to be visible


   at pageManager/webapp/components/conversationSidebar.component.ts:64

  62 |
  63 |   async isPageLoaded() {
> 64 |     await this.preferencesButton.waitFor({state: 'visible', timeout: this.pageLoadingTimeout});
     |                                  ^
  65 |   }
  66 |
  67 |   async clickArchive() {
    at ConversationSidebar.isPageLoaded (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/components/conversationSidebar.component.ts:64:34)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AccountSettingsSpecs/accountSettings.spec.ts:115:46

❌ Verify links to manage and create teams are shown when logged in as team owner (tags: TC-1723, regression)

Location: specs/AccountSettingsSpecs/accountSettings.spec.ts:159
Duration: 27071ms

Errors:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "https://wire-teams-dev.zinfra.io/login/"
Received: "https://wire-teams-staging.zinfra.io/login/"

  166 |
  167 |       await expect(components.conversationSidebar().manageTeamButton).toBeVisible();
> 168 |       expect(await components.conversationSidebar().manageTeamButton.getAttribute('href')).toBe(
      |                                                                                            ^
  169 |         'https://wire-teams-dev.zinfra.io/login/',
  170 |       );
  171 |     },
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AccountSettingsSpecs/accountSettings.spec.ts:168:92

❌ Web: App should not lock if I switch back to webapp tab in time (during inactivity timeout) (tags: TC-2752, TC-2753, regression)

Location: specs/AppLock/AppLock.spec.ts:60
Duration: 70401ms

Errors:

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name=\'applock-modal\'] [data-uie-name=\'applock-modal-header\']')
Expected: visible
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name=\'applock-modal\'] [data-uie-name=\'applock-modal-header\']')


  80 |         await unrelatedPage.waitForTimeout(31_000);
  81 |         await webappPageA.bringToFront();
> 82 |         await expect(modals.appLock().appLockModalHeader).toBeVisible();
     |                                                           ^
  83 |       });
  84 |     },
  85 |   );
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AppLock/AppLock.spec.ts:82:59
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AppLock/AppLock.spec.ts:77:7

❌ I want to be asked to share telemetry data when I log in (tags: TC-8780, regression)

Location: specs/Authentication/authentication.spec.ts:48
Duration: 15184ms

Errors:

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name="primary-modals-container"][aria-label=\'Consent to share user data\']').locator('[data-uie-name="status-modal-title"]')
Expected: visible
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name="primary-modals-container"][aria-label=\'Consent to share user data\']').locator('[data-uie-name="status-modal-title"]')


  56 |       await pages.login().login(user);
  57 |
> 58 |       await expect(modals.dataShareConsent().modalTitle).toBeVisible();
     |                                                          ^
  59 |     },
  60 |   );
  61 |
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Authentication/authentication.spec.ts:58:58

❌ Verify 1on1 conversation is not created on the second end after you ignore connection request (tags: TC-365, TC-369, TC-370, TC-371, regression)

Location: specs/Connections/Connections.spec.ts:36
Duration: 52555ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for getByTestId('[data-uie-name=\'primary-modals-container\']').getByTestId('do-action')


   at pageManager/webapp/modals/base.modal.ts:60

  58 |
  59 |   async clickAction() {
> 60 |     await this.actionButton.click();
     |                             ^
  61 |   }
  62 | }
  63 |
    at CancelRequestModal.clickAction (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/modals/base.modal.ts:60:29)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Connections/Connections.spec.ts:60:38
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Connections/Connections.spec.ts:58:7

❌ Account Management (tags: TC-8639, crit-flow-web)

Location: specs/CriticalFlow/accountManagement-TC-8639.spec.ts:37
Duration: 90842ms

Errors:

Test timeout of 90000ms exceeded.

❌ Team owner adds whole team to an all team chat (tags: TC-8631, crit-flow-web)

Location: specs/CriticalFlow/addMembersToChat-TC-8631.spec.ts:43
Duration: 91472ms

Errors:

Test timeout of 90000ms exceeded.

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ hasText: 'Hello from Freddie!' }).locator('[data-uie-name="message-reactions"]').locator('[data-uie-name="emoji-pill"][title="heart"]')
Expected: visible
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ hasText: 'Hello from Freddie!' }).locator('[data-uie-name="message-reactions"]').locator('[data-uie-name="emoji-pill"][title="heart"]')


  185 |       await pages.conversationList().openConversation(conversationName);
  186 |       const ownerMessage = pages.conversation().getMessage({content: `Hello from ${owner.firstName}!`});
> 187 |       await expect(pages.conversation().getReactionOnMessage(ownerMessage, 'heart')).toBeVisible();
      |                                                                                      ^
  188 |       await expect(pages.conversation().getReactionOnMessage(ownerMessage, 'joy')).toBeVisible();
  189 |
  190 |       // Member1 verifies they can see thumbs up (+1) and joy (😂) reactions on their message from owner and member2
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/addMembersToChat-TC-8631.spec.ts:187:86
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/addMembersToChat-TC-8631.spec.ts:183:5

❌ Channels Management (tags: TC-8752, crit-flow-web)

Location: specs/CriticalFlow/channelsManagement-TC-8752.spec.ts:37
Duration: 64393ms

Errors:

TimeoutError: locator.waitFor: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('#conversation-details') to be visible


   at pageManager/webapp/pages/conversationDetails.page.ts:46

  44 |
  45 |   async waitForSidebar() {
> 46 |     await this.conversationDetails.waitFor({state: 'visible'});
     |                                    ^
  47 |   }
  48 |
  49 |   async isOpen(conversationName: string) {
    at ConversationDetailsPage.waitForSidebar (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversationDetails.page.ts:46:36)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/channelsManagement-TC-8752.spec.ts:124:39
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/channelsManagement-TC-8752.spec.ts:121:3

❌ Planning group call with sending various messages during call (tags: TC-8632, crit-flow-web)

Location: specs/CriticalFlow/groupCalls-TC-8632.spec.ts:37
Duration: 40055ms

Errors:

Error: expect(received).toBeFalsy()

Received: true

  122 |       await memberCalling.unmuteSelfInFullScreen();
  123 |       await memberPageManager.waitForTimeout(250);
> 124 |       expect(await memberCalling.isSelfUserMutedInFullScreen()).toBeFalsy();
      |                                                                 ^
  125 |     });
  126 |
  127 |     await test.step('Validation: Owner sees member is unmuted', async () => {
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/groupCalls-TC-8632.spec.ts:124:65
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/groupCalls-TC-8632.spec.ts:121:5

❌ Group Video call (tags: TC-8637, crit-flow-web)

Location: specs/CriticalFlow/groupVideoCall-TC-8637.spec.ts:39
Duration: 56470ms

Errors:

TimeoutError: locator.waitFor: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('#add-participants [data-uie-name="search-list"] [aria-label="Open profile of Delta Breitenberg"]').locator('input[type="checkbox"]')


   at pageManager/webapp/pages/conversationDetails.page.ts:70

  68 |       await userLocator.click();
  69 |       // Wait for the user to be selected (checkbox should be checked)
> 70 |       await userLocator.locator('input[type="checkbox"]').waitFor({state: 'attached'});
     |                                                           ^
  71 |     }
  72 |
  73 |     await this.page.locator(`${selectById('add-participants')} ${selectByDataAttribute('do-create')}`).click();
    at ConversationDetailsPage.addUsersToConversation (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversationDetails.page.ts:70:59)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/groupVideoCall-TC-8637.spec.ts:127:7
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/groupVideoCall-TC-8637.spec.ts:123:5

❌ New person joins team and setups up device (tags: TC-8635, crit-flow-web)

Location: specs/CriticalFlow/joinTeam-TC-8635.spec.ts:38
Duration: 80396ms

Errors:

TimeoutError: locator.waitFor: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('#add-participants [data-uie-name="search-list"] [aria-label="Open profile of Gayle Howe"]').locator('input[type="checkbox"]')


   at pageManager/webapp/pages/conversationDetails.page.ts:70

  68 |       await userLocator.click();
  69 |       // Wait for the user to be selected (checkbox should be checked)
> 70 |       await userLocator.locator('input[type="checkbox"]').waitFor({state: 'attached'});
     |                                                           ^
  71 |     }
  72 |
  73 |     await this.page.locator(`${selectById('add-participants')} ${selectByDataAttribute('do-create')}`).click();
    at ConversationDetailsPage.addUsersToConversation (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversationDetails.page.ts:70:59)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/joinTeam-TC-8635.spec.ts:125:7
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/CriticalFlow/joinTeam-TC-8635.spec.ts:114:5

❌ I can see the changed message was edited from another user (tags: TC-692, regression)

Location: specs/Edit/edit.spec.ts:175
Duration: 46291ms

Errors:

TimeoutError: locator.hover: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ has: locator('.content-message-wrapper[aria-label*="Brisa Mohr"]') })


   at pageManager/webapp/pages/conversation.page.ts:251

  249 |    */
  250 |   async openMessageOptions(message: Locator) {
> 251 |     await message.hover();
      |                   ^
  252 |     await message.getByTestId('message-actions').getByTestId('go-options').click();
  253 |     // The context menu containing the edit button is positioned globally as an overlay
  254 |     return this.page.getByRole('menu');
    at ConversationPage.openMessageOptions (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:251:19)
    at ConversationPage.editMessage (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:259:29)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Edit/edit.spec.ts:191:39

❌ Block specs (tags: TC-141, regression)

Location: specs/RegressionSpecs/block-messages.spec.ts:32
Duration: 44484ms

Errors:

TimeoutError: locator.waitFor: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="item-user"] [data-uie-name="status-username"]').first() to be visible


   at pageManager/webapp/pages/startUI.page.ts:51

  49 |
  50 |   private async clickUserFromSearchResults(username: string) {
> 51 |     await this.searchResults.first().waitFor({state: 'visible'});
     |                                      ^
  52 |
  53 |     const timeout = 30000;
  54 |     const delayBetweenAttempts = 500;
    at StartUIPage.clickUserFromSearchResults (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/startUI.page.ts:51:38)
    at StartUIPage.selectUser (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/startUI.page.ts:43:16)
    at StartUIPage.selectUsers (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/startUI.page.ts:37:7)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/RegressionSpecs/block-messages.spec.ts:73:5
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/RegressionSpecs/block-messages.spec.ts:69:3

❌ I want to see a placeholder text as quote when original message is not available anymore (tags: TC-2994, regression)

Location: specs/Reply/reply.spec.ts:60
Duration: 48124ms

Errors:

TimeoutError: locator.hover: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ hasText: 'Test' }).filter({ has: locator('.content-message-wrapper[aria-label*="Myrtie Hoeger"]') })


   at pageManager/webapp/pages/conversation.page.ts:251

  249 |    */
  250 |   async openMessageOptions(message: Locator) {
> 251 |     await message.hover();
      |                   ^
  252 |     await message.getByTestId('message-actions').getByTestId('go-options').click();
  253 |     // The context menu containing the edit button is positioned globally as an overlay
  254 |     return this.page.getByRole('menu');
    at ConversationPage.openMessageOptions (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:251:19)
    at ConversationPage.deleteMessage (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:269:29)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Reply/reply.spec.ts:79:39

❌ I should not be able to send a reply after I got removed from the conversation (tags: TC-3014, regression)

Location: specs/Reply/reply.spec.ts:288
Duration: 34309ms

Errors:

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ hasText: 'Oswald Gutmann removed you' })
Expected: visible
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ hasText: 'Oswald Gutmann removed you' })


  308 |       await userAPages.conversation().removeMemberFromGroup(userB.fullName);
  309 |       await new ConfirmModal(userAPages.conversation().page).clickAction();
> 310 |       await expect(userBPages.conversation().getMessage({content: `${userA.fullName} removed you`})).toBeVisible();
      |                                                                                                      ^
  311 |
  312 |       await message.hover();
  313 |       await expect(message.getByTestId('do-reply-message')).not.toBeAttached();
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Reply/reply.spec.ts:310:102

Flaky Tests:

⚠️ Verify current browser is set as temporary device (tags: TC-3460, regression)

Location: specs/Authentication/authentication.spec.ts:76

Attempt 1
Result: ❌ Failed
Duration: 35850ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="go-preferences"]')


   at pageManager/webapp/components/conversationSidebar.component.ts:52

  50 |
  51 |   async clickPreferencesButton() {
> 52 |     await this.preferencesButton.click();
     |                                  ^
  53 |   }
  54 |
  55 |   async clickAllConversationsButton() {
    at ConversationSidebar.clickPreferencesButton (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/components/conversationSidebar.component.ts:52:34)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Authentication/authentication.spec.ts:110:48
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Authentication/authentication.spec.ts:109:18

Attempt 2
Result: ✅ Passed
Duration: 31455ms

⚠️ Make sure user does not see data of user of previous sessions on same browser (tags: TC-1311, regression)

Location: specs/Authentication/authentication.spec.ts:192

Attempt 1
Result: ❌ Failed
Duration: 2497ms

Errors:

AxiosError: Request failed with status code 409

   at backend/userRepository.e2e.ts:35

  33 |
  34 |   public async setUniqueUsername(username: string, token: string) {
> 35 |     await this.axiosInstance.put(
     |     ^
  36 |       'self/handle',
  37 |       {handle: username},
  38 |       {
    at settle (/home/runner/work/wire-webapp/wire-webapp/node_modules/axios/lib/core/settle.js:19:12)
    at Unzip.handleStreamEnd (/home/runner/work/wire-webapp/wire-webapp/node_modules/axios/lib/adapters/http.js:599:11)
    at Axios.request (/home/runner/work/wire-webapp/wire-webapp/node_modules/axios/lib/core/Axios.js:45:41)
    at UserRepositoryE2E.setUniqueUsername (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/backend/userRepository.e2e.ts:35:5)
    at ApiManagerE2E.createPersonalUser (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/backend/apiManager.e2e.ts:91:5)
    at createUser (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/test.fixtures.ts:180:3)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/test.fixtures.ts:125:15
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Authentication/authentication.spec.ts:197:20

Attempt 2
Result: ✅ Passed
Duration: 42115ms

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud