Failing with GitHub+You on internal test of signin failure

[plocket]

This is a simplified summary of what I did, and my observations.

Symptoms

I have run tests in three different environments:

1. Sandbox tests (GitHub CI creates its own GitHub server, installs the package straight onto the server, and then sends a bot to click and type on the GitHub server)
2. GitHub+You (GitHub CI creates a Project on our dev server and sends a bot to click and type in the form there)
3. Local-to-remote (From my local machine, I send a bot to click and type on a Project in my da account on our dev server)

Sandbox and local-to-remote always pass, even after dozens of attempts. GitHub+You often fails, but not always. The fact that it's not consistent tells me it's probably some kind of race condition. The fact that it behaves differently in different environments tells me that something is different between those environments. The difference could just be a matter of speed.

The example test, @example_scenario_tag, would fail first, after which all tests would fail. I excluded @example_scenario_tag in the tag expression and the test that was next in line failed instead, as well as all the following tests. That led me to look at the test before @example_scenario_tag which is @e10 ("wrong sign-in"). I removed @e10 and all the tests passed, 10 times in a row. So @e10 is an important clue, but...

The error that comes with the failure is a timeout after 30 seconds (the timeout we set for cucumber and puppeteer, and this message is coming from cucumber). Some tests do take that long to (purposefully) fail. Both the tests I named, though, take milliseconds, so that timeout message indicates to me that there's some kind of global state involved. That is, the timer for a test a bit over 30 seconds previous to @example_scenario_tag is somehow still active. But then why does removing @e10 remove the problem?

In addition, the logs behave quite strangely. Despite the debug logs containing more information about the tests that follow, the console report shows up like this:

...previous test results...
━━━━━━━━━━━━━━━
Scenario: Fail with no interview at fully arbitrary url
Tags: @establishing @fast @failure @e8 @rfe8 @arbitraryurl
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
the Scenario report should include:
the max seconds for each step in this scenario is 5
I start the interview at "https://www.usa.gov/"
  Trying to load the interview at "https://www.usa.gov/"
  🤕 ALK0116 ERROR: On final attempt to load interview at "https://www.usa.gov/", got "ALKiln could not find any interview question page with the given information"
━━━━━━━━━━━━━━━
Scenario: I fail to sign in with wrong email and password
Tags: @establishing @fast @e10 @signin @failure
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
I sign in with "WRONG_EMAIL", "WRONG_PASSWORD"
  🤕 ALK0209 ERROR: Failed to sign into ***/user/sign-in. Make sure you followed the instructions at https://assemblyline.suffolklitlab.org/docs/alkiln/writing/#sign-in.
━━━━━━━━━━━━━━━
Scenario: I fail to sign in with wrong email and password
Tags: @establishing @fast @e10 @signin @failure
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
I sign in with "WRONG_EMAIL", "WRONG_PASSWORD"
  🤕 ALK0209 ERROR: Failed to sign into ***/user/sign-in. Make sure you followed the instructions at https://assemblyline.suffolklitlab.org/docs/alkiln/writing/#sign-in.
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
━━━━━━━━━━━━━━━
Scenario: I fail to sign in with wrong email and password
Tags: @establishing @fast @e10 @signin @failure
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
I sign in with "WRONG_EMAIL", "WRONG_PASSWORD"
  🤕 ALK0209 ERROR: Failed to sign into ***/user/sign-in. Make sure you followed the instructions at https://assemblyline.suffolklitlab.org/docs/alkiln/writing/#sign-in.
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
━━━━━━━━━━━━━━━
Scenario: I fail to sign in with wrong email and password
Tags: @establishing @fast @e10 @signin @failure
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
I sign in with "WRONG_EMAIL", "WRONG_PASSWORD"
  🤕 ALK0209 ERROR: Failed to sign into ***/user/sign-in. Make sure you followed the instructions at https://assemblyline.suffolklitlab.org/docs/alkiln/writing/#sign-in.
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
━━━━━━━━━━━━━━━
Scenario: I fail to sign in with wrong email and password
Tags: @establishing @fast @e10 @signin @failure
━━━━━━━━━━━━━━━
the final Scenario status should be "failed"
the Scenario report should include:
I sign in with "WRONG_EMAIL", "WRONG_PASSWORD"
  🤕 ALK0209 ERROR: Failed to sign into ***/user/sign-in. Make sure you followed the instructions at https://assemblyline.suffolklitlab.org/docs/alkiln/writing/#sign-in.
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**
For security, ALKiln will avoid creating a picture of the page for this error. It's possible a secret is being used on this screen.
**-- Scenario Failed --**

Work so far

I'm not sure if this is a bug exactly, or something that's up with our pipeline. So far I've focused on reducing the differences between the different environments. I've merged code to update our actions to

• Install the same version of docassemblecli by default
• Update their node dependencies

Those haven't worked.

Planned work

1. I looked at the code for @e10. It tries to sign in using an incorrect email and password. I looked at the code for that Step and it could use improvement. It gives our code control of the timeout, but then doesn't ensure the operation won't timeout. That seems like it would cause an infinite hang instead of a timeout error, but it may be worth chasing down.
2. @BryceStevenWilley brought up the idea that our screenshot-taking practices can change to allow us to see more about what's going on while staying secure. We would avoid taking a screenshot of the live screen, but would still copy the html into a file. The HTML shouldn't contain the answers to the fields. With real people's information, this sort of capture could still be insecure because, for example, a user's answer could reveal a hidden field and thus we would know something about their answer. For testing, though, it could work very well. [See When avoiding taking screenshots of sensitive answers, still get page HTML #1099].

[plocket]

I'm still not 100% sure what the deep source of the problem was. Specifically, the fact that all subsequent tests failed. But tests are consistently passing now and I'm making a PR to close this (see timeline above).

Waiting for navigation after signing in/failing to sign in and timing may have lined up just right in GitHub+You tests to trigger a failure when a user fails to log in. That, in combination with the disabled timeout and our dev server's setting to allow long loading times may have been enough to cause all of the subsequent tests to fail. The tests should have kept running long enough for the server to recover, though (over an hour). I can no longer replicate the bug, and I don't have a lot of time to give to this right now.

Replicating and digging might help us find out if adding a server health check to the sign-in Step could help at least give more helpful error messages. Maybe we should add a debug log in the server status check itself when the server isn't available. Maybe we already have it! Maybe we should fully log it to the authors.

Potential replication for further debugging: Maybe undoing the navigation detection and timeout handling would allow us to replicate the bug and dig deeper.