force:apex:test:run in JWT context seems to rely on the jwtkeyfile always being present

[ImJohnMDaniel]

Summary

force:apex:test:run command fails when using JWT connection and the jwtkeyfile file is missing from project folder

Steps To Reproduce:

• Have jwtKeyFile at project root
• Authenticate to Dev Hub via JWT auth and make it defaultDevHub at least within the context of current SFDX project.
• Execute a scratch org creation
  • force:org:create --definitionfile config/project-scratch-def.json --json --setdefaultusername --durationdays 1
• Push of source code
  • force:source:push --json
  • executes correctly
• Conduct an Apex Test run
  • force:apex:test:run --testlevel RunLocalTests --outputdir target --resultformat tap --json
  • executes correctly
• remove jwtKeyFile from project directory
• Push of source code
  • force:source:push --json
  • executes correctly
• Conduct an Apex Test run
  • force:apex:test:run --testlevel RunLocalTests --outputdir target --resultformat tap --json
  • execution fails

Expected result

Execution of force:apex:test:run should succeed regardless of the presence of the jwtKeyFile

Actual result

Execution of force:apex:test:run fails if jwtKeyFile is not present in project directory.

Additional information

This becomes a blocker in Jenkins CI processes on SFDX projects. The recommended way to manage the jwtKeyFile on Jenkins is store it in the "Jenkins Admin Credentials interface." During build job execution, Jenkins will checkout all code to the build job's "workspace" folder. It will download the the jetKeyFile and other secret files to the adjacent folder "workspace@tmp" and inject that file during commands that explicitly utilize it (like the force:auth:jwt:grant command).

As it stands now, I am unable to utilize force:apex:test:run as part of my CI process.

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):

• sfdx-cli/7.4.0-99233fd3af darwin-x64 node-v10.15.3

SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)

• @oclif/plugin-commands 1.2.2 (core)
• @oclif/plugin-help 2.1.6 (core)
• @oclif/plugin-not-found 1.2.2 (core)
• @oclif/plugin-plugins 1.7.8 (core)
• @oclif/plugin-update 1.3.9 (core)
• @oclif/plugin-warn-if-update-available 1.7.0 (core)
• @oclif/plugin-which 1.0.3 (core)
• @salesforce/sfdx-trust 3.0.2 (core)
• analytics 1.1.2 (core)
• generator 1.1.0 (core)
• salesforcedx 45.11.0 (core)
  • force-language-services 45.10.0 (core)
  • salesforce-alm 45.14.0 (core)
• sfdx-cli 7.4.0 (core)

OS and version:

• MacOS v10.14.4
• Ubuntu Linux (amd64) latest version

[ImJohnMDaniel]

The error message that I get is the following:

sfdx force:apex:test:run --testlevel RunLocalTests --outputdir target/27  --resultformat tap --json
{
    "status": 1,
    "name": "InvalidAsyncTestJob",
    "message": "Unable to invoke async test job: ENOENT: no such file or directory, open 'server.key'",
    "exitCode": 1,
    "commandName": "ApexTestRunCommand",
    "stack": "InvalidAsyncTestJob: Unable to invoke async test job: ENOENT: no such file or directory, open 'server.key'\n    at ALMError (/Users/johndaniel/.local/share/sfdx/client/7.
4.0-99233fd3af/node_modules/salesforce-alm/dist/lib/core/almError.js:44:19)\n    at waitTillConnected.then.then.then.then.then.catch.err (/Users/johndaniel/.local/share/sfdx/client/7.4
.0-99233fd3af/node_modules/salesforce-alm/dist/lib/apex/apexTestApi.js:532:29)\n    at tryCatcher (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-a
lm/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/no
de_modules/bluebird/js/release/promise.js:510:31)\n    at Promise._settlePromise (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/b
luebird/js/release/promise.js:567:18)\n    at Promise._settlePromise0 (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/bluebird/js/
release/promise.js:612:10)\n    at Promise._settlePromises (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/bluebird/js/release/pro
mise.js:687:18)\n    at Async._drainQueue (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:138:16)\n  
  at Async._drainQueues (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:148:10)\n    at Immediate.Asy
nc.drainQueues [as _onImmediate] (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules/salesforce-alm/node_modules/bluebird/js/release/async.js:17:14)\n    at runCa
llback (timers.js:705:18)\n    at tryOnImmediate (timers.js:676:5)\n    at processImmediate (timers.js:658:5)\nOuter stack:\n    at Function.wrap (/Users/johndaniel/.local/share/sfdx/c
lient/7.4.0-99233fd3af/node_modules/@salesforce/core/lib/sfdxError.js:151:27)\n    at ApexTestRunCommand.catch (/Users/johndaniel/.local/share/sfdx/client/7.4.0-99233fd3af/node_modules
/salesforce-alm/dist/ToolbeltCommand.js:216:46)",
    "warnings": []
}```

[ImJohnMDaniel]

Other commands that are affected by this are:

• force:source:status
• force:source:pull
• force:source:push (yes, I listed above that it worked and it did, but now it doesn't work)

[ImJohnMDaniel]

Is this an issue where the jwt connection works for a while and then the CLI needs to send a refreshtoken request and when that request occurs, it needs the jwtkeyfile to make that refresh request? Is there a way to observe this? Does this suggest that the connected app on Salesforce is not configured correctly?

[shetzel]

This is working as designed. Calling Salesforce APIs requires an access token. Access tokens are valid for a certain amount of time before they expire. In the case of JWT auth there are no refresh tokens. When the access token expires, the CLI will use the persisted auth data to get a new, valid access token and update the local auth data so that API calls will succeed.
For CI, you can use an environment variable for the path to your jwtkeyfile.
@clairebianchi

[ImJohnMDaniel]

@shetzel thanks for the update. That may be that it is working as designed, but it breaks the use case that I mentioned above about the "....Jenkins CI processes on SFDX projects" and that Salesforce recommends that the management of the jwtKeyFile on Jenkins is to store it in the "Jenkins Admin Credentials interface." (see documentation here, section 2b). The "Jenkins Admin Credentials interface" stores the file in another directory outside of the Jenkins workspace folder and thus the error. The force:auth:jwt:grant command gets the JWT file injected by the withCredentials( [ file( credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file') ] ) command.

Also, why would it be that I am able to execute a couple of commands (i.e. force:org:create, force:source:push, etc.) before the process fails on force:apex:test:run??

@clairebianchi

[clairebianchi]

Sounds like there might still be an issue if you were able to run some commands and not others. Also maybe the documentation on Jenkins is wrong the way we are suggesting you set up isn't correct. @shetzel could you shed some light on why the other commands worked but test:run failed?

[shetzel]

Commands will work as long as the token is valid or if the command doesn't make an API call, such as force:alias:list. When the token expires and you run a command that makes an API call, it will fail during auth.

As for Jenkins, we had a similar issue but it was resolved by wrapping all the steps inside a withCredentials block in order to guarantee that the location Jenkins creates to store the jwt key file remains the same. If multiple withCredential blocks are used then jenkins may actually place the key file in a different spot, and the persisted auth files that the CLI uses will point to an invalid jwtkeyfile location. Try using a single withCredentials block in your Jenkins script.

[ImJohnMDaniel]

@shetzel - the force:apex:test:run command is in a different stage. I did wrap that command with the withCredentials block. That did not help. Still got the same error. I did not wrap the force:source:push with the withCredentials block and that part worked just fine again.

I am not certain how wrapping the force:apex:test:run command in a withCredentials block would do. The withCredentials command simply exposes the server.key file as an environment variable within that block. Since the force:apex:test:run command does not utilize the jwtKeyFile environment variable like the force:auth:jwt:grant does, I do see how this is helping.

Thoughts?

[shetzel]

As long as everything is wrapped in the same withCredentials it should work. The auth files that are written during force:auth:jwt:grant and force:org:create reference the temporary location of the jwt file. That location should be there as long as the stages are executing within the same withCredentials scope, so the behind the scenes re-auth should work. Here is a sample Jenkinsfile from an old repo we used in the past:

`
#!groovy
import groovy.json.JsonSlurperClassic
node {

def BUILD_NUMBER=env.BUILD_NUMBER
def RUN_ARTIFACT_DIR="tests/${BUILD_NUMBER}"
def SFDC_USERNAME

def HUB_ORG=env.HUB_ORG_DH
def SFDC_HOST = env.SFDC_HOST_DH
def JWT_KEY_CRED_ID = env.JWT_CRED_ID_DH
def CONNECTED_APP_CONSUMER_KEY=env.CONNECTED_APP_CONSUMER_KEY_DH

def toolbelt = tool 'toolbelt'

stage('checkout source') {
    // when running in multi-branch job, one must issue this command
    checkout scm
}

withCredentials([file(credentialsId: JWT_KEY_CRED_ID, variable: 'jwt_key_file')]) {
    stage('Create Scratch Org') {

        rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:auth:jwt:grant --clientid ${CONNECTED_APP_CONSUMER_KEY} --username ${HUB_ORG} --jwtkeyfile ${jwt_key_file} --setdefaultdevhubusername --instanceurl ${SFDC_HOST}"
        if (rc != 0) { error 'hub org authorization failed' }

        // need to pull out assigned username
        rmsg = sh returnStdout: true, script: "${toolbelt}/sfdx force:org:create --definitionfile config/project-scratch-def.json --json --setdefaultusername"
        printf rmsg
        def jsonSlurper = new JsonSlurperClassic()
        def robj = jsonSlurper.parseText(rmsg)
        if (robj.status != 0) { error 'org creation failed: ' + robj.message }
        SFDC_USERNAME=robj.result.username
        robj = null

    }

    stage('Push To Test Org') {
        rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:source:push --targetusername ${SFDC_USERNAME}"
        if (rc != 0) {
            error 'push failed'
        }
        // assign permset
        rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:user:permset:assign --targetusername ${SFDC_USERNAME} --permsetname DreamHouse"
        if (rc != 0) {
            error 'permset:assign failed'
        }
    }

    stage('Run Apex Test') {
        sh "mkdir -p ${RUN_ARTIFACT_DIR}"
        timeout(time: 120, unit: 'SECONDS') {
            rc = sh returnStatus: true, script: "${toolbelt}/sfdx force:apex:test:run --testlevel RunLocalTests --outputdir ${RUN_ARTIFACT_DIR} --resultformat tap --targetusername ${SFDC_USERNAME}"
            if (rc != 0) {
                error 'apex test run failed'
            }
        }
    }

    stage('collect results') {
        junit keepLongStdio: true, testResults: 'tests/**/*-junit.xml'
    }
}

}
`

[ImJohnMDaniel]

@shetzel -- a couple of questions:

• Have you had any success using that approach with the pipeline -> stages -> stage syntax?. The tests that I have run thus far prevent the withCrendentials block from wrapping the pipeline block as pipeline has to be top tier.
• Why does the CLI need to connect to the DevHub in the JWT Auth Flow for each command. I understand that it needs to authenticate to the DevHub in that manner, but once it is authenticated, doesn't the CLI get an access token to use from there forward? Why would the CLI not utilize the access token for the subsequent commands?
• The command fails when trying to access the scratch org. Why is the connection to the scratch org dependent on a JWT type flow? The connection to the DevHub needs to be JWT, I get that. But I don't understand why that is ? Can you shed some light on why this is the case or at least why this appears to be the case?

[ImJohnMDaniel]

@shetzel -- additional question would be:

• Why is it that the force:apex:test:run command fails when the jwtKeyFile is not present but the force:source:push command works. Both make API calls but only to the scratch org; not the Dev Hub

[shetzel]

re: jenkins - Seems like you're using the newer, declarative method, and it appears that the withCredentials plugin has issues with that method. I don't have any experience with it so doubt I'll be much help there.

The CLI doesn't need to connect to the DevHub for each command. The auth for the scratch org uses the same flow as for the devhub, as opposed to the auth code from the scratch org signup request (in which case it would follow the refresh token flow). API requests to the scratch org should be using the access token obtained for the initial scratch org auth. It's possible that the jwtkeyfile is being checked when the access token is still valid, which sounds like what you're describing.

Assuming there is no access token expiration happening between source:push and apex:test:run, then you're correct that the code is checking the jwtkeyfile existence unnecessarily. I'll create an investigation so we can track this. @clairebianchi