Deployment changes for nativeRL and Model Analyzer

[slinlee]

I'm starting to write down notes about what I'm learning about our deployment of nativerl plus webapp. @alexamakarov @xneyder @kepricon

1. When we commit changes to NativeRL branches (dev, test, staging, prod) they are built and copied to the S3 buckets automatically. This is a new change. [link to jenkins file] [link to S3 buckets]
2. Then to get those changes in the webapp, or really the model analyzer, we need to deploy the webapp again for the relevant branch.
3. When deploying the webapp, we also build MA as a docker image:
   https://github.com/SkymindIO/pathmind-webapp/blob/dev/Jenkinsfile#L193-L201
4. This calls this function which gets the right resource files from the model analyzer static files S3 bucket:
   https://github.com/SkymindIO/pathmind-webapp/blob/da0cb7256a9522ccfd066c30468d0efba7afc6ef/Jenkinsfile#L31
5. Only after all of the docker images are built and deployed, we hit our api endpoint that copies updated libraries to the model analyzer S3 bucket
   https://github.com/SkymindIO/pathmind-webapp/blob/d2534aaef4de747036095a2ca32117d6fc45d614/infra/scripts/canary_deploy.sh#L62

Issues I think I see:

• the model analyze static file buckets are only updated at the end of a webapp deployment via an API call: https://github.com/SkymindIO/pathmind-webapp/pull/3315/files#diff-ff600fe7e62165def54c74bbcb688744ecfe0e3377aa916bc78f77a85787d046R111-R153
• the check for newer version code ignores changes if we don't bump the version number with each commit.
  • We can either always copy the files over to the S3 bucket, or use a better automated system for bumping version numbers
• building the docker image for MA seems aggressive at using cache. We /could/ use --no-cache when building, but of course that slows down deploying.

[slinlee]

• @xneyder can you add this call to the /api/MAlib endpoint in the deploy process before the Model Analyzer docker is built? https://github.com/SkymindIO/pathmind-webapp/pull/3584/files#diff-9eea8b8542c9d2a3a86b135e873a348c6789be479f644a85cff6f974d918a1eeR62
• I hardcoded the update function to always copy over the libraries. This will temporarily result in many more backup folders in S3 https://github.com/SkymindIO/pathmind-webapp/pull/3589/files
• Added --no-cache for the docker image build [wip] not doing this one yet
• Kick off webapp deployment when there is a new build of nativeRL nativerl#417 @xneyder

[slinlee]

@xneyder After thinking about it I think it will be easier to just update the code to get the required libraries from the *-training-static-files S3 buckets. I created two PR here:

• Build model anaylzer docker using training-static S3 bucket nativerl#418
• Build Model Analyzer directly from training-static libraries #3594

I reverted the changes we had done above.
It will still be nice to kick off webapp deployment if there are updates to nativeRL

[slinlee]

Btw, the changes I proposed above are just small steps as we work on the larger refactoring outlined by DH here https://3.basecamp.com/3684163/buckets/11875773/messages/3916527719