ClickHouse version of import users script

[jamesqo]

Summary of changes

• Removed all read/write interactions with MySQL
• Moved all read operations with MySQL -> ClickHouse
• Replace usage of clickhouse CLI with clickhouse-connect library (will be useful in the future when this script is moved to the Airflow server)
• Update users / authorities table in a single batched SQL transaction
• Removed --ssl-ca flag which no longer applies (this was only relevant for RDS)

The portal.properties files also need to be updated in order to include clickhouse creds -- will submit a PR to pipelines-configuration later

cc @averyniceday @sheridancbio

[jamesqo]

still need to test changes here

[jamesqo]

Did some testing. Further changes:

• Converted script to Python3
• Added more logging
• Speed it up by doing a batched query for authorities in 1 swoop

[averyniceday]

let's add error checking like we were doing before for SQL here?

[sheridancbio]

Yes .. if the insert fails I wonder what happens. I suppose ch_client will throw an exception? At the very least we should confirm that if this happens we will get a stacktrace or some kind of indication in the run logs. A notice to our slack channel (letting us know that we failed an attempt to add a user to the user table) should probably also be made.

[jamesqo]

I added back code that will email the failure message to users if there is an exception thrown during inserts.

[averyniceday]

General changes look good - couple suggestions /questions regarding readiability and organization

[sheridancbio]

I breezed over the parts which seemed to be clear language conversions to accommodate python 3 syntax. So long as this works in production I approve. I suppose the inherent retry is still present because each time this runs we re-compute the list of users who are missing from the database and try to import them again.

[sheridancbio]

Yes .. if the insert fails I wonder what happens. I suppose ch_client will throw an exception? At the very least we should confirm that if this happens we will get a stacktrace or some kind of indication in the run logs. A notice to our slack channel (letting us know that we failed an attempt to add a user to the user table) should probably also be made.

[jamesqo]

@sheridancbio I'm wary of adding code for Slack notifications -- if something goes wrong with the network for example, we will be getting pinged every minute by this script. I did improve the error handling for failed DB operations however

[jamesqo]

@copilot resolve the merge conflicts in this pull request

[jamesqo]

We decided that we will be leaving the behavior on failure as-is for now but we should probably address it in the future so that users don't get spammed with emails if something goes wrong with our networking, for example.

[jamesqo]

removed these lines as we're no longer using the clickhouse CLI

[averyniceday]

Overall looks good to me - only one change -- let's just replace the contents of the existing importUsers.py script instead of having a separate importUsersClickhouse.py script. That way we do cleanup as part of this rollout, thanks!

[sheridancbio]

Overall, I'm good with merging this PR. But one thing I'm not sure I understand ... why did we switch from using the command line executable "clickhouse client" and switch to using the python third party library "clickhouse_connect"?

I'm not against this ... but it means that now we have some parts of our code using clickhouse client and some parts using "clickhouse_connect". Both libraries come from the same organization ... but having 2 dependencies is worse than having 1. Perhaps we should stop using clickhouse client and do all clickhouse interactions through clickhouse_connect.