CORE-746 Quicksilver re-migration #3525
Conversation
| responses: | ||
| 200: | ||
| description: Stuff. | ||
| content: { } |
There was a problem hiding this comment.
ignore; this is for ad-hoc validation of the existing migrations
| def mostRecentMigration: ReadAction[Timestamp] = | ||
| sql"""select max(LAST_UPDATED) | ||
| from WORKSPACE_SETTINGS | ||
| where SETTING_TYPE='CompactDataTables'""".as[Timestamp].head | ||
|
|
||
| def getRecentlyMigratedWorkspaces(maxDate: Timestamp, hours: Int): ReadAction[Seq[UUID]] = | ||
| sql"""select WORKSPACE_ID | ||
| from WORKSPACE_SETTINGS | ||
| where SETTING_TYPE='CompactDataTables' | ||
| and LAST_UPDATED > DATE_SUB($maxDate, INTERVAL $hours HOUR)""".as[UUID] |
There was a problem hiding this comment.
ignore; this is for ad-hoc validation of the existing migrations
|
|
||
| import AuditJsonSupport._ | ||
| logger.info("Entity operation audit", StructuredArguments.raw("audit", auditInfo.toJson.compactPrint)) | ||
| logger.trace("Entity operation audit", StructuredArguments.raw("audit", auditInfo.toJson.compactPrint)) |
There was a problem hiding this comment.
this just reduces log volume for the re-migration. Since this is all happening in a new instance with a new database we don't need the audit logging.
There was a problem hiding this comment.
Just checking - we have other logging that will give us insight into what is happening with the migrations?
There was a problem hiding this comment.
I've improved the logging in the most recent commits to (try to) make sure we have enough insight!
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
some helper code to get exactly the provider we want, regardless of the state of the workspace
There was a problem hiding this comment.
ignore this entire file; this is for ad-hoc validation of the existing migrations
There was a problem hiding this comment.
for this special Rawls instance, which is dedicated to re-migrating entities, I've turned off ALL back-rawls boot monitors ... and then started the new QuicksilverMigrationMonitor.
| // (GoogleBilling, checkGoogleBilling(googleServicesDAO)), | ||
| // (GoogleBuckets, checkGoogleBuckets(googleServicesDAO, bucketsToCheck)), | ||
| // (GooglePubSub, checkGooglePubsub(googlePubSubDAO, topicsToCheck)), | ||
| // (Sam, checkSam(samDAO)) |
There was a problem hiding this comment.
for this special Rawls instance, which is dedicated to re-migrating entities, no need for any of these other health checks. This also helps to reduce logging volume.
| @@ -0,0 +1,210 @@ | |||
| package org.broadinstitute.dsde.rawls.monitor | |||
There was a problem hiding this comment.
This is the main coding effort for re-migrating entities!
There was a problem hiding this comment.
ignore this entire file; this is for ad-hoc validation of the existing migrations
davidangb
changed the title
calypsomatic
left a comment
calypsomatic
left a comment
There was a problem hiding this comment.
Looks good to me - are the test failures due to mixing and matching branches or do they need to be looked into?
@calypsomatic yeah - test failures can be ignored. I have turned off a bunch of functionality (like health checks) and didn't update the tests. I suppose I could just go and update the tests too to make sure this returns cleanly ... |
Note the branches involved in this PR:
Also note this branch has two distinct classes of code:
/api/workspaces/{workspaceNamespace}/{workspaceName}/submissions/validateAPI. I suggest you ignore all this code. This is a work-in-progress attempt to validate how many discrepancies we have in the already-migrated data.The plan is to run this in a separate environment, parallel to prod. The existing prod Rawls service and database will remain untouched for this effort.