Updates to summission.py to deal with ExternalQualityMetric ingestion#44
Open
aschroed wants to merge 9 commits into
Open
Updates to summission.py to deal with ExternalQualityMetric ingestion#44aschroed wants to merge 9 commits into
aschroed wants to merge 9 commits into
Conversation
|
Great job! No new security vulnerabilities introduced in this pull requestUse @Checkmarx to interact with Checkmarx PR Assistant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: DO NOT MERGE - this has a dependency on an update to dcicutils - need to merge that in prior to this and make sure pyproject.toml, poetry.lock are updated prior to this merge.
The submissions workbooks for ExternalQualityMetric items are formatted for ease of entry. However, to ingest in a schema conformant way there needs to be a transformation to deal with the arrays of nested objects for EQMs.
Sarah make updates to custom_excel.py in dcicutils/submitr utils to query the portal for field mappings required for the transform. She has a companion PR with those updates - 4dn-dcic/utils#331
This PR are changes in submitr to deal with the transformation and allow that to actually validate and then be ingested.
This at a high level involved transforming the Structured data and then serializing it to a temporary json file that is then used as the actual ingestion input into the portal.
Note that the metadata for the submission is generated from the untransformed workbook just like previously.
For 'ease' of maintenance and consistency all excel workbooks are now treated like this even if there is no EQM sheet.
NOTE: I've successfully submitted a DSA to devtest using this branch - however, some more testing is required to make sure things like resume-uploads works as expected.