DataPackage Utilities part 1 - SystemMetadata and supporting utilities - #2812
Open
robyngit wants to merge 56 commits into
Open
DataPackage Utilities part 1 - SystemMetadata and supporting utilities#2812robyngit wants to merge 56 commits into
robyngit wants to merge 56 commits into
Conversation
- Class handles persistent storage of object versions using localForage, to be integrated with VersionTracker - Add unit tests for VersionStorage class Issue #2783
- Add DataONEHttpError, HttpRetryPolicy, and UrlBuilder models to eventually be used within a DataONEHttpClient model for handling DataONE-specific HTTP requests, retries, deduplication, and error handling. Issue #2783
- Convert the VersionStorage model to a more generic PersistentStorage model that can be used for other caching needs. - Add more complete tests for PersistentStorage. Issue #2783
- Class is used for DataONE API calls and handles URL building, retries, dedupe of in‑flight requests, abort/timeouts, and response parsing. - Add unit tests for the new class. Issue #2783
- SysMeta class purly handles parsing (and eventuallt serialization) of XML Issue #2783
- Create DataONEService, an extendable base class that handles token resolution, caching, upload/download, and error handling. Issue #2783
- Extends DataONEService - Returns SysMeta object on download - Add unit tests for SysMetaService Issue #2783
buildUrl now handles empty paths without resetting to root, and strips trailing slashes. Issue #2783
- Remove logic for caching SysMeta and leverage SysMetaService instead - Uses cached SysMeta instead of full version chains which are fragile - Remove add version logic and istead rely on SysMetaService/DataONEService to cache successfully uploaded sysMeta Issue #2783
In Utilities: - add stableStringify for deterministic object/array hashing to be used to create singleton and cache keys - add normalizeUrl helper (for keys and general use) that trims and removes trailing slashes, with optional fallback - add buildInstanceKey and getSingleton for consistent singleton management (DataONEHttpClient, SysMetaService) - add awaitMetacatUI helper for deferred MetacatUI access (e.g., in tests) - add unit tests covering new utilities Issue #2783
- normalize config passed to constructor, stricter validation
- replace "namespace" with deterministic key based on config
- derive singleton/store keys using the new Utilities.buildInstanceKey
- allow localforageConfig overrides for name/storeName/version
- expose removeItem errors
- switch record shape to `{ value, expiresAt }` from `{ value, updatedAt, ttlMs }`
- set default TTL to 1 hour
- add and update tests to match new behaviour
Issue #2783
DataONEHttpClient: - Make sure we don't lose functions when cloning options during normalization - Make header merging case-insensitive so request headers override defaults without duplication DataONEService: - Fix bug where per-request auth option was ignored - Create private local storage cache based on loggedin user username rather than token because token can change during a session - Remove ability to pass explicit token in request options for simplification (always rely on getToken function) SysMetaService: - On upload, preserve any headers passed in options - Avoid caching sysMeta XML that is invalid For all the above: - Add stricter validation for client & request options - Build instance and deduplication keys based on normalized config objects - Add more unit tests - Also increase timeout for fetching token (UserModel) to account for slow responses/connections Issue #2783
- Stop creating singletons for VersionTracker class, only underlying DataONEHttpClient (to dedup SysMeta fetches) and PersistentStorage (to share SysMeta cache) are singletons. - Add stricter input validation (especially for constructor options) - Make sure getAdjacent doesn't ignore useCache cacheKey and options - Make sure request options (e.g. for HTTP requests to get SysMeta) are passed through properly - Queue notify promises during version traversal so that errors can be handled explicitly, while still not slowing down the loop. - Ensure notify emits errors even when no adjacent SysMeta is found - Expand test coverage Also: Fix parsing of numeric fields in SysMeta Issue #2783
- Use new PersistentStorage class rather than localForage directly (since it now includes more robust error handling, etc.) - Remove excess RM verification in guessPid step - Fix overly general listener removal in resolveFromSeriesId (only remove the listener for the specific handler) - Don't make RM resolver a singleton, only underlying Persistent Storage and DataONEHttpClient need to be singletons for shared caching and request dedup - Catch errors such that Solr fails, keep resolution going to next steps - Improve JSdocs - Update unit tests and add new ones - escape PID values in searchIndex queries via QueryService.escapeLucene - correct checkLogForMultipleRMs so only rms.length > 1 sets multipleRMs - handle index lookup failures during walkSysmeta - Add more unit tests Issue #2783
Update VersionNav & EMLEditor to conform to changes made to the ResourceMapResolver and VersionTracker. Issue #2783
DataONEObjects - to organize sysMeta for each version VersionTimelineGroups - to group versions by date Issue #2766
- When sysMeta download for notification is aborted, stop notification and don't throw an error. This can happen on re-render. Issue #2766
Also move VersionHistoryView into the VersionHistory folder Issue #2766
- Set a max width on the entire version history view instead of just the version timeline - Change the warning messages to show the error part of the message first, followed by the # of versions found Issue #2766
- Use local timezone consistently for grouping and displaying dates in the version history view - Re-render timeline groups when date or date label change - Add a new DateUtility class and move date helpers from Version history views and collections to this class Issue #2766
- Pass the VersionTimeLineGroup model directly to the VersionTimelineGroupView for consistency, rather than passing individual attributes - Rename `renderAll` method from VersionTimelineGroupView to `render` for consistency with other views Issue #2766
- Add badge next to versions with DOI and add filter to hide all except those with DOIs - Add tooltips to ToggleView, clean up some code & jsdocs - Add disabled state to ToggleView - Along the way, clean up VersionTimelineGroup css class names and usage - Remove unused class/element in VersionHistoryView Issue #2766
- Use the "tokenChecked" property of the user model to determine whether the token has already been checked instead of calling multiple times Issue #2766
- Show relative dates rather than older/newer - Show separate badges for errors, warnings, and key points like current, newest, oldest. A single version can have multiple badges. - Add descriptions to badges (will use as tooltips) - Add getRelativeDateString method + tests for DateUtility - Fix JSdocs linting issues in DateUtility - Remove unused "showStatus" option from ObjectVersionView (always show badges) Issue #2766
- Add a badge to show the relative position of each version in the version history timeline - Add tooltips to all badges - Make badge descriptions dynamic, include the date diff or version diff in the descriptions - Remove redundant "hasPopupModule" function from ToggleView since view won't render without Semantic import Issue #2766
- When the VersionTracker reaches the maximum number of hops allowed in a single search, it will now set a `maxHopsReached` property on the record for that PID - The VersionHistoryView includes an appropriate message in the error message shown to the user when one or both sides of the version chain are incomplete. Issue #2766
- During obsolesence chain traversal in VersionTracker, detect when an object's dateUploaded is more recent than the object that obsoletes it (i.e. not chronological) Misc: - Forward options for downloads through VersionTracker getNext/getPrev - Make notify always emit placeholder sysmeta with errors/versionHistory Issue #2766
- Flag date conflicts in version history view
- date conflict summary banner to VersionHistoryView
- inline date conflict notes to ObjectVersionView
- Highlight timeline groups containing visible date conflicts
- Sort by obsolesence chain order (newest to oldest) instead of
date uploaded in version history timeline
- Group timeline entries by contiguous chain-order date segments,
with date conflict segments grouped separately and highlighted
with a warning style
- Unify hidden-state styling with version-history--hidden class
- Normalize border color CSS custom property naming
- Expand test coverage for version history views and collections,
including date conflict scenarios
Issue #2766
- Reduce the number of total re-renders, update views instead when possible - Only render badge tooltips once, after all versions are found (or an error occurs) - Prevent remaining "progress" messages from showing once the the full chain is found or an error occurs (otherwise it sometimes overwrites the final status message) Issue #2766
- New AppModel setting, "showVersionHistory", defaults to true Issue #2766
Files missing from commit e4d9ac6 - Version history test - autoAddTooltip setting in ObjectVersionView
Refresh button clears all versions in local storage and/or memory and re-fetches the version history from the server. Issue #2766
Use empty string instead of empty span elements for badges that are not applicable. Badges are in a container now and the empty badges are no longer needed to maintain grid structure. Issue #2766
- Remove top-level summary banner and show only per-version date notes in Version History - Update per-version message wording to be less alarming - Switch timeline markers to blue info styling instead of yellow warning styling - Update tests to reflect these changes Issue #2766
Need to be updated to reflect design changes, especially the multiple badges displayed and date format Issue #2766
Needed to show upload/download progress Issue #2804
- Refactor XMLUtilities around DOMParser and XMLSerializer - Replace cleanXMLText with DOMParser/XMLSerializer-based XML helpers - Native APIs handle character encoding and escaping - Rename some functions for clarity and consistency - Add element lookup, selector, and invalid-character utilities - Update MetadataView to use new fn name - Switch EMLEntityView back to legacy model-level XML cleaning - Manual text encoding required for XML parsed with jQuery's HTML parser - Add tests for XMLUtilities Issue #2804
- extract generic value, URL, comparison, and formatting helpers into ValueUtilities - move DOI validation and shared validation helpers into ValidationUtilities - add DataONEXmlUtilities for parsing DataONE error XML and plain-error serialization - expand XMLUtilities with stricter parsing, root/namespace checks, sequence validation, and serialization helpers - update service, model, and view call sites to use the new utility modules - centralize object format fetching through Utilities.getObjectFormats() - add unit coverage for the new utility modules and XML helper behavior Issue #2804
- SysMeta can now parse, serialize, and validate all SysMeta docs (covers all fields in the schema) - Add sub-classes for access policy, replication, replica, and media type - move SysMeta field metadata into SysMetaSchema - Update editor/version-history integration to use the new SysMeta model shape - Add more SysMeta and SysMetaService unit tests Issue #2804
- Service handles /reserve/ and /generate/ DataONE endpoints Issue #2804
- add identifier XML parsing & default header helpers for DataONE services - add CN generate URL coverage in AppModel - add tests for editor/version history and upload progress validation - add missing ObjectFormats import to Utilities.js Issue #2804
- Move URL utilities from dataONEServices/UrlBuilder and ValueUtilities to common/UrlUtilities, and remove UrlBuilder - Update tests and callers to use new UrlUtilities Issue #2804
- add reusable validation and value utility helpers for normalization, sorting, and exceptions - centralize shared DataONEService client config and request option handling - standardize IdentifierService and SysMetaService validation - simplify SysMeta validation exception handling - add unit tests for all new utilities and services Issue #2804
- Add normalizePid and encodePidPath utilities to DataONEService - Use in SysMetaService and IdentifierService - Add and update unit tests Issue #2804
- ObjectService can download, upload, and update DataONE objects - Also fix linting issues in DataONEService - Add ObjectService tests Issue #2804
- extend UrlUtilities with RFC3986/DataONE PID path-segment encode/decode methods - add ValueUtilities helpers for list coercion, plain-object checks, cloned array-valued records, readable string lists, and validated string choice handling - rename integer validation helpers positiveInteger --> nonNegativeInteger (since it includes zero) - fix MetacatUI property lookup so undefined and falsy values are handled correctly - rename objectFormat method to reflect async behavior - rename DateUtility to DateUtilities and add improve date parsing and formatting methods - update callers to use new module/method names - add tests Issue #2804
- add XMLTypes plus new ValueUtilities and ValidationUtilities helpers - add DataONEXmlUtilities.parseRequiredDocument and cover DataONE error XML handling - simplify XMLUtilities by removing methods that can be handled by native DOM APIs - add & update unit tests Issue #2804
- rename SysMeta to SystemMetadata - expose properties directly on the instance instead of a .data object - add add, replace, and remove methods for properties that can have multiple values (e.g. accessPolicy, mediaType, replica, etc.) - allow some irregular XML to be parsed and repaired, record problems in parseWarnings - always serialize sysmeta to v2 - use simple field normalization/validation via XMLTypes - add a checksum class and use it for checksum properties instead of a plain string - update SysMetaService, version history integrations, and unit tests Issue #2804
- Use `ResourceMapResolver` to resolve resourceMap IDs when Solr returns none or multiple - Check for existence of sysMeta if Solr lookup fails, use that to determine if dataset is indexing vs truly not found - break apart `MetadataView` model getting function into `onModelSync`/`onModelError` handlers - use `QueryService` to find metadata for data objects - normalize event-listener cleanup to avoid duplicate error handling - add `MetadataView` unit tests Issue #569
Remove dev code Issue #2804
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates the
SystemMetadataclass (prev.SysMeta) and adds supporting sub-classes, XML helpers, and validation utilities. With this update, theSystemMetadataclass covers the complete sysMeta schema, including parsing, validation, and serialization of all elements. It also adds methods to enable editing, likeadd,remove, andreplacemethods on nested complex (e.g.AccessPolicyandReplicationPolicy).Changes
SystemMetadatato represent nested sysmeta sections / child elements:ChecksumAccessPolicy/AccessRuleReplicationPolicyReplica/ReplicaListMediaType/MediaTypePropertySysMetaSchemawhere XML schema definition and related constants live.SystemMetadata. These utilities will be used byResourceMapand can replace some repeated code across the codebase:ValueUtilitiesValidationUtilitiesDateUtilitiesUrlUtilitiesDataONEXmlUtilitiesXMLUtilities/XMLTypesSysMetaServiceand related DataONE service plumbing to use normalized URLs/PIDs, stronger request config handling, andSystemMetadataparsing/serialization.SystemMetadatato use the new model shape and methods, including:VersionTrackerResourceMapResolverVersionHistoryViewHow
SystemMetadataWorksSystemMetadatais a typed model with direct properties (rather than the previous nested.dataobject). Basic fields likeidentifier,seriesId,obsoletes,dateUploaded, etc. live directly on the instance as simlpe strings, numbers, etc. More complex elements that have their own nested structure are represented as child classes/objects, likeaccessPolicy(which has its own nestedAccessRuleobjects),replicationPolicy(with nested fields and a list ofReplicaobjects), andmediaType(with nestedMediaTypePropertyobjects).The model is designed around four operations:
SystemMetadata.fromXml()/.parse()reads DataONE sysmeta XML into normalized root properties and child objects..validate()returns structured validation issues for root fields and nested content..serialize()writes canonical v2 sysmeta XML..toJSON()returns a JSON-safe domain-shaped snapshot.Parsing is tolerant of common XML irregularities, so that viewing and editing objects is not blocked by minor sysMeta errors. Errors that are recoverable include:
systemMetadataand child elements. Parsing matches element local names case-insensitively and ignores prefixes, so namespace-free XML and mixed-prefix XML still parse.seriesIdis accepted even when the xml is v1.mediaTypeproperties,accessPolicyrules, andreplicaentries are dropped individually while valid sibling are preservedFixes are tracked in
parseWarnings.Truly invalid required content still fails parsing with an error, including malformed XML, DataONE
<error>responses, the wrong document root, and missing or invalid required properties.Upon serialization, the model writes valid, normalized v2 SysMeta.
Reviewing
This is a big PR since it involves a key model and creates many new supporting files. The most important review areas are:
src/js/models/sysmeta/SystemMetadata.jsand its child modelssrc/js/models/dataONEServices/SysMetaService.jssrc/js/models/sysmeta/VersionTracker.jssrc/js/models/resourceMap/ResourceMapResolver.js