Skip to content

Use the portal's time zone for raw data query dates instead of UTC.#1779

Merged
aaronweeden merged 1 commit intoubccr:xdmod11.0from
aaronweeden:remove-raw-data-utc
Dec 5, 2023
Merged

Use the portal's time zone for raw data query dates instead of UTC.#1779
aaronweeden merged 1 commit intoubccr:xdmod11.0from
aaronweeden:remove-raw-data-utc

Conversation

@aaronweeden
Copy link
Contributor

@aaronweeden aaronweeden commented Sep 20, 2023

Description

This PR makes it so the warehouse raw data queries involving start and end dates use the portal's local time zone instead of UTC.

There are corresponding PRs for ubccr/xdmod-supremm#356 and https://github.com/ubccr/xdmod-xsede/pull/439.

Motivation and Context

The rest of the XDMoD portal uses the portal's local time zone when dealing with date ranges. For consistency, this PR makes the raw data queries also use the portal's local time zone.

For the Jobs realm, this will have the added benefit of enabling #1780.

Tests performed

In addition to ensuring the regression tests still pass:

  1. Run the script from https://github.com/ubccr/xdmod-xsede/pull/436.
  2. For the differences in the Cloud realm:
    1. Copy the "only in old" lines into a file, run awk -F "', '" '{print $1}' old to get the instance ID, and for a sample of the IDs, run the MySQL query below:
      SELECT i.instance_id, from_unixtime(start_time_ts), from_unixtime(end_time_ts)
      FROM modw_cloud.session_records s
      JOIN modw_cloud.instance i ON i.instance_id = s.instance_id
      WHERE provider_identifier = 'INSTANCE_ID_GOES_HERE';
      
      and confirm that some of the end dates occur after 7–8pm on the day before the given start date, indicating that these entries were in the date range when the query used UTC since Eastern Time is 4–5 hours behind UTC.
    2. Do the same for the "only in new" lines, but confirm that some of the start dates occur after 7–8pm on the given end date, showing that these entries were not in the date range when the query used UTC.
  3. For the differences in the Jobs realm:
    1. Copy the "only in old" lines into a file, run awk -F "', '" '{print $1 ", " $18}' old to get the job ID and resource name pairs, and for a sample of the pairs, run the MySQL query below:
      SELECT from_unixtime(start_time_ts), from_unixtime(end_time_ts)
      FROM modw.job_tasks j
      JOIN modw.resourcefact r ON j.resource_id = r.id
      WHERE j.local_jobid = 'JOB_ID_GOES_HERE'
      AND r.name = 'RESOURCE_NAME_GOES_HERE';
      
      and confirm that some of the end dates occur after 7–8pm on the day before the given start date, indicating that these entries were in the date range when the query used UTC.
    2. Do the same for the "only in new" lines, but confirm that some of the end dates occur after 7–8pm on the given end date, showing that these entries were not in the date range when the query used UTC.

To test the documentation changes, on my laptop:

  1. Make sure Jekyll is installed.
  2. Change to the docs directory.
  3. Run these commands:
    git clone https://github.com/ubccr/xdmod-jekyll-theme
    bundle install
    bundle add webrick
    bundle exec jekyll serve
    
  4. In Chrome, browse to http://127.0.0.1:4000/rest.html and confirm the page changes look correct.
  5. In Chrome, File -> Save Page As -> Webpage, Single File -> save as a temporary file new.mht.
  6. Revert the changes to docs/xdmod-rest-schema.json and let the site regenerate.
  7. In Chrome, refresh, File -> Save Page As -> Webpage, Single File -> save as a temporary file old.mht.
  8. Diff old.mht and new.mht and confirm the only difference besides random IDs is the updates that were made in this PR.

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@aaronweeden aaronweeden mentioned this pull request Sep 20, 2023
3 tasks
@aaronweeden aaronweeden force-pushed the remove-raw-data-utc branch 3 times, most recently from ec5090e to 6b7f0e5 Compare October 16, 2023 20:35
@aaronweeden aaronweeden mentioned this pull request Oct 16, 2023
3 tasks
@aaronweeden aaronweeden added this to the 11.0.0 milestone Oct 17, 2023
@aaronweeden aaronweeden added qa / testing Updates/additions to tests Category:Data Warehouse Export Data Warehouse Export Category: Data Analytics Framework and removed qa / testing Updates/additions to tests labels Oct 17, 2023
@aaronweeden aaronweeden marked this pull request as ready for review November 2, 2023 18:04
@aaronweeden aaronweeden merged commit 797c363 into ubccr:xdmod11.0 Dec 5, 2023
@aaronweeden aaronweeden deleted the remove-raw-data-utc branch December 5, 2023 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants