Optimize raw data requests.

[aaronweeden]

Description

This improves the performance of raw data requests.

Instead of repeatedly requesting the entire time range in chunks of 10,000 rows (or whatever the portal has configured as the rest_raw_row_limit), the method now makes a separate request for each day in the range, still in chunks of 10,000 rows (or whateverrest_raw_row_limit is).

For example, instead of:

start_date=2023-01-01&end_date=2023-01-31&offset=0
start_date=2023-01-01&end_date=2023-01-31&offset=10000
start_date=2023-01-01&end_date=2023-01-31&offset=20000
...

it is now (note that for each request, start_date == end_date):

start_date=2023-01-01&end_date=2023-01-01&offset=0
start_date=2023-01-01&end_date=2023-01-01&offset=10000
start_date=2023-01-01&end_date=2023-01-01&offset=20000
...
start_date=2023-01-02&end_date=2023-01-02&offset=0
start_date=2023-01-02&end_date=2023-01-02&offset=10000
start_date=2023-01-02&end_date=2023-01-02&offset=20000
...

The full result is received faster, even though more queries are made (e.g., ~1min for 7 days (258,836 rows) of SUPREMM data as opposed to ~1m20s seconds), and for a portal that implements ubccr/xdmod#1780, requests for raw Jobs realm data are received much faster (around 20 seconds to fetch two days (63,251 rows) of Jobs data as opposed to around 25 minutes).

This also updates the show_progress feature to also print the number of days that have been retrieved so far.

TODO: add tests

Motivation and Context

Requests for raw data in the ACCESS XDMoD Jobs realm are incredibly slow. This is also the case for metrics-staging.

Tests performed

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• The pull request description is suitable for a Changelog entry
• The milestone is set correctly on the pull request
• The appropriate labels have been added to the pull request
• Running the automated tests (see docs/developing.md) produces no errors.

[aaronweeden]

Closed in favor of #19.