Skip to content

Cache dryrun results locally #8104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
+375 −16
Draft

Cache dryrun results locally #8104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fe72df5
Cache dryrun results locally
scholtzan Sep 16, 2025
4391661
[run-tests] formatting
scholtzan Sep 16, 2025
443ba41
[run-tests] formatting
scholtzan Sep 16, 2025
a25ed39
Fix cache key
scholtzan Sep 16, 2025
4d3c61f
[run-tests] formatting
scholtzan Sep 16, 2025
11f23f4
[run-tests] enable cache use for dryrun
scholtzan Sep 17, 2025
6af651d
Remove outdated test
scholtzan Sep 17, 2025
037db4e
[run-tests] log cache key
scholtzan Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 31 additions & 15 deletions .circleci/workflows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,16 @@ executors:
machine:
image: ubuntu-2404:current

# environment variables for caching
environment: &cache_environment
DRYRUN_CACHE_ENABLED: "true"
DRYRUN_CACHE_DIR: "/tmp/workspace/dryrun_cache"

jobs:
build:
docker: &docker
- image: python:<< pipeline.parameters.python-version >>
environment: *cache_environment
steps:
- checkout
- &restore_venv_cache
Expand Down Expand Up @@ -166,7 +172,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- &attach_generated_sql
- &attach_workspace_artifacts
attach_workspace:
at: /tmp/workspace
- &copy_staged_sql
Expand All @@ -181,6 +187,11 @@ jobs:
name: Run SQL tests
command: |
PATH="venv/bin:$PATH" script/entrypoint -m sql -n 8 -p no:bigquery_etl.pytest_plugin.routine
- &persist_dryrun_cache
persist_to_workspace:
root: /tmp/workspace
paths:
- dryrun_cache
- &copy_debug_sql
run:
name: Copy generated SQL to save for debugging
Expand Down Expand Up @@ -214,7 +225,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_staged_sql
- *authenticate
- run:
Expand Down Expand Up @@ -242,8 +253,9 @@ jobs:
PATHS="sql/bigquery-etl-integration-test"
fi
echo $PATHS
PATH="venv/bin:$PATH" script/bqetl dryrun --validate-schemas $PATHS
PATH="venv/bin:$PATH" script/bqetl dryrun --use-cache --validate-schemas $PATHS
# yamllint enable rule:line-length
- *persist_dryrun_cache
- *copy_debug_sql
- *store_debug_artifacts
- unless:
Expand All @@ -262,7 +274,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- &copy_generated_sql
run:
name: Move generated-sql into place
Expand Down Expand Up @@ -293,7 +305,7 @@ jobs:
name: Validate workgroup access configuration on main
command: |
PATH="venv/bin:$PATH" script/bqetl metadata validate-workgroups sql/
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_staged_sql
- *authenticate
- run:
Expand Down Expand Up @@ -341,7 +353,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *authenticate
- &add_private_bigquery_etl_ssh_keys
add_ssh_keys:
Expand Down Expand Up @@ -391,7 +403,7 @@ jobs:
name: Pull telemetry-airflow
command: |
git clone https://github.com/mozilla/telemetry-airflow.git ~/telemetry-airflow
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_generated_sql
- run:
name: Replace telemetry-airflow DAGs with BigQuery ETL DAGs
Expand Down Expand Up @@ -441,7 +453,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_staged_sql
- run:
name: Run routine tests
Expand All @@ -468,7 +480,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_staged_sql
- *authenticate
- run:
Expand All @@ -491,7 +503,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- add_ssh_keys:
fingerprints:
- "22:b9:3c:1b:82:ab:3f:e4:b5:79:70:d1:7b:b9:28:d2"
Expand Down Expand Up @@ -520,6 +532,7 @@ jobs:
- *restore_venv_cache
- *build
- *authenticate
- *attach_workspace_artifacts
- run:
name: Generate SQL content
command: |
Expand Down Expand Up @@ -614,6 +627,7 @@ jobs:
root: /tmp/workspace
paths:
- generated-sql
- dryrun_cache
- unless:
condition: *validate-sql-or-routines
steps:
Expand All @@ -628,7 +642,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *copy_generated_sql
- add_ssh_keys:
fingerprints:
Expand Down Expand Up @@ -664,6 +678,7 @@ jobs:
root: /tmp/workspace
paths:
- staged-generated-sql
- dryrun_cache
- *copy_debug_sql
- *store_debug_artifacts
- unless:
Expand All @@ -678,7 +693,7 @@ jobs:
or:
- << pipeline.parameters.deploy >>
steps:
- *attach_generated_sql
- *attach_workspace_artifacts
- add_ssh_keys:
fingerprints:
- "22:b9:3c:1b:82:ab:3f:e4:b5:79:70:d1:7b:b9:28:d2"
Expand Down Expand Up @@ -790,7 +805,7 @@ jobs:
- checkout
- *restore_venv_cache
- *build
- *attach_generated_sql
- *attach_workspace_artifacts
- *authenticate
- *add_private_bigquery_etl_ssh_keys
- run:
Expand Down Expand Up @@ -824,6 +839,7 @@ jobs:
paths:
- private-generated-sql
- PRIVATE_BIGQUERY_ETL_SHA
- dryrun_cache
- unless:
condition: *validate-sql
steps:
Expand All @@ -834,7 +850,7 @@ jobs:
- when:
condition: *deploy
steps:
- *attach_generated_sql
- *attach_workspace_artifacts
- *add_private_bigquery_etl_ssh_keys
- run:
name: Push to private-generated-sql branch
Expand Down Expand Up @@ -871,7 +887,7 @@ jobs:
condition: *deploy
steps:
- checkout
- *attach_generated_sql
- *attach_workspace_artifacts
- run:
name: Move generated-sql into place
command: |
Expand Down
29 changes: 29 additions & 0 deletions bigquery_etl/cli/dryrun.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,33 @@
help="GCP project to perform dry run in when --use_cloud_function=False",
default=ConfigLoader.get("default", "project", fallback="moz-fx-data-shared-prod"),
)
@click.option(
"--use-cache/--no-cache",
help="Enable or disable local caching of dry run results. Default is --no-cache.",
default=ConfigLoader.get("dry_run", "cache_enabled", fallback=False),
)
@click.option(
"--cache-ttl-hours",
help="Cache time-to-live in hours. Default is 1 hour.",
type=int,
default=ConfigLoader.get("dry_run", "cache_ttl_hours", fallback=1),
)
@click.option(
"--cache-dir",
help="Directory to store cache files. Default is system temp directory.",
type=click.Path(),
default=None,
)
@billing_project_option()
def dryrun(
paths: List[str],
use_cloud_function: bool,
validate_schemas: bool,
respect_skip: bool,
project: str,
use_cache: bool,
cache_ttl_hours: int,
cache_dir: str,
billing_project: str,
):
"""Perform a dry run."""
Expand Down Expand Up @@ -118,6 +138,9 @@ def dryrun(
use_cloud_function,
respect_skip,
validate_schemas,
use_cache,
cache_ttl_hours,
cache_dir,
credentials=credentials,
id_token=id_token,
billing_project=billing_project,
Expand All @@ -141,6 +164,9 @@ def _sql_file_valid(
use_cloud_function,
respect_skip,
validate_schemas,
use_cache,
cache_ttl_hours,
cache_dir,
sqlfile,
credentials,
id_token,
Expand All @@ -154,6 +180,9 @@ def _sql_file_valid(
respect_skip=respect_skip,
id_token=id_token,
billing_project=billing_project,
cache_enabled=use_cache,
cache_ttl_hours=cache_ttl_hours,
cache_dir=cache_dir,
)
if validate_schemas:
try:
Expand Down
Loading