feat: improve runtime filter [Part 1] #18893
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
pr-feature
SkyFan2002
force-pushed
the
improve_runtime_filter
branch
from
4707bac to
a8bd6cb
Compare
SkyFan2002
added
ci-cloud
Contributor
Docker Image for PR
|
SkyFan2002
added
ci-cloud
Contributor
Docker Image for PR
|
SkyFan2002
added
ci-cloud
Contributor
Docker Image for PR
|
Member
Author
|
Marked as draft due to some tests failing. |
zhang2014
approved these changes
Nov 6, 2025
…2/databend into improve_runtime_filter
# Conflicts: # src/query/storages/fuse/src/operations/read/native_data_source_deserializer.rs # src/query/storages/fuse/src/operations/read/parquet_data_source_deserializer.rs
# Conflicts: # src/query/service/tests/it/sql/planner/optimizer/data/results/obfuscated/01_multi_join_avg_case_expression_physical.txt # src/query/service/tests/it/sql/planner/optimizer/data/results/obfuscated/01_multi_join_sum_case_expression_physical.txt # src/query/service/tests/it/sql/planner/optimizer/data/results/tpcds/Q01_physical.txt # src/query/service/tests/it/sql/planner/optimizer/data/results/tpcds/Q03_physical.txt # src/query/storages/fuse/src/operations/read/native_data_source_deserializer.rs # src/query/storages/fuse/src/operations/read/parquet_data_source_deserializer.rs # tests/sqllogictests/suites/mode/cluster/shuffle.test # tests/sqllogictests/suites/mode/standalone/ee/explain_virtual_column.test # tests/sqllogictests/suites/mode/standalone/explain/explain.test # tests/sqllogictests/suites/mode/standalone/explain/index/explain_agg_index.test # tests/sqllogictests/suites/mode/standalone/explain/index/explain_vector_index.test # tests/sqllogictests/suites/mode/standalone/explain/lateral.test # tests/sqllogictests/suites/mode/standalone/explain/limit.test # tests/sqllogictests/suites/mode/standalone/explain/push_down_filter/push_down_filter_eval_scalar.test # tests/sqllogictests/suites/mode/standalone/explain/select.test # tests/sqllogictests/suites/mode/standalone/explain/subquery.test # tests/sqllogictests/suites/mode/standalone/explain_native/limit.test # tests/sqllogictests/suites/mode/standalone/explain_native/push_down_filter/push_down_filter_eval_scalar.test # tests/sqllogictests/suites/mode/standalone/explain_native/subquery.test # tests/sqllogictests/suites/no_table_meta_cache/col_stats_of_all_null.test
sundy-li
reviewed
Nov 10, 2025
sundy-li
approved these changes
Nov 11, 2025
# Conflicts: # src/query/service/src/physical_plans/physical_join_filter.rs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
This PR improves the runtime filter mechanism with several optimizations to enhance join performance and reduce unnecessary filtering overhead.
Changes
Push down runtime filters to probe key equivalence classes - Extends runtime filter coverage by propagating filters through equivalent expressions, enabling more join scenarios to benefit from early filtering. Performance Optimization: Propagate Runtime Filters Through Join Equivalence Classes #18857
Scan waits for runtime filter construction - Synchronizes table scans with runtime filter generation to ensure filters are applied before reading data, maximizing filtering effectiveness.
Runtime-based filter construction decision - Dynamically determines whether to build runtime filters based on runtime statistics, avoiding overhead when filters provide minimal benefit.
Performance
TPC-DS SF100, Large Warehouse:
Tests
Type of change
This change is