Multi-relation support, a pre-requisite for views and sub-queries #136780

craigtaverner · 2025-10-19T15:36:52Z

This is extracted from the views prototype at #134995. It provides the underlying support for multiple FROM ... commands within the same ES|QL query, something that both non-materialized views and subqueries require.

A key feature of this work is that all UnresolvedRelation instances are treated equivalently, which means they all support CPS and CPS just as before. This implies that sub-queries using this will support CCS/CPS directly, and for views it means:

The underlying indexes within the view definitions will support CCS/CPS as if they were in the original FROM command
Views are strictly resolved in the coordinator of the local cluster, so that the indexes defined within the views can then be resolved using CCS/CPS rules as normal.

For reviewing this PR, it should be noted that most of the changes are in tests that had mistakes that went unnoticed because they only ever expected a single IndexPattern, and so could get away with mocking a single IndexResolution even if it was for a different index pattern than expressed in the test. The mismatch in index name between FROM and IndexResolution had no consequences. This is no longer possible since we need the IndexPattern to differentiate between different FROM commands. If you want to focus on the functional changes in this PR, look mostly at EsqlSession, AnalyzerContext, Analyzer and EsqlCCSUtils.

elasticsearchmachine · 2025-10-19T15:37:16Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

github-actions · 2025-10-19T15:38:34Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

This is extracted from the views prototype, which also includes necessary support for sub-queries.

fang-xing-esql

The changes to Analyzer, AnalyzerContext and PreAnalyzer make sense to me. I applied the changes to the subquery PR, and subqueries can be resolved properly, thank you @craigtaverner !

The changes to EsqlSession and EsqlCCSUtils also look fine to me.

smalyshev · 2025-10-21T22:07:30Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/inlinestats.csv-spec

 ;

-country:text      |language_name:keyword |MAX(language_code):integer |language_code:integer
+country:keyword   |language_name:keyword |MAX(language_code):integer |language_code:integer


I am curious - why this change is happening in this pull?

smalyshev · 2025-10-21T22:25:57Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+        }
+    }
+
+    static void handleFieldCapsFailures(


Not sure I get this part - why do we need two functions?

smalyshev · 2025-10-22T00:07:53Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

        }

-        private LogicalPlan resolveInsist(Insist insist, List<Attribute> childrenOutput, IndexResolution indexResolution) {
+        private LogicalPlan resolveInsist(Insist insist, List<Attribute> childrenOutput, Collection<IndexResolution> indexResolution) {


Maybe we should use plural here if we're passing a collection now?

smalyshev · 2025-10-22T00:11:31Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java


            // Field is partially unmapped.
-            if (resolvedCol instanceof FieldAttribute fa && indexResolution.get().isPartiallyUnmappedField(fa.name())) {
+            // TODO: Should the check for partially unmapped fields be done specific to each sub-query in a fork?


I'm not sure I understand this part correctly - if multiple resolutions come with subqueries, aren't they scoped within the subqueries? How does it work - if I have a mapping in index in one subquery, I can't use it in another subquery, can I? What about propagating mappings up/down the subquery tree?

smalyshev · 2025-10-22T00:12:13Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

 ignoreOrder:true
 language_code:integer | country:text       | language_name:keyword
-2                     | [Austria, Germany] | German
+2                     | [Germany, Austria] | German


Why the order change?

smalyshev · 2025-10-22T00:16:19Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

    public record PreAnalysis(
-        IndexMode indexMode,
-        IndexPattern indexPattern,
+        Map<IndexPattern, IndexMode> indexes,


I think Elastic code uses "indices" not "indexes"?

smalyshev · 2025-10-22T00:20:42Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/AnalyzerContext.java

    EsqlFunctionRegistry functionRegistry,
-    IndexResolution indexResolution,
+    Map<IndexPattern, IndexResolution> indexResolution,
    Map<String, IndexResolution> lookupResolution,


I think here a comment why we index one by String and another by IndexPattern would help. I think I understand why (because lookup only uses single index while general case can have arbitrary expressions) but it took me some work to figure it out.

smalyshev · 2025-10-22T00:26:27Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

-                indexMode.set(p.indexMode());
-                indexPattern.set(p.indexPattern());
+            } else if (indexes.containsKey(p.indexPattern()) == false || indexes.get(p.indexPattern()) == p.indexMode()) {
+                indexes.put(p.indexPattern(), p.indexMode());


Not sure I get why we do the put if we already have exactly the same element already. Can't we simplify this as:

} else { IndexMode previous = indexes.put(p.indexPattern(), p.indexMode()); if(previous != null && previous != p.indexMode()) { throw new IllegalStateException( "index pattern '" + p.indexPattern() + "' found with with different index mode: " + previous + " != " + p.indexMode() ); } }

Also, when exactly can such a situation happen anyway? And why lookup mode is excluded from the check?

smalyshev · 2025-10-22T00:37:22Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

        Holder<Boolean> supportsDenseVector = new Holder<>(false);
+        indexes.forEach((ip, mode) -> {
+            if (mode == IndexMode.TIME_SERIES) {
+                supportsAggregateMetricDouble.set(true);


Here I am not sure how it's supposed to work with subqueries. Should this be enabled for the whole query, or only for the subquery that this particular pattern belongs to? Maybe this is out of scope for this patch, just trying to figure out the concept.

smalyshev · 2025-10-22T00:38:32Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/UnionAll.java

@@ -0,0 +1,127 @@
+/*


A bit confused about how UnionAll relates to the rest of the patch. It's here but nothing else is mentioning or using it. Am I missing something in the picture?

craigtaverner requested review from alex-spies, fang-xing-esql and idegtiarenko October 19, 2025 15:36

craigtaverner added >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Oct 19, 2025

elasticsearchmachine added the v9.3.0 label Oct 19, 2025

craigtaverner requested a review from smalyshev October 19, 2025 15:37

github-actions bot deployed to docs-preview October 19, 2025 15:37 View deployment

craigtaverner force-pushed the multi_es_relation branch from aa533ad to 1fc334e Compare October 19, 2025 15:48

github-actions bot deployed to docs-preview October 19, 2025 15:48 View deployment

craigtaverner added 2 commits October 20, 2025 15:11

Make EsqlSpecTestCase more resilient

ef5ab4e

Multi-relation support, a Pre-requisite for views and sub-queries

9352610

This is extracted from the views prototype, which also includes necessary support for sub-queries.

craigtaverner force-pushed the multi_es_relation branch from 1fc334e to 9352610 Compare October 20, 2025 13:11

fang-xing-esql reviewed Oct 20, 2025

View reviewed changes

craigtaverner mentioned this pull request Oct 21, 2025

ES|QL Views prototype #134995

Draft

20 tasks

smalyshev reviewed Oct 22, 2025

View reviewed changes

Multi-relation support, a pre-requisite for views and sub-queries #136780

Are you sure you want to change the base?

Multi-relation support, a pre-requisite for views and sub-queries #136780

Conversation

craigtaverner commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 19, 2025

Uh oh!

github-actions bot commented Oct 19, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

craigtaverner commented Oct 19, 2025 •

edited

Loading