From 74d9e418e65bb63122bbcb210b178d2e4cb81f5c Mon Sep 17 00:00:00 2001 From: elasticsearchmachine Date: Tue, 21 Oct 2025 19:34:45 +0000 Subject: [PATCH 1/3] Update docs for v9.2.0 release --- docs/release-notes/breaking-changes.md | 20 +- .../release-notes/changelog-bundles/9.2.0.yml | 1579 +++++++++++++++++ docs/release-notes/deprecations.md | 14 +- docs/release-notes/index.md | 543 +++++- 4 files changed, 2119 insertions(+), 37 deletions(-) create mode 100644 docs/release-notes/changelog-bundles/9.2.0.yml diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md index 7177b40b12407..b5bd003b28a00 100644 --- a/docs/release-notes/breaking-changes.md +++ b/docs/release-notes/breaking-changes.md @@ -12,14 +12,30 @@ If you are migrating from a version prior to version 9.0, you must first upgrade % ## Next version [elasticsearch-nextversion-breaking-changes] -## 9.1.5 [elasticsearch-9.1.5-breaking-changes] +```{applies_to} +stack: ga 9.2.0 +``` +## 9.2.0 [elasticsearch-9.2.0-breaking-changes] + +Ingest Node: +* Simulate API: Return 400 on invalid processor(s) [#130325](https://github.com/elastic/elasticsearch/pull/130325) (issue: [#120731](https://github.com/elastic/elasticsearch/issues/120731)) + +Mapping: +* Don't enable norms for fields of type text when the index mode is LogsDB or TSDB [#131317](https://github.com/elastic/elasticsearch/pull/131317) + +Vector Search: +* Enable `exclude_source_vectors` by default for new indices [#131907](https://github.com/elastic/elasticsearch/pull/131907) + -There are no breaking changes associated with this release. ## 9.0.8 [elasticsearch-9.0.8-breaking-changes] There are no breaking changes associated with this release. +## 9.1.5 [elasticsearch-9.1.5-breaking-changes] + +There are no breaking changes associated with this release. + ## 9.1.4 [elasticsearch-9.1.4-breaking-changes] There are no breaking changes associated with this release. diff --git a/docs/release-notes/changelog-bundles/9.2.0.yml b/docs/release-notes/changelog-bundles/9.2.0.yml new file mode 100644 index 0000000000000..24dfa95bd78fa --- /dev/null +++ b/docs/release-notes/changelog-bundles/9.2.0.yml @@ -0,0 +1,1579 @@ +version: 9.2.0 +released: false +generated: 2025-10-21T19:34:33.426960938Z +changelogs: + - pr: 74274 + summary: Introduce new rescorer based on script + area: Search + type: feature + issues: + - 52338 + - pr: 92568 + summary: Support nested fields for term vectors API when using artificial documents + area: Search + type: enhancement + issues: + - 91902 + - pr: 105718 + summary: Add option for Append Processor to skip/allow empty values + area: Ingest Node + type: enhancement + issues: + - 104813 + - pr: 113757 + summary: Added AI21 Completion and Chat Completion support to the Inference Plugin + area: Machine Learning + type: enhancement + issues: [] + - pr: 113949 + summary: Support kNN filter on nested metadata + area: Vector Search + type: enhancement + issues: + - 128803 + - 106994 + - pr: 125552 + summary: Expose S3 connection max idle time as a setting + area: Snapshot/Restore + type: enhancement + issues: [] + - pr: 125921 + summary: Allow direct IO for BBQ rescoring + area: Vector Search + type: feature + issues: [] + highlight: + notable: false + title: Allow direct IO for BBQ rescoring + body: |- + BBQ rescoring performance can be drastically affected by the amount of available + off-heap RAM for use by the system page cache. When there is not enough off-heap RAM + to fit all the vector data in memory, BBQ search latencies can be affected by as much as 5000x. + Specifying the `vector.rescoring.directio=true` Java option on all vector search + nodes modifies rescoring to use direct IO, which eliminates these very high latencies + from searches in low-memory scenarios, at a cost of a reduction + in vector search performance for BBQ indices when the vectors do all fit in memory. + + This option is released in 9.1 as a tech preview whilst we analyse its effect + for a variety of use cases. + pr: 125921 + - pr: 127223 + summary: Wrap ES KNN queries with PatienceKNN query + area: Vector Search + type: feature + issues: [] + - pr: 128627 + summary: Fail `profile` on text response formats + area: ES|QL + type: feature + issues: [] + - pr: 128639 + summary: Substitue `date_trunc` with `round_to` when the pre-calculated rounding points are available + area: ES|QL + type: enhancement + issues: [] + - pr: 128866 + summary: Add `age_in_millis` to ILM Explain Response + area: ILM+SLM + type: enhancement + issues: + - 103659 + - pr: 128917 + summary: Adopt a "LogicalPlan" approach to running multiple sub-queries (with INLINESTATS so far) + area: ES|QL + type: enhancement + issues: [] + - pr: 129003 + summary: Allow timeout during trained model download process + area: Machine Learning + type: bug + issues: [] + - pr: 129013 + summary: Add remote index support to LOOKUP JOIN + area: ES|QL + type: feature + issues: [] + highlight: + notable: false + title: Add remote index support to LOOKUP JOIN + body: |- + Queries containing LOOKUP JOIN now can be preformed on cross-cluster indices, for example: + [source,yaml] + ---------------------------- + FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100 + ---------------------------- + pr: 129013 + - pr: 129090 + summary: Enable force inference endpoint deleting for invalid models and after stopping model deployment fails + area: Machine Learning + type: enhancement + issues: [] + - pr: 129146 + summary: "[ML] Add IBM watsonx Completion and Chat Completion support to the Inference Plugin" + area: Machine Learning + type: enhancement + issues: [] + - pr: 129369 + summary: Support semantic reranking using contextual snippets instead of entire field text + area: Relevance + type: enhancement + issues: [] + - pr: 129581 + summary: "Support geohash, geotile and geohex grid types" + area: ES|QL + type: enhancement + issues: [] + - pr: 129662 + summary: "[Security] Add entity store and asset criticality index privileges to built in Editor, Viewer and Kibana System roles" + area: Authorization + type: enhancement + issues: [] + - pr: 129693 + summary: Add top level normalizer for linear retriever + area: Search + type: enhancement + issues: [] + - pr: 129848 + summary: "[ML] Add Azure AI Rerank support to the Inference Plugin" + area: Machine Learning + type: enhancement + issues: [] + - pr: 129872 + summary: Run `TransportClusterStateAction` on local node + area: Distributed + type: enhancement + issues: [] + - pr: 129929 + summary: Add support for RLIKE (LIST) with pushdown + area: ES|QL + type: enhancement + issues: [] + - pr: 129945 + summary: "Enhancement: ILM sets `indexing_complete` to true from `ReadOnly` action" + area: ILM+SLM + type: enhancement + issues: [] + - pr: 129967 + summary: Support returning default `index_options` for `semantic_text` fields when `include_defaults` is true + area: Search + type: bug + issues: [] + - pr: 129990 + summary: Make forecast write load accurate when shard numbers change + area: Allocation + type: bug + issues: [] + - pr: 130027 + summary: "Fix: prevent duplication of \"invalid index name\" string in the final exception error message" + area: ES|QL + type: bug + issues: [] + - pr: 130092 + summary: Added Llama provider support to the Inference Plugin + area: Machine Learning + type: enhancement + issues: [] + - pr: 130140 + summary: Correct slow log user for RCS 2.0 + area: Authentication + type: enhancement + issues: [] + - pr: 130251 + summary: Speed up (filtered) KNN queries for flat vector fields + area: Vector Search + type: enhancement + issues: [] + - pr: 130325 + summary: "Simulate API: Return 400 on invalid processor(s)" + area: Ingest Node + type: bug + issues: + - 120731 + breaking: + area: Ingest + title: Return 400 on invalid processor(s) in Simulate API + details: "In earlier versions of {es}, the Simulate API would return a 500 error when encountering invalid processors. Now, it returns a 400 Bad Request error instead." + impact: Callers should expect a 400 Bad Request response when the Simulate API encounters invalid processors. This change improves error handling and provides clearer feedback on request issues. + notable: false + essSettingChange: false + - pr: 130330 + summary: "TopNOperator, release Row on failure" + area: ES|QL + type: bug + issues: + - 130215 + - pr: 130382 + summary: Remove vectors from `_source` transparently + area: Vector Search + type: enhancement + issues: [] + - pr: 130409 + summary: Add Dependency Checker for `LogicalLocalPlanOptimizer` + area: ES|QL + type: enhancement + issues: [] + - pr: 130463 + summary: Refresh potential lost connections at query start for `_search` + area: Search + type: enhancement + issues: [] + - pr: 130485 + summary: Add `RerankRequestChunker` + area: Machine Learning + type: enhancement + issues: [] + - pr: 130510 + summary: Add fast path for single value in VALUES aggregator + area: ES|QL + type: enhancement + issues: [] + - pr: 130531 + summary: Adding check for `isIndexed` in text fields when generating field exists queries to avoid ISE when field is stored but not indexed or with `doc_values` + area: Analysis + type: bug + issues: [] + - pr: 130544 + summary: Sync Inference with Trained Model stats + area: Machine Learning + type: bug + issues: + - 130339 + - pr: 130593 + summary: Add new `CachePopulationReason` + area: Store + type: enhancement + issues: [] + - pr: 130594 + summary: Add audit logging for stream content + area: Network + type: enhancement + issues: [] + - pr: 130635 + summary: "Add low-level optimized Neon, AVX2, and AVX 512 float32 vector operations" + area: Vector Search + type: enhancement + issues: [] + - pr: 130658 + summary: Add support for weighted RRF in retrievers + area: Relevance + type: enhancement + issues: [] + - pr: 130824 + summary: Add read permissions for osquery manager result indices + area: Security + type: enhancement + issues: [] + - pr: 130834 + summary: Ensure vectors are always included in reindex actions + area: Vector Search + type: enhancement + issues: [] + - pr: 130847 + summary: "Pipelines: Add `created_date` and `modified_date`" + area: Ingest Node + type: enhancement + issues: [] + - pr: 130855 + summary: Add checks that optimizers do not modify the layout + area: ES|QL + type: enhancement + issues: + - 125576 + - pr: 130909 + summary: Allow adjustment of transport TLS handshake timeout + area: Network + type: enhancement + issues: [] + - pr: 130939 + summary: Expose HTTP connection metrics to telemetry + area: Network + type: enhancement + issues: [] + - pr: 130940 + summary: Block trained model updates from inference + area: Machine Learning + type: enhancement + issues: + - 129999 + - pr: 130944 + summary: Remove unnecessary calls to Fold + area: ES|QL + type: enhancement + issues: + - 119756 + - pr: 130947 + summary: "[main]Prepare Index Like fix for backport to 9.1 and 8.19" + area: ES|QL + type: bug + issues: [] + - pr: 131027 + summary: Handle structured log messages + area: Ingest Node + type: feature + issues: + - 130333 + - pr: 131056 + summary: Add existing shards allocator settings to failure store allowed list + area: Data streams + type: bug + issues: [] + - pr: 131058 + summary: Adds sparse vector index options settings to semantic_text field + area: Search + type: enhancement + issues: [] + - pr: 131061 + summary: Speed up reading multivalued keywords + area: ES|QL + type: enhancement + issues: [] + - pr: 131173 + summary: Add attribute count to `SamlAttribute` `toString` + area: Authentication + type: enhancement + issues: [] + - pr: 131200 + summary: Improve lost-increment message in repo analysis + area: Snapshot/Restore + type: enhancement + issues: [] + - pr: 131204 + summary: Allow pruning columns added by `InlineJoin` + area: ES|QL + type: enhancement + issues: [] + - pr: 131261 + summary: Enable Failure Store for new logs-*-* data streams + area: Data streams + type: feature + issues: + - 131105 + highlight: + notable: true + title: Enable Failure Store for new logs data streams + body: "The [Failure Store](docs-content://manage-data/data-store/data-streams/failure-store.md) is now enabled by default for new logs data streams matching the pattern `logs-*-*`. This means that such data streams will now store invalid documents in a\ndedicated failure index instead of rejecting them, allowing better visibility and control over data quality issues without loosing data. This can be [enabled manually](docs-content://manage-data/data-store/data-streams/failure-store.md#set-up-failure-store-existing) for existing data streams. \nNote: With the failure store enabled, the http response code clients receive when indexing invalid documents will change from `400 Bad Request` to `201 Created`, with an additional response attribute `\"failure_store\" : \"used\"`. " + pr: 131261 + - pr: 131296 + summary: Enable failure store for newly created APM datastreams + area: Ingest Node + type: enhancement + issues: [] + - pr: 131317 + summary: Don't enable norms for fields of type text when the index mode is LogsDB or TSDB + area: Mapping + type: breaking + issues: [] + breaking: + area: Mapping + title: Don't enable norms for fields of type text when the index mode is LogsDB or TSDB + details: "This changes the default behavior for norms on `text` fields in logsdb and tsdb indices. Prior to this change, norms were enabled by default, with the option to disable them via manual configurations. After this change, norms will be disabled by default. Note, because we dont support enabling norms from a disabled state, users will not be able to enable norms on `text` fields in logsdb and tsdb indices." + impact: Text fields will no longer be normalized by default in LogsDB and TSDB indicies. + notable: false + essSettingChange: false + - pr: 131341 + summary: Consider min/max from predicates when transform date_trunc/bucket to `round_to` + area: ES|QL + type: enhancement + issues: [] + - pr: 131390 + summary: Add optimized path for intermediate values aggregator + area: ES|QL + type: enhancement + issues: [] + - pr: 131395 + summary: Enable failure store for newly created OTel data streams + area: Data streams + type: enhancement + issues: [] + - pr: 131442 + summary: Track inference deployments + area: Machine Learning + type: enhancement + issues: [] + - pr: 131485 + summary: Run single phase aggregation when possible + area: ES|QL + type: enhancement + issues: [] + - pr: 131510 + summary: Upgrade apm-agent to 1.55.0 + area: Infra/Metrics + type: upgrade + issues: [] + - pr: 131517 + summary: Refresh potential lost connections at query start for field caps + area: Search + type: enhancement + issues: [] + - pr: 131531 + summary: Integrate LIKE/RLIKE LIST with `ReplaceStringCasingWithInsensitiveRegexMatch` rule + area: ES|QL + type: enhancement + issues: [] + - pr: 131536 + summary: "Component Templates: Add created and modified date" + area: Ingest Node + type: enhancement + issues: [] + - pr: 131551 + summary: Added support to configure query timeout for inference + area: Inference + type: enhancement + issues: [] + - pr: 131559 + summary: Add support for LOOKUP JOIN on multiple fields + area: ES|QL + type: enhancement + issues: [] + highlight: + notable: true + title: Add support for Lookup Join on Multiple Fields + body: |- + Add support for Lookup Join on Multiple Fields e.g. FROM index1 + | LOOKUP JOIN lookup_index on field1, field2 + pr: 131559 + - pr: 131581 + summary: Support Fields API in conditional ingest processors + area: Infra/Core + type: enhancement + issues: [] + - pr: 131599 + summary: Speed up `OptimizedScalarQuantizer` + area: Vector Search + type: enhancement + issues: [] + - pr: 131641 + summary: Add exception for perform embedding inference requests with query provided + area: Machine Learning + type: bug + issues: [] + - pr: 131694 + summary: Accept unsigned longs on MAX and MIN aggregations + area: ES|QL + type: enhancement + issues: [] + - pr: 131706 + summary: Add extension points to remediate index metadata in during snapshot restore + area: Snapshot/Restore + type: enhancement + issues: [] + - pr: 131723 + summary: Tests for FORK's evaluation of field names used in `field_caps` resolve calls + area: Search + type: bug + issues: [] + - pr: 131733 + summary: Replace `RoundTo` linear search evaluator with manual evaluators + area: ES|QL + type: enhancement + issues: [] + - pr: 131775 + summary: Replace "representable" type error messages + area: ES|QL + type: enhancement + issues: [] + - pr: 131907 + summary: Enable `exclude_source_vectors` by default for new indices + area: Vector Search + type: breaking + issues: [] + breaking: + area: Search + title: Enable `exclude_source_vectors` by default for new indices + details: |- + The `exclude_source_vectors` setting is now enabled by default for newly created indices. + This means that vector fields (e.g., `dense_vector`) are no longer stored in the `_source` field + by default, although they remain fully accessible through search and retrieval operations. + + Instead of being persisted in `_source`, vectors are now rehydrated on demand from the underlying + index structures when needed. This reduces index size and improves performance for typical vector + search workloads where the original vector values do not need to be part of the `_source`. + + If your use case requires vector fields to be stored in `_source`, you can disable this behavior by + setting `exclude_source_vectors: false` at index creation time. + impact: |- + Vector fields will no longer be stored in `_source` by default for new indices. Applications or tools + that expect to see vector fields in `_source` (for raw document inspection) + may need to be updated or configured to explicitly retain vectors using `exclude_source_vectors: false`. + + Retrieval of vector fields via search or the `_source` API remains fully supported. + notable: true + essSettingChange: false + - pr: 131937 + summary: Fix race condition in `RemoteClusterService.collectNodes()` + area: Distributed + type: bug + issues: [] + - pr: 131940 + summary: Allow remote enrich after LOOKUP JOIN + area: ES|QL + type: enhancement + issues: [] + - pr: 132003 + summary: Add `copy_from` option to the Append processor + area: Ingest Node + type: enhancement + issues: [] + - pr: 132011 + summary: Restrict Indexing To Child Streams When Streams Is Enabled + area: Data streams + type: enhancement + issues: [] + - pr: 132048 + summary: Fix stuck in STOPPING by retrying the startup task indefinitely until it succeeds + area: Transform + type: bug + issues: + - 128221 + - pr: 132064 + summary: Only Allow Enabling Streams If No Conflicting Indices Exist + area: Data streams + type: enhancement + issues: [] + - pr: 132083 + summary: "Index template: Add created_date and modified_date" + area: Ingest Node + type: enhancement + issues: [] + - pr: 132131 + summary: Updating `TransportSimulateIndexTemplateAction.resolveTemplate()` to account for data stream overrides + area: Indices APIs + type: bug + issues: + - 131425 + - pr: 132138 + summary: Fix lookup index resolution when field-caps returns empty mapping + area: ES|QL + type: bug + issues: + - 132105 + - pr: 132143 + summary: Consider min/max from predicates when transform date_trunc/bucket to `round_to` option 2 + area: ES|QL + type: enhancement + issues: [] + - pr: 132210 + summary: Adding a `merge_type` parameter to the ingest simulate API + area: Ingest Node + type: enhancement + issues: + - 131608 + - pr: 132243 + summary: Fix `NullPointerException` in transport trace logger + area: Network + type: bug + issues: [] + - pr: 132321 + summary: "Avoid internal server error when suggester requires unigrams but no unigrams are provided, return bad request instead" + area: Analysis + type: bug + issues: + - 131928 + - pr: 132362 + summary: Inference API disable partial search results + area: Machine Learning + type: bug + issues: [] + - pr: 132408 + summary: Correct exception for missing nested path + area: Search + type: bug + issues: [] + - pr: 132410 + summary: Add support for retrieving semantic_text's indexed chunks via fields API + area: Vector Search + type: feature + issues: [] + - pr: 132414 + summary: Adjust date docvalue formatting to return 4xx instead of 5xx + area: Search + type: bug + issues: [] + - pr: 132456 + summary: Some optimizations for constant blocks + area: ES|QL + type: enhancement + issues: [] + - pr: 132497 + summary: Add cache miss and read metrics + area: Searchable Snapshots + type: enhancement + issues: [] + - pr: 132506 + summary: Add .integration_knowledge system index for usage by AI assistants + area: Infra/Core + type: feature + issues: [] + - pr: 132511 + summary: Handle special regex cases for version fields + area: Search + type: bug + issues: [] + - pr: 132512 + summary: Rewrite `RoundTo` to `QueryAndTags` + area: ES|QL + type: enhancement + issues: [] + - pr: 132536 + summary: Avoid stack overflow in `IndicesClusterStateService` `applyClusterState` + area: Cluster Coordination + type: bug + issues: [] + - pr: 132547 + summary: Add epoch blob-cache metric + area: Searchable Snapshots + type: enhancement + issues: [] + - pr: 132548 + summary: Have top level knn searches tracked in query stats + area: Vector Search + type: bug + issues: [] + - pr: 132622 + summary: Push down compute engine value loading of long based singleton numeric doc value to the es819 tsdb doc values codec. + area: Codec + type: enhancement + issues: [] + - pr: 132651 + summary: Use java8 variant of apm-agent + area: Infra/Core + type: upgrade + issues: [] + - pr: 132675 + summary: Add second max queue latency stat to `ClusterInfo` + area: Allocation + type: enhancement + issues: [] + - pr: 132680 + summary: Add support for per-field weights in simplified RRF retriever syntax + area: Search + type: enhancement + issues: [] + - pr: 132689 + summary: Add support for dimensions in google vertex ai request + area: Machine Learning + type: enhancement + issues: [] + - pr: 132738 + summary: Fix `AsyncOperator` status values and add emitted rows + area: ES|QL + type: bug + issues: [] + - pr: 132744 + summary: Fix async operator warnings not always sent when blocking + area: ES|QL + type: bug + issues: + - 130642 + - 132554 + - 132778 + - 130296 + - 132555 + - 131563 + - 131148 + - 132604 + - 128030 + - pr: 132765 + summary: Implement `v_magnitude` function + area: ES|QL + type: feature + issues: + - 132768 + - pr: 132774 + summary: Improve cpu utilization with dynamic slice size in doc partitioning + area: ES|QL + type: enhancement + issues: [] + - pr: 132833 + summary: Adding simulate ingest effective mapping + area: Ingest Node + type: enhancement + issues: [] + - pr: 132845 + summary: Expose existing DLS cache x-pack usage statistics + area: Authorization + type: enhancement + issues: [] + - pr: 132858 + summary: Add index mode to resolve index response + area: Indices APIs + type: feature + issues: [] + - pr: 132934 + summary: Support filters on inlinestats + area: ES|QL + type: enhancement + issues: [] + - pr: 132950 + summary: Speed up loading keyword fields with index sorts + area: ES|QL + type: enhancement + issues: [] + - pr: 132959 + summary: Adds the `v_hamming` function for calculating the Hamming distance between two dense vectors + area: ES|QL + type: feature + issues: + - 132056 + - pr: 132967 + summary: ES-11331 streams params restriction + area: Data streams + type: enhancement + issues: [] + - pr: 133016 + summary: Adding Contains ESQL String function + area: ES|QL + type: feature + issues: [] + - pr: 133018 + summary: Add ordinal range encode for tsid + area: TSDB + type: enhancement + issues: [] + - pr: 133030 + summary: Implement `failIfAlreadyExists` in S3 repositories + area: Snapshot/Restore + type: enhancement + issues: + - 128565 + - pr: 133064 + summary: Mark LOOKUP JOIN as `ExecutesOn.Any` by default + area: ES|QL + type: bug + issues: [] + - pr: 133074 + summary: Adds transport-only flag to always include indices in the field caps transport response + area: Mapping + type: enhancement + issues: [] + - pr: 133087 + summary: Esql skip null metrics + area: ES|QL + type: enhancement + issues: + - 129524 + - pr: 133111 + summary: Return 429 instead of 500 for timeout handlers + area: Network + type: enhancement + issues: [] + - pr: 133154 + summary: Allow configuring SAML private attributes + area: Authentication + type: enhancement + issues: [] + - pr: 133166 + summary: Improve Expanding Lookup Join performance by pushing a filter to the right side of the lookup join + area: ES|QL + type: enhancement + issues: [] + - pr: 133193 + summary: Fix offset handling in Murmur3Hasher + area: Infra/Core + type: bug + issues: [] + - pr: 133245 + summary: Add query heads priority to `SliceQueue` + area: ES|QL + type: enhancement + issues: [] + - pr: 133314 + summary: Add hits and misses timing stats to DLS cache + area: Authorization + type: enhancement + issues: [] + - pr: 133357 + summary: Switch to Sending a Bad Request User When Function Score Query Generates Negative Scores + area: Search + type: bug + issues: + - 133358 + - pr: 133360 + summary: Add recover_failure_document processor to remediate failurestore docs + area: Ingest Node + type: feature + issues: [] + - pr: 133365 + summary: Skip iterating DISI when reading metric values + area: Codec + type: enhancement + issues: [] + - pr: 133369 + summary: Enable `date` `date_nanos` implicit casting + area: ES|QL + type: enhancement + issues: [] + - pr: 133397 + summary: Push down loading of singleton dense double based field types to the … + area: Codec + type: enhancement + issues: [] + - pr: 133403 + summary: Fix service destination template file name + area: Data streams + type: bug + issues: [] + - pr: 133546 + summary: "Support geohash, geotile and geohex grid types in ST_INTERSECTS and ST_DISJOINT" + area: ES|QL + type: enhancement + issues: [] + - pr: 133576 + summary: Text similarity reranker chunks and scores snippets + area: Relevance + type: enhancement + issues: [] + - pr: 133599 + summary: Support Gemini thinking budget in inference API + area: Machine Learning + type: enhancement + issues: [] + - pr: 133601 + summary: Fix bug in topn + area: ES|QL + type: bug + issues: + - 133600 + - 133574 + - 133607 + - pr: 133616 + summary: Add mode filter to _resolve/index + area: Indices APIs + type: enhancement + issues: [] + - pr: 133636 + summary: Esql `mv_contains` function + area: ES|QL + type: enhancement + issues: [] + - pr: 133659 + summary: Fixed GeneralScriptException to return 400 http status code + area: Infra/Scripting + type: bug + issues: [] + - pr: 133675 + summary: Support using the semantic query across multiple inference IDs + area: Vector Search + type: enhancement + issues: [] + - pr: 133683 + summary: Avoid running asynchronous ILM actions while ILM is stopped + area: ILM+SLM + type: bug + issues: + - 99859 + - 81234 + - 85097 + - pr: 133718 + summary: Remove upper limit for chunking settings + area: Machine Learning + type: enhancement + issues: [] + - pr: 133720 + summary: Support querying multiple indices with the simplified linear retriever + area: Relevance + type: enhancement + issues: [] + - pr: 133722 + summary: Fix disk usage estimation for SORTED_SET doc values + area: Codec + type: bug + issues: [] + - pr: 133745 + summary: Supporting more timestamp formats in `_text_structure/find_structure` + area: Machine Learning + type: feature + issues: [] + - pr: 133860 + summary: Cache Inference Endpoints + area: Machine Learning + type: enhancement + issues: + - 133135 + - pr: 133861 + summary: Implementing latency improvements for EIS integration + area: Machine Learning + type: bug + issues: [] + - pr: 133897 + summary: "Runtime fields: pass down runtime field name as source filter when source mode is synthetic" + area: Mapping + type: enhancement + issues: [] + - pr: 133916 + summary: Fix model assignment error handling and assignment explanation generation + area: Machine Learning + type: bug + issues: [] + - pr: 133930 + summary: Improve memory estimation methods accuracy in `TrainedModelAssignmentRebalancer` and related classes + area: Machine Learning + type: bug + issues: [] + - pr: 133952 + summary: Fix warning when creating an OTel data stream + area: TSDB + type: bug + issues: + - 132918 + - pr: 133954 + summary: "ILM: Force merge on zero-replica cloned index before snapshotting for searchable snapshots" + area: ILM+SLM + type: enhancement + issues: + - 75478 + - pr: 133968 + summary: Extend kibana-system permissions to manage security entities + area: Infra/Core + type: enhancement + issues: [] + - pr: 133980 + summary: Upgrade elasticsearch to lucene 10.3.0 + area: Search + type: upgrade + issues: [] + highlight: + notable: false + title: New lucene 10.3.0 release + body: |- + - Improved performance for lexical, vector and primary-key searches + - Use optimistic-with-checking KNN Query execution strategy in place of cross-thread global queue min-score checking. Improves performance and consistency. + - Bulk scoring added for floating point vectors in HNSW. Improving query latency and indexing throughput + - Multiple improvements to HNSW graph traversal and storage + pr: 133980 + - pr: 134029 + summary: Add SET instruction + area: ES|QL + type: enhancement + issues: [] + - pr: 134033 + summary: Fix FORK with union-types + area: ES|QL + type: bug + issues: + - 133973 + - pr: 134080 + summary: Added Google Model Garden Anthropic Completion and Chat Completion support to the Inference Plugin + area: Machine Learning + type: enhancement + issues: [] + - pr: 134098 + summary: LOOKUP JOIN with expressions + area: ES|QL + type: enhancement + issues: [] + - pr: 134117 + summary: Improve block loader for source only runtime fields of type long + area: Mapping + type: enhancement + issues: [] + - pr: 134137 + summary: Add signing configuration for cross cluster api keys + area: Security + type: enhancement + issues: [] + - pr: 134152 + summary: Correctly update SLM stats with master shutdown + area: ILM+SLM + type: bug + issues: [] + - pr: 134178 + summary: Return 429 status when `RequestExecutorService` queue full + area: Machine Learning + type: bug + issues: [] + - pr: 134198 + summary: Improve `ShardLockObtainFailedException` message + area: Store + type: enhancement + issues: [] + - pr: 134214 + summary: "[Downsampling++] Add time series telemetry in xpack usage" + area: Downsampling + type: enhancement + issues: + - 133953 + - pr: 134232 + summary: Add relevant attributes to search took time APM metrics + area: Search + type: enhancement + issues: [] + - pr: 134309 + summary: Telemetry with inlinestats + area: ES|QL + type: bug + issues: [] + - pr: 134317 + summary: ES|QL - Allow multivalued query parameters + area: ES|QL + type: enhancement + issues: [] + - pr: 134323 + summary: Make INLINESTATS (and subplans) work with CCS + area: ES|QL + type: enhancement + issues: + - 124748 + - pr: 134349 + summary: Add `LoadedSecureSettings` for keeping temporary secure settings loaded + area: Security + type: enhancement + issues: [] + - pr: 134365 + summary: Remove `PeerFinder` request timeout + area: Cluster Coordination + type: deprecation + issues: [] + deprecation: + area: Cluster and node setting + title: Remove `PeerFinder` request timeout + details: "There is no need to time out requests sent by the `PeerFinder` during discovery and cluster formation, and this timeout may sometimes cause spurious failures. With this change the `PeerFinder` requests will wait indefinitely for responses. The `discovery.request_peers_timeout` setting no longer has any effect." + impact: Discontinue use of the `discovery.request_peers_timeout` setting. + notable: false + essSettingChange: false + - pr: 134374 + summary: Add file extension metadata to cache miss counter from `SharedBlobCacheService` + area: Search + type: enhancement + issues: [] + - pr: 134415 + summary: Differentiate between initial and reconnect RCS connections + area: Network + type: enhancement + issues: [] + - pr: 134446 + summary: Add support for `include_execution_metadata` parameter + area: ES|QL + type: enhancement + issues: [] + - pr: 134457 + summary: Improve scalability of get-license action + area: License + type: enhancement + issues: [] + - pr: 134458 + summary: Enable caching of all filters in `knn` queries + area: Vector Search + type: enhancement + issues: [] + - pr: 134475 + summary: ES|QL Absent and `AbsentOverTime` functions + area: ES|QL + type: enhancement + issues: + - 131069 + - pr: 134497 + summary: Limit when we push topn to lucene + area: ES|QL + type: bug + issues: [] + - pr: 134504 + summary: Adding headers support for OpenAI chat completion + area: Machine Learning + type: enhancement + issues: [] + - pr: 134524 + summary: Add support for flexible access pattern to `NormalizeForStreamProcessor` + area: Ingest Node + type: bug + issues: [] + - pr: 134585 + summary: Add classes to represent raw docs sampling configs + area: Ingest Node + type: feature + issues: [] + - pr: 134609 + summary: Add 'SslProfileExtension' SPI interface + area: TLS + type: enhancement + issues: [] + - pr: 134629 + summary: Improve block loader for source only runtime fields of type double + area: Mapping + type: enhancement + issues: [] + - pr: 134717 + summary: Allow including semantic field embeddings in `_source` + area: Vector Search + type: enhancement + issues: [] + - pr: 134798 + summary: Add relevant attributes to shard search latency APM metrics + area: Search + type: enhancement + issues: [] + - pr: 134822 + summary: Support querying multiple indices with the simplified RRF retriever + area: Relevance + type: enhancement + issues: [] + - pr: 134835 + summary: Add new `/_security/stats` endpoint + area: Authorization + type: enhancement + issues: [] + - pr: 134851 + summary: Remove ingest conditionals `_type` deprecation warning + area: Ingest Node + type: enhancement + issues: [] + - pr: 134879 + summary: Throw 4xx instead of 5xx for ESQL malformed query params + area: ES|QL + type: bug + issues: + - 134618 + - pr: 134893 + summary: Add trust configuration for cross cluster api keys + area: Security + type: enhancement + issues: [] + - pr: 134894 + summary: "[Cyera] Add `manage`, `create_index`, `read`, `index`, `write`, `delete`, permission for third party agent indices `kibana_system`" + area: Authorization + type: enhancement + issues: + - 134183 + - pr: 134933 + summary: Add ContextualAI Rerank Service Implementation to the Inference API + area: Machine Learning + type: enhancement + issues: [] + - pr: 134936 + summary: Fixing conditional processor mutability bugs + area: Ingest Node + type: bug + issues: [] + - pr: 134941 + summary: "DLM: Better `max_age` rollover for tiny retentions" + area: Data streams + type: enhancement + issues: + - 130960 + - pr: 134942 + summary: Add telemetry support for Lookup Join On Expression + area: ES|QL + type: enhancement + issues: [] + - pr: 134952 + summary: Add support for expressions with LOOKUP JOIN in tech preview + area: ES|QL + type: enhancement + issues: [] + highlight: + notable: true + title: Add support for expressions with LOOKUP JOIN in tech preview + body: |- + Enable Lookup Join on Expression Tech Preview + FROM index1 | LOOKUP JOIN lookup_index on left_field1 > right_field1 AND left_field2 <= right_field2 + pr: 134952 + - pr: 134960 + summary: Adding custom headers support openai text embeddings + area: Machine Learning + type: enhancement + issues: [] + - pr: 135011 + summary: Handle right hand side of Inline Stats coming optimized with `LocalRelation` shortcut + area: ES|QL + type: bug + issues: [] + - pr: 135026 + summary: Improve block loader for source only runtime fields of type keyword + area: Mapping + type: enhancement + issues: [] + - pr: 135036 + summary: Performance improvements for Lookup Join on Expression + area: ES|QL + type: enhancement + issues: [] + - pr: 135039 + summary: Use optimized field visitor for ignored source queries + area: Mapping + type: enhancement + issues: [] + - pr: 135053 + summary: Make `SecureString` comparisons constant time + area: Infra/Core + type: enhancement + issues: [] + - pr: 135087 + summary: "Optimize `BytesArray::indexOf,` which is used heavily in ndjson parsing" + area: Performance + type: feature + issues: [] + - pr: 135097 + summary: Fixes `countDistinctWithConditions` in csv-spec tests + area: ES|QL + type: bug + issues: + - 134380 + - pr: 135198 + summary: Enable `chunk_rescorer` in `text_similarity_reranker` + area: Relevance + type: enhancement + issues: [] + - pr: 135204 + summary: Make `_tsid` available in metadata + area: ES|QL + type: enhancement + issues: + - 133205 + - pr: 135244 + summary: Add reload listener to `SslProfile` + area: TLS + type: enhancement + issues: [] + - pr: 135247 + summary: Fix alias id when drop all aggregates + area: ES|QL + type: bug + issues: [] + - pr: 135262 + summary: Add usage stats for `semantic_text` fields + area: Vector Search + type: enhancement + issues: [] + - pr: 135263 + summary: Optimize `dotCount` in expanding dot parser + area: Mapping + type: enhancement + issues: [] + - pr: 135271 + summary: Add DLS stats to `_security/stats` + area: Authorization + type: enhancement + issues: [] + - pr: 135295 + summary: Replace any Attribute type when pushing down past Project + area: ES|QL + type: bug + issues: + - 134407 + - pr: 135299 + summary: Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields + area: Vector Search + type: feature + issues: [] + highlight: + notable: true + title: Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields + body: |- + This provides a new index type called DiskBBQ (`bbq_disk`). + DiskBBQ is a cluster based format that provides: + - faster and cheaper indexing than HNSW + - Better behavior in lower memory environments (degrades linearly, not exponentially) + - Is near HNSW for QPS when the index is in memory + + Current restrictions: + - only floating point values are allowed currently + - quantization is only to a single bit, so not recommended for low dimensionality vectors + - all other restrictions that exist for `dense_vector` fields still apply + + To utilize the format, its just like any other: + [source,yaml] + ---------------------------- + PUT vectors + { + "mappings": { + "properties": { + "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"} + } + } + } + ---------------------------- + Querying is just like any other field. + [source,yaml] + ---------------------------- + POST vectors/_search{ + "query": { + "knn": { + "field": "vector", + "query_vector": , + "k": 3 + } + } + } + ---------------------------- + `num_candidates` can be used for tuning approximate nature of the search. + Or, more granular control can be provided by setting `visit_percentage` directly. + pr: 135299 + - pr: 135306 + summary: Add support for extended search usage telemetry + area: Relevance + type: enhancement + issues: [] + - pr: 135309 + summary: Enable semantic search CCS when ccs_minimize_roundtrips=true + area: Vector Search + type: enhancement + issues: [] + - pr: 135337 + summary: Do not pass `ProjectMetadata` to lazy index permissions builder + area: Security + type: enhancement + issues: [] + - pr: 135342 + summary: Add 'profile' support for knn query on HNSW with early termination + area: Vector Search + type: enhancement + issues: [] + - pr: 135370 + summary: Add new `pattern_text` field mapper in tech preview + area: Mapping + type: feature + issues: [] + - pr: 135373 + summary: Improve block loader for source only runtime date fields + area: Mapping + type: enhancement + issues: [] + - pr: 135393 + summary: Improve block loader for source only runtime IP fields + area: Mapping + type: enhancement + issues: [] + - pr: 135401 + summary: Adds an OTLP metrics endpoint (`_otlp/v1/metrics`) as tech preview + area: TSDB + type: enhancement + issues: [] + - pr: 135402 + summary: "Improve TSDB ingestion by hashing dimensions only once, using a new auto-populeted `index.dimensions` private index setting" + area: TSDB + type: enhancement + issues: [] + - pr: 135403 + summary: Take INLINE STATS out of snapshot + area: ES|QL + type: enhancement + issues: [] + highlight: + notable: true + title: Enable INLINE STATS in non-snapshot builds + body: |- + This effectively releases INLINE STATS into tech preview. + - Enable the lexing/grammar for INLINE STATS in non-snapshot builds. + - Enable more tests with FORK and INLINE STATS + pr: 135403 + - pr: 135431 + summary: "[Downsampling++] Allow merging of passthrough mappers with object mappers under certain conditions" + area: Mapping + type: bug + issues: [] + - pr: 135446 + summary: Fix projection generation when pruning left join + area: ES|QL + type: bug + issues: [] + - pr: 135484 + summary: Add origin to client in SLM task + area: ILM+SLM + type: bug + issues: [] + - pr: 135524 + summary: Add time range bucketing attribute to APM shard search latency metrics + area: Search + type: enhancement + issues: [] + - pr: 135545 + summary: Add GPUPlugin for indexing vectors on GPU + area: Vector Search + type: feature + issues: [] + - pr: 135547 + summary: Fix union types lost attributes in `StubRelation` for inlinestats + area: ES|QL + type: bug + issues: [] + - pr: 135603 + summary: Make FUSE available in release builds + area: ES|QL + type: feature + issues: [] + - pr: 135604 + summary: ES|QL - add `dense_vector` field type + area: ES|QL + type: feature + issues: [] + - pr: 135635 + summary: Add executor name attribute to cache miss metrics + area: Search + type: enhancement + issues: [] + - pr: 135652 + summary: DFS search phase per shard duration APM metric + area: Search + type: enhancement + issues: [] + - pr: 135673 + summary: Add index setting that disables the `index.dimensions` based routing and `_tsid` creation strategy + area: TSDB + type: enhancement + issues: [] + - pr: 135709 + summary: ES|QL - KNN function + area: ES|QL + type: feature + issues: [] + - pr: 135767 + summary: Pass fix size instead of `maxPageSize` to `LuceneTopNOperator` scorer + area: ES|QL + type: bug + issues: [] + - pr: 135838 + summary: Remove rate limit field from services API for EIS + area: Machine Learning + type: bug + issues: [] + - pr: 135897 + summary: Apply source excludes early when retrieving the `_inference_fields` + area: Search + type: bug + issues: [] + - pr: 135966 + summary: Ensure queued `AbstractRunnables` are notified when executor stops + area: Machine Learning + type: bug + issues: + - 134651 + - pr: 135987 + summary: Avoid rewrite `round_to` with expensive queries + area: ES|QL + type: bug + issues: [] + - pr: 135994 + summary: Cardinality Aggregator Throws `UnsupportedOperationException` When Field Type is Vector + area: Vector Search + type: bug + issues: [] + - pr: 136017 + summary: Cohere service Model Id field is required + area: Machine Learning + type: bug + issues: [] + - pr: 136030 + summary: Update to lucene 10.3.1 + area: Search + type: feature + issues: [] + - pr: 136058 + summary: Configurable HTTP read and connect timeouts for url based SAML metadata resolution + area: Security + type: bug + issues: [] + - pr: 136105 + summary: Fix a breaker bug + area: ES|QL + type: bug + issues: + - 135224 + - 135260 + - pr: 136119 + summary: Fix logsdb settings provider mapping filters + area: Logs + type: bug + issues: + - 136107 + - pr: 136214 + summary: Using index setting providers for data stream setting validation + area: Data streams + type: bug + issues: + - 136166 + - pr: 136279 + summary: Initialize `TermsEnum` eagerly + area: Search + type: bug + issues: [] + - pr: 136312 + summary: Fix _inference_fields handling on old indices + area: Vector Search + type: bug + issues: + - 136130 + - pr: 136315 + summary: Store full path in `_ignored` when ignoring dynamic array field + area: Mapping + type: bug + issues: [] + - pr: 136348 + summary: Add error message when using inline stats on TS + area: ES|QL + type: bug + issues: + - 136092 + - pr: 136444 + summary: Create new block when filter `OrdinalBytesRefBlock` + area: ES|QL + type: bug + issues: + - 136423 + - pr: 136492 + summary: Make `ResolveUnionTypes` rule stateless + area: ES|QL + type: bug + issues: [] + - pr: 136501 + summary: "Reindex-from-remote: Validate basic auth params" + area: Indices APIs + type: bug + issues: + - 135925 + - pr: 136556 + summary: Bugfix 136545 + area: Vector Search + type: bug + issues: [] + - pr: 136563 + summary: Bugfix/disable matches highlight knn + area: Vector Search + type: bug + issues: [] + - pr: 136577 + summary: Clean up inference indices on failed endpoint creation + area: Machine Learning + type: bug + issues: + - 123726 + - pr: 136625 + summary: Optimize Index Permission Automatons for Has Privileges + area: Security + type: bug + issues: [] + - pr: 136649 + summary: Fix append processor `ignore_empty_values` edge case + area: Ingest Node + type: bug + issues: [] + - pr: 136664 + summary: Drop project-id from threadcontext for CCS + area: Authorization + type: bug + issues: [] + - pr: 136684 + summary: Lazy compute and cache `grantsAll` per privilege + area: Authorization + type: enhancement + issues: [] + - pr: 136709 + summary: Fix log formatting in `SnapshotLifecycleTask` + area: ILM+SLM + type: bug + issues: [] + - pr: 136751 + summary: Adjust jinaai rerank response parser to handle document field as string or object + area: Machine Learning + type: bug + issues: [] + - pr: 136759 + summary: Avoid counting snapshot failures twice in SLM + area: ILM+SLM + type: bug + issues: [] + - pr: 136769 + summary: Release cluster state + area: Machine Learning + type: bug + issues: + - 123243 diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md index cc6e198e2d24d..66bb6adcb1c84 100644 --- a/docs/release-notes/deprecations.md +++ b/docs/release-notes/deprecations.md @@ -16,14 +16,24 @@ To give you insight into what deprecated features you’re using, {{es}}: % ## Next version [elasticsearch-nextversion-deprecations] -## 9.1.5 [elasticsearch-9.1.5-deprecations] +```{applies_to} +stack: ga 9.2.0 +``` +## 9.2.0 [elasticsearch-9.2.0-deprecations] + +Cluster Coordination: +* Remove `PeerFinder` request timeout [#134365](https://github.com/elastic/elasticsearch/pull/134365) + -There are no deprecations associated with this release. ## 9.0.8 [elasticsearch-9.0.8-deprecations] There are no deprecations associated with this release. +## 9.1.5 [elasticsearch-9.1.5-deprecations] + +There are no deprecations associated with this release. + ## 9.1.4 [elasticsearch-9.1.4-deprecations] There are no deprecations associated with this release. diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md index 9eb04a7f059a0..4508b06fbb76d 100644 --- a/docs/release-notes/index.md +++ b/docs/release-notes/index.md @@ -20,26 +20,489 @@ To check for security updates, go to [Security announcements for the Elastic sta % ### Fixes [elasticsearch-next-fixes] % * -## 9.1.5 [elasticsearch-9.1.5-release-notes] +## 9.2.0 [elasticsearch-9.2.0-release-notes] +```{applies_to} +stack: ga 9.2.0 +``` -### Highlights [elasticsearch-9.1.5-highlights] +### Highlights [elasticsearch-9.2.0-highlights] -::::{dropdown} Prevent LIMIT + MV_EXPAND before remote ENRICH -Queries using LIMIT followed by MV_EXPAND before a remote ENRICH can produce incorrect results due to distributed execution semantics. -These queries are now unsupported and produce an error. Example: +::::{dropdown} Enable Failure Store for new logs data streams +The [Failure Store](docs-content://manage-data/data-store/data-streams/failure-store.md) is now enabled by default for new logs data streams matching the pattern `logs-*-*`. This means that such data streams will now store invalid documents in a +dedicated failure index instead of rejecting them, allowing better visibility and control over data quality issues without loosing data. This can be [enabled manually](docs-content://manage-data/data-store/data-streams/failure-store.md#set-up-failure-store-existing) for existing data streams. +Note: With the failure store enabled, the http response code clients receive when indexing invalid documents will change from `400 Bad Request` to `201 Created`, with an additional response attribute `"failure_store" : "used"`. +:::: -```yaml -FROM *:events | SORT @timestamp | LIMIT 2 | MV_EXPAND ip | ENRICH _remote:clientip_policy ON ip -``` +::::{dropdown} Add support for Lookup Join on Multiple Fields +Add support for Lookup Join on Multiple Fields e.g. FROM index1 +| LOOKUP JOIN lookup_index on field1, field2 +:::: -To avoid this error, reorder your query, for example by moving ENRICH earlier in the pipeline. +::::{dropdown} Add support for expressions with LOOKUP JOIN in tech preview +Enable Lookup Join on Expression Tech Preview +FROM index1 | LOOKUP JOIN lookup_index on left_field1 > right_field1 AND left_field2 <= right_field2 :::: +::::{dropdown} Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields +This provides a new index type called DiskBBQ (`bbq_disk`). +DiskBBQ is a cluster based format that provides: + - faster and cheaper indexing than HNSW + - Better behavior in lower memory environments (degrades linearly, not exponentially) + - Is near HNSW for QPS when the index is in memory + +Current restrictions: + - only floating point values are allowed currently + - quantization is only to a single bit, so not recommended for low dimensionality vectors + - all other restrictions that exist for `dense_vector` fields still apply + +To utilize the format, its just like any other: +[source,yaml] +---------------------------- +PUT vectors + { + "mappings": { + "properties": { + "vector": {"type": "dense_vector", "index_options": {"type": "disk_bbq"} + } + } + } +---------------------------- + Querying is just like any other field. +[source,yaml] +---------------------------- +POST vectors/_search{ + "query": { + "knn": { + "field": "vector", + "query_vector": , + "k": 3 + } + } +} +---------------------------- +`num_candidates` can be used for tuning approximate nature of the search. +Or, more granular control can be provided by setting `visit_percentage` directly. +:::: + +::::{dropdown} Enable INLINE STATS in non-snapshot builds +This effectively releases INLINE STATS into tech preview. +- Enable the lexing/grammar for INLINE STATS in non-snapshot builds. +- Enable more tests with FORK and INLINE STATS +:::: + +::::{dropdown} Allow direct IO for BBQ rescoring +BBQ rescoring performance can be drastically affected by the amount of available +off-heap RAM for use by the system page cache. When there is not enough off-heap RAM +to fit all the vector data in memory, BBQ search latencies can be affected by as much as 5000x. +Specifying the `vector.rescoring.directio=true` Java option on all vector search +nodes modifies rescoring to use direct IO, which eliminates these very high latencies +from searches in low-memory scenarios, at a cost of a reduction +in vector search performance for BBQ indices when the vectors do all fit in memory. + +This option is released in 9.1 as a tech preview whilst we analyse its effect +for a variety of use cases. +:::: + +::::{dropdown} Add remote index support to LOOKUP JOIN +Queries containing LOOKUP JOIN now can be preformed on cross-cluster indices, for example: +[source,yaml] +---------------------------- +FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100 +---------------------------- +:::: + +::::{dropdown} New lucene 10.3.0 release +- Improved performance for lexical, vector and primary-key searches +- Use optimistic-with-checking KNN Query execution strategy in place of cross-thread global queue min-score checking. Improves performance and consistency. +- Bulk scoring added for floating point vectors in HNSW. Improving query latency and indexing throughput +- Multiple improvements to HNSW graph traversal and storage +:::: + +### Features and enhancements [elasticsearch-9.2.0-features-enhancements] + +Allocation: +* Add second max queue latency stat to `ClusterInfo` [#132675](https://github.com/elastic/elasticsearch/pull/132675) + +Authentication: +* Add attribute count to `SamlAttribute` `toString` [#131173](https://github.com/elastic/elasticsearch/pull/131173) +* Allow configuring SAML private attributes [#133154](https://github.com/elastic/elasticsearch/pull/133154) +* Correct slow log user for RCS 2.0 [#130140](https://github.com/elastic/elasticsearch/pull/130140) + +Authorization: +* Add DLS stats to `_security/stats` [#135271](https://github.com/elastic/elasticsearch/pull/135271) +* Add hits and misses timing stats to DLS cache [#133314](https://github.com/elastic/elasticsearch/pull/133314) +* Add new `/_security/stats` endpoint [#134835](https://github.com/elastic/elasticsearch/pull/134835) +* Expose existing DLS cache x-pack usage statistics [#132845](https://github.com/elastic/elasticsearch/pull/132845) +* Lazy compute and cache `grantsAll` per privilege [#136684](https://github.com/elastic/elasticsearch/pull/136684) +* [Cyera] Add `manage`, `create_index`, `read`, `index`, `write`, `delete`, permission for third party agent indices `kibana_system` [#134894](https://github.com/elastic/elasticsearch/pull/134894) (issue: [#134183](https://github.com/elastic/elasticsearch/issues/134183)) +* [Security] Add entity store and asset criticality index privileges to built in Editor, Viewer and Kibana System roles [#129662](https://github.com/elastic/elasticsearch/pull/129662) + +Codec: +* Push down compute engine value loading of long based singleton numeric doc value to the es819 tsdb doc values codec. [#132622](https://github.com/elastic/elasticsearch/pull/132622) +* Push down loading of singleton dense double based field types to the … [#133397](https://github.com/elastic/elasticsearch/pull/133397) +* Skip iterating DISI when reading metric values [#133365](https://github.com/elastic/elasticsearch/pull/133365) + +Data streams: +* DLM: Better `max_age` rollover for tiny retentions [#134941](https://github.com/elastic/elasticsearch/pull/134941) (issue: [#130960](https://github.com/elastic/elasticsearch/issues/130960)) +* ES-11331 streams params restriction [#132967](https://github.com/elastic/elasticsearch/pull/132967) +* Enable Failure Store for new logs-*-* data streams [#131261](https://github.com/elastic/elasticsearch/pull/131261) (issue: [#131105](https://github.com/elastic/elasticsearch/issues/131105)) +* Enable failure store for newly created OTel data streams [#131395](https://github.com/elastic/elasticsearch/pull/131395) +* Only Allow Enabling Streams If No Conflicting Indices Exist [#132064](https://github.com/elastic/elasticsearch/pull/132064) +* Restrict Indexing To Child Streams When Streams Is Enabled [#132011](https://github.com/elastic/elasticsearch/pull/132011) + +Distributed: +* Run `TransportClusterStateAction` on local node [#129872](https://github.com/elastic/elasticsearch/pull/129872) + +Downsampling: +* [Downsampling++] Add time series telemetry in xpack usage [#134214](https://github.com/elastic/elasticsearch/pull/134214) (issue: [#133953](https://github.com/elastic/elasticsearch/issues/133953)) + +ES|QL: +* Accept unsigned longs on MAX and MIN aggregations [#131694](https://github.com/elastic/elasticsearch/pull/131694) +* Add Dependency Checker for `LogicalLocalPlanOptimizer` [#130409](https://github.com/elastic/elasticsearch/pull/130409) +* Add SET instruction [#134029](https://github.com/elastic/elasticsearch/pull/134029) +* Add checks that optimizers do not modify the layout [#130855](https://github.com/elastic/elasticsearch/pull/130855) (issue: [#125576](https://github.com/elastic/elasticsearch/issues/125576)) +* Add fast path for single value in VALUES aggregator [#130510](https://github.com/elastic/elasticsearch/pull/130510) +* Add optimized path for intermediate values aggregator [#131390](https://github.com/elastic/elasticsearch/pull/131390) +* Add query heads priority to `SliceQueue` [#133245](https://github.com/elastic/elasticsearch/pull/133245) +* Add remote index support to LOOKUP JOIN [#129013](https://github.com/elastic/elasticsearch/pull/129013) +* Add support for LOOKUP JOIN on multiple fields [#131559](https://github.com/elastic/elasticsearch/pull/131559) +* Add support for RLIKE (LIST) with pushdown [#129929](https://github.com/elastic/elasticsearch/pull/129929) +* Add support for `include_execution_metadata` parameter [#134446](https://github.com/elastic/elasticsearch/pull/134446) +* Add support for expressions with LOOKUP JOIN in tech preview [#134952](https://github.com/elastic/elasticsearch/pull/134952) +* Add telemetry support for Lookup Join On Expression [#134942](https://github.com/elastic/elasticsearch/pull/134942) +* Adding Contains ESQL String function [#133016](https://github.com/elastic/elasticsearch/pull/133016) +* Adds the `v_hamming` function for calculating the Hamming distance between two dense vectors [#132959](https://github.com/elastic/elasticsearch/pull/132959) (issue: [#132056](https://github.com/elastic/elasticsearch/issues/132056)) +* Adopt a "LogicalPlan" approach to running multiple sub-queries (with INLINESTATS so far) [#128917](https://github.com/elastic/elasticsearch/pull/128917) +* Allow pruning columns added by `InlineJoin` [#131204](https://github.com/elastic/elasticsearch/pull/131204) +* Allow remote enrich after LOOKUP JOIN [#131940](https://github.com/elastic/elasticsearch/pull/131940) +* Consider min/max from predicates when transform date_trunc/bucket to `round_to` [#131341](https://github.com/elastic/elasticsearch/pull/131341) +* Consider min/max from predicates when transform date_trunc/bucket to `round_to` option 2 [#132143](https://github.com/elastic/elasticsearch/pull/132143) +* ES|QL - Allow multivalued query parameters [#134317](https://github.com/elastic/elasticsearch/pull/134317) +* ES|QL - KNN function [#135709](https://github.com/elastic/elasticsearch/pull/135709) +* ES|QL - add `dense_vector` field type [#135604](https://github.com/elastic/elasticsearch/pull/135604) +* ES|QL Absent and `AbsentOverTime` functions [#134475](https://github.com/elastic/elasticsearch/pull/134475) (issue: [#131069](https://github.com/elastic/elasticsearch/issues/131069)) +* Enable `date` `date_nanos` implicit casting [#133369](https://github.com/elastic/elasticsearch/pull/133369) +* Esql `mv_contains` function [#133636](https://github.com/elastic/elasticsearch/pull/133636) +* Esql skip null metrics [#133087](https://github.com/elastic/elasticsearch/pull/133087) (issue: [#129524](https://github.com/elastic/elasticsearch/issues/129524)) +* Fail `profile` on text response formats [#128627](https://github.com/elastic/elasticsearch/pull/128627) +* Implement `v_magnitude` function [#132765](https://github.com/elastic/elasticsearch/pull/132765) (issue: [#132768](https://github.com/elastic/elasticsearch/issues/132768)) +* Improve Expanding Lookup Join performance by pushing a filter to the right side of the lookup join [#133166](https://github.com/elastic/elasticsearch/pull/133166) +* Improve cpu utilization with dynamic slice size in doc partitioning [#132774](https://github.com/elastic/elasticsearch/pull/132774) +* Integrate LIKE/RLIKE LIST with `ReplaceStringCasingWithInsensitiveRegexMatch` rule [#131531](https://github.com/elastic/elasticsearch/pull/131531) +* LOOKUP JOIN with expressions [#134098](https://github.com/elastic/elasticsearch/pull/134098) +* Make FUSE available in release builds [#135603](https://github.com/elastic/elasticsearch/pull/135603) +* Make INLINESTATS (and subplans) work with CCS [#134323](https://github.com/elastic/elasticsearch/pull/134323) (issue: [#124748](https://github.com/elastic/elasticsearch/issues/124748)) +* Make `_tsid` available in metadata [#135204](https://github.com/elastic/elasticsearch/pull/135204) (issue: [#133205](https://github.com/elastic/elasticsearch/issues/133205)) +* Performance improvements for Lookup Join on Expression [#135036](https://github.com/elastic/elasticsearch/pull/135036) +* Remove unnecessary calls to Fold [#130944](https://github.com/elastic/elasticsearch/pull/130944) (issue: [#119756](https://github.com/elastic/elasticsearch/issues/119756)) +* Replace "representable" type error messages [#131775](https://github.com/elastic/elasticsearch/pull/131775) +* Replace `RoundTo` linear search evaluator with manual evaluators [#131733](https://github.com/elastic/elasticsearch/pull/131733) +* Rewrite `RoundTo` to `QueryAndTags` [#132512](https://github.com/elastic/elasticsearch/pull/132512) +* Run single phase aggregation when possible [#131485](https://github.com/elastic/elasticsearch/pull/131485) +* Some optimizations for constant blocks [#132456](https://github.com/elastic/elasticsearch/pull/132456) +* Speed up loading keyword fields with index sorts [#132950](https://github.com/elastic/elasticsearch/pull/132950) +* Speed up reading multivalued keywords [#131061](https://github.com/elastic/elasticsearch/pull/131061) +* Substitue `date_trunc` with `round_to` when the pre-calculated rounding points are available [#128639](https://github.com/elastic/elasticsearch/pull/128639) +* Support filters on inlinestats [#132934](https://github.com/elastic/elasticsearch/pull/132934) +* Support geohash, geotile and geohex grid types [#129581](https://github.com/elastic/elasticsearch/pull/129581) +* Support geohash, geotile and geohex grid types in ST_INTERSECTS and ST_DISJOINT [#133546](https://github.com/elastic/elasticsearch/pull/133546) +* Take INLINE STATS out of snapshot [#135403](https://github.com/elastic/elasticsearch/pull/135403) + +ILM+SLM: +* Add `age_in_millis` to ILM Explain Response [#128866](https://github.com/elastic/elasticsearch/pull/128866) (issue: [#103659](https://github.com/elastic/elasticsearch/issues/103659)) +* Enhancement: ILM sets `indexing_complete` to true from `ReadOnly` action [#129945](https://github.com/elastic/elasticsearch/pull/129945) +* ILM: Force merge on zero-replica cloned index before snapshotting for searchable snapshots [#133954](https://github.com/elastic/elasticsearch/pull/133954) (issue: [#75478](https://github.com/elastic/elasticsearch/issues/75478)) + +Indices APIs: +* Add index mode to resolve index response [#132858](https://github.com/elastic/elasticsearch/pull/132858) +* Add mode filter to _resolve/index [#133616](https://github.com/elastic/elasticsearch/pull/133616) + +Inference: +* Added support to configure query timeout for inference [#131551](https://github.com/elastic/elasticsearch/pull/131551) + +Infra/Core: +* Add .integration_knowledge system index for usage by AI assistants [#132506](https://github.com/elastic/elasticsearch/pull/132506) +* Extend kibana-system permissions to manage security entities [#133968](https://github.com/elastic/elasticsearch/pull/133968) +* Make `SecureString` comparisons constant time [#135053](https://github.com/elastic/elasticsearch/pull/135053) +* Support Fields API in conditional ingest processors [#131581](https://github.com/elastic/elasticsearch/pull/131581) +* Use java8 variant of apm-agent [#132651](https://github.com/elastic/elasticsearch/pull/132651) + +Infra/Metrics: +* Upgrade apm-agent to 1.55.0 [#131510](https://github.com/elastic/elasticsearch/pull/131510) + +Ingest Node: +* Add `copy_from` option to the Append processor [#132003](https://github.com/elastic/elasticsearch/pull/132003) +* Add classes to represent raw docs sampling configs [#134585](https://github.com/elastic/elasticsearch/pull/134585) +* Add option for Append Processor to skip/allow empty values [#105718](https://github.com/elastic/elasticsearch/pull/105718) (issue: [#104813](https://github.com/elastic/elasticsearch/issues/104813)) +* Add recover_failure_document processor to remediate failurestore docs [#133360](https://github.com/elastic/elasticsearch/pull/133360) +* Adding a `merge_type` parameter to the ingest simulate API [#132210](https://github.com/elastic/elasticsearch/pull/132210) (issue: [#131608](https://github.com/elastic/elasticsearch/issues/131608)) +* Adding simulate ingest effective mapping [#132833](https://github.com/elastic/elasticsearch/pull/132833) +* Component Templates: Add created and modified date [#131536](https://github.com/elastic/elasticsearch/pull/131536) +* Enable failure store for newly created APM datastreams [#131296](https://github.com/elastic/elasticsearch/pull/131296) +* Handle structured log messages [#131027](https://github.com/elastic/elasticsearch/pull/131027) (issue: [#130333](https://github.com/elastic/elasticsearch/issues/130333)) +* Index template: Add created_date and modified_date [#132083](https://github.com/elastic/elasticsearch/pull/132083) +* Pipelines: Add `created_date` and `modified_date` [#130847](https://github.com/elastic/elasticsearch/pull/130847) +* Remove ingest conditionals `_type` deprecation warning [#134851](https://github.com/elastic/elasticsearch/pull/134851) + +License: +* Improve scalability of get-license action [#134457](https://github.com/elastic/elasticsearch/pull/134457) + +Machine Learning: +* Add ContextualAI Rerank Service Implementation to the Inference API [#134933](https://github.com/elastic/elasticsearch/pull/134933) +* Add `RerankRequestChunker` [#130485](https://github.com/elastic/elasticsearch/pull/130485) +* Add support for dimensions in google vertex ai request [#132689](https://github.com/elastic/elasticsearch/pull/132689) +* Added AI21 Completion and Chat Completion support to the Inference Plugin [#113757](https://github.com/elastic/elasticsearch/pull/113757) +* Added Google Model Garden Anthropic Completion and Chat Completion support to the Inference Plugin [#134080](https://github.com/elastic/elasticsearch/pull/134080) +* Added Llama provider support to the Inference Plugin [#130092](https://github.com/elastic/elasticsearch/pull/130092) +* Adding custom headers support openai text embeddings [#134960](https://github.com/elastic/elasticsearch/pull/134960) +* Adding headers support for OpenAI chat completion [#134504](https://github.com/elastic/elasticsearch/pull/134504) +* Block trained model updates from inference [#130940](https://github.com/elastic/elasticsearch/pull/130940) (issue: [#129999](https://github.com/elastic/elasticsearch/issues/129999)) +* Cache Inference Endpoints [#133860](https://github.com/elastic/elasticsearch/pull/133860) (issue: [#133135](https://github.com/elastic/elasticsearch/issues/133135)) +* Enable force inference endpoint deleting for invalid models and after stopping model deployment fails [#129090](https://github.com/elastic/elasticsearch/pull/129090) +* Remove upper limit for chunking settings [#133718](https://github.com/elastic/elasticsearch/pull/133718) +* Support Gemini thinking budget in inference API [#133599](https://github.com/elastic/elasticsearch/pull/133599) +* Supporting more timestamp formats in `_text_structure/find_structure` [#133745](https://github.com/elastic/elasticsearch/pull/133745) +* Track inference deployments [#131442](https://github.com/elastic/elasticsearch/pull/131442) +* [ML] Add Azure AI Rerank support to the Inference Plugin [#129848](https://github.com/elastic/elasticsearch/pull/129848) +* [ML] Add IBM watsonx Completion and Chat Completion support to the Inference Plugin [#129146](https://github.com/elastic/elasticsearch/pull/129146) + +Mapping: +* Add new `pattern_text` field mapper in tech preview [#135370](https://github.com/elastic/elasticsearch/pull/135370) +* Adds transport-only flag to always include indices in the field caps transport response [#133074](https://github.com/elastic/elasticsearch/pull/133074) +* Improve block loader for source only runtime IP fields [#135393](https://github.com/elastic/elasticsearch/pull/135393) +* Improve block loader for source only runtime date fields [#135373](https://github.com/elastic/elasticsearch/pull/135373) +* Improve block loader for source only runtime fields of type double [#134629](https://github.com/elastic/elasticsearch/pull/134629) +* Improve block loader for source only runtime fields of type keyword [#135026](https://github.com/elastic/elasticsearch/pull/135026) +* Improve block loader for source only runtime fields of type long [#134117](https://github.com/elastic/elasticsearch/pull/134117) +* Optimize `dotCount` in expanding dot parser [#135263](https://github.com/elastic/elasticsearch/pull/135263) +* Runtime fields: pass down runtime field name as source filter when source mode is synthetic [#133897](https://github.com/elastic/elasticsearch/pull/133897) +* Use optimized field visitor for ignored source queries [#135039](https://github.com/elastic/elasticsearch/pull/135039) + +Network: +* Add audit logging for stream content [#130594](https://github.com/elastic/elasticsearch/pull/130594) +* Allow adjustment of transport TLS handshake timeout [#130909](https://github.com/elastic/elasticsearch/pull/130909) +* Differentiate between initial and reconnect RCS connections [#134415](https://github.com/elastic/elasticsearch/pull/134415) +* Expose HTTP connection metrics to telemetry [#130939](https://github.com/elastic/elasticsearch/pull/130939) +* Return 429 instead of 500 for timeout handlers [#133111](https://github.com/elastic/elasticsearch/pull/133111) + +Performance: +* Optimize `BytesArray::indexOf,` which is used heavily in ndjson parsing [#135087](https://github.com/elastic/elasticsearch/pull/135087) + +Relevance: +* Add support for extended search usage telemetry [#135306](https://github.com/elastic/elasticsearch/pull/135306) +* Add support for weighted RRF in retrievers [#130658](https://github.com/elastic/elasticsearch/pull/130658) +* Enable `chunk_rescorer` in `text_similarity_reranker` [#135198](https://github.com/elastic/elasticsearch/pull/135198) +* Support querying multiple indices with the simplified RRF retriever [#134822](https://github.com/elastic/elasticsearch/pull/134822) +* Support querying multiple indices with the simplified linear retriever [#133720](https://github.com/elastic/elasticsearch/pull/133720) +* Support semantic reranking using contextual snippets instead of entire field text [#129369](https://github.com/elastic/elasticsearch/pull/129369) +* Text similarity reranker chunks and scores snippets [#133576](https://github.com/elastic/elasticsearch/pull/133576) + +Search: +* Add executor name attribute to cache miss metrics [#135635](https://github.com/elastic/elasticsearch/pull/135635) +* Add file extension metadata to cache miss counter from `SharedBlobCacheService` [#134374](https://github.com/elastic/elasticsearch/pull/134374) +* Add relevant attributes to search took time APM metrics [#134232](https://github.com/elastic/elasticsearch/pull/134232) +* Add relevant attributes to shard search latency APM metrics [#134798](https://github.com/elastic/elasticsearch/pull/134798) +* Add support for per-field weights in simplified RRF retriever syntax [#132680](https://github.com/elastic/elasticsearch/pull/132680) +* Add time range bucketing attribute to APM shard search latency metrics [#135524](https://github.com/elastic/elasticsearch/pull/135524) +* Add top level normalizer for linear retriever [#129693](https://github.com/elastic/elasticsearch/pull/129693) +* Adds sparse vector index options settings to semantic_text field [#131058](https://github.com/elastic/elasticsearch/pull/131058) +* DFS search phase per shard duration APM metric [#135652](https://github.com/elastic/elasticsearch/pull/135652) +* Introduce new rescorer based on script [#74274](https://github.com/elastic/elasticsearch/pull/74274) (issue: [#52338](https://github.com/elastic/elasticsearch/issues/52338)) +* Refresh potential lost connections at query start for `_search` [#130463](https://github.com/elastic/elasticsearch/pull/130463) +* Refresh potential lost connections at query start for field caps [#131517](https://github.com/elastic/elasticsearch/pull/131517) +* Support nested fields for term vectors API when using artificial documents [#92568](https://github.com/elastic/elasticsearch/pull/92568) (issue: [#91902](https://github.com/elastic/elasticsearch/issues/91902)) +* Update to lucene 10.3.1 [#136030](https://github.com/elastic/elasticsearch/pull/136030) +* Upgrade elasticsearch to lucene 10.3.0 [#133980](https://github.com/elastic/elasticsearch/pull/133980) + +Searchable Snapshots: +* Add cache miss and read metrics [#132497](https://github.com/elastic/elasticsearch/pull/132497) +* Add epoch blob-cache metric [#132547](https://github.com/elastic/elasticsearch/pull/132547) + +Security: +* Add `LoadedSecureSettings` for keeping temporary secure settings loaded [#134349](https://github.com/elastic/elasticsearch/pull/134349) +* Add read permissions for osquery manager result indices [#130824](https://github.com/elastic/elasticsearch/pull/130824) +* Add signing configuration for cross cluster api keys [#134137](https://github.com/elastic/elasticsearch/pull/134137) +* Add trust configuration for cross cluster api keys [#134893](https://github.com/elastic/elasticsearch/pull/134893) +* Do not pass `ProjectMetadata` to lazy index permissions builder [#135337](https://github.com/elastic/elasticsearch/pull/135337) + +Snapshot/Restore: +* Add extension points to remediate index metadata in during snapshot restore [#131706](https://github.com/elastic/elasticsearch/pull/131706) +* Expose S3 connection max idle time as a setting [#125552](https://github.com/elastic/elasticsearch/pull/125552) +* Implement `failIfAlreadyExists` in S3 repositories [#133030](https://github.com/elastic/elasticsearch/pull/133030) (issue: [#128565](https://github.com/elastic/elasticsearch/issues/128565)) +* Improve lost-increment message in repo analysis [#131200](https://github.com/elastic/elasticsearch/pull/131200) + +Store: +* Add new `CachePopulationReason` [#130593](https://github.com/elastic/elasticsearch/pull/130593) +* Improve `ShardLockObtainFailedException` message [#134198](https://github.com/elastic/elasticsearch/pull/134198) + +TLS: +* Add 'SslProfileExtension' SPI interface [#134609](https://github.com/elastic/elasticsearch/pull/134609) +* Add reload listener to `SslProfile` [#135244](https://github.com/elastic/elasticsearch/pull/135244) + +TSDB: +* Add index setting that disables the `index.dimensions` based routing and `_tsid` creation strategy [#135673](https://github.com/elastic/elasticsearch/pull/135673) +* Add ordinal range encode for tsid [#133018](https://github.com/elastic/elasticsearch/pull/133018) +* Adds an OTLP metrics endpoint (`_otlp/v1/metrics`) as tech preview [#135401](https://github.com/elastic/elasticsearch/pull/135401) +* Improve TSDB ingestion by hashing dimensions only once, using a new auto-populeted `index.dimensions` private index setting [#135402](https://github.com/elastic/elasticsearch/pull/135402) + +Vector Search: +* Add 'profile' support for knn query on HNSW with early termination [#135342](https://github.com/elastic/elasticsearch/pull/135342) +* Add GPUPlugin for indexing vectors on GPU [#135545](https://github.com/elastic/elasticsearch/pull/135545) +* Add low-level optimized Neon, AVX2, and AVX 512 float32 vector operations [#130635](https://github.com/elastic/elasticsearch/pull/130635) +* Add support for retrieving semantic_text's indexed chunks via fields API [#132410](https://github.com/elastic/elasticsearch/pull/132410) +* Add usage stats for `semantic_text` fields [#135262](https://github.com/elastic/elasticsearch/pull/135262) +* Allow direct IO for BBQ rescoring [#125921](https://github.com/elastic/elasticsearch/pull/125921) +* Allow including semantic field embeddings in `_source` [#134717](https://github.com/elastic/elasticsearch/pull/134717) +* Enable caching of all filters in `knn` queries [#134458](https://github.com/elastic/elasticsearch/pull/134458) +* Enable semantic search CCS when ccs_minimize_roundtrips=true [#135309](https://github.com/elastic/elasticsearch/pull/135309) +* Ensure vectors are always included in reindex actions [#130834](https://github.com/elastic/elasticsearch/pull/130834) +* Release DiskBBQ(`bbq_disk`) index type for `dense_vector` fields [#135299](https://github.com/elastic/elasticsearch/pull/135299) +* Remove vectors from `_source` transparently [#130382](https://github.com/elastic/elasticsearch/pull/130382) +* Speed up (filtered) KNN queries for flat vector fields [#130251](https://github.com/elastic/elasticsearch/pull/130251) +* Speed up `OptimizedScalarQuantizer` [#131599](https://github.com/elastic/elasticsearch/pull/131599) +* Support kNN filter on nested metadata [#113949](https://github.com/elastic/elasticsearch/pull/113949) (issues: [#128803](https://github.com/elastic/elasticsearch/issues/128803), [#106994](https://github.com/elastic/elasticsearch/issues/106994)) +* Support using the semantic query across multiple inference IDs [#133675](https://github.com/elastic/elasticsearch/pull/133675) +* Wrap ES KNN queries with PatienceKNN query [#127223](https://github.com/elastic/elasticsearch/pull/127223) + + +### Fixes [elasticsearch-9.2.0-fixes] + +Allocation: +* Make forecast write load accurate when shard numbers change [#129990](https://github.com/elastic/elasticsearch/pull/129990) + +Analysis: +* Adding check for `isIndexed` in text fields when generating field exists queries to avoid ISE when field is stored but not indexed or with `doc_values` [#130531](https://github.com/elastic/elasticsearch/pull/130531) +* Avoid internal server error when suggester requires unigrams but no unigrams are provided, return bad request instead [#132321](https://github.com/elastic/elasticsearch/pull/132321) (issue: [#131928](https://github.com/elastic/elasticsearch/issues/131928)) + +Authorization: +* Drop project-id from threadcontext for CCS [#136664](https://github.com/elastic/elasticsearch/pull/136664) + +Cluster Coordination: +* Avoid stack overflow in `IndicesClusterStateService` `applyClusterState` [#132536](https://github.com/elastic/elasticsearch/pull/132536) + +Codec: +* Fix disk usage estimation for SORTED_SET doc values [#133722](https://github.com/elastic/elasticsearch/pull/133722) + +Data streams: +* Add existing shards allocator settings to failure store allowed list [#131056](https://github.com/elastic/elasticsearch/pull/131056) +* Fix service destination template file name [#133403](https://github.com/elastic/elasticsearch/pull/133403) +* Using index setting providers for data stream setting validation [#136214](https://github.com/elastic/elasticsearch/pull/136214) (issue: [#136166](https://github.com/elastic/elasticsearch/issues/136166)) + +Distributed: +* Fix race condition in `RemoteClusterService.collectNodes()` [#131937](https://github.com/elastic/elasticsearch/pull/131937) + +ES|QL: +* Add error message when using inline stats on TS [#136348](https://github.com/elastic/elasticsearch/pull/136348) (issue: [#136092](https://github.com/elastic/elasticsearch/issues/136092)) +* Avoid rewrite `round_to` with expensive queries [#135987](https://github.com/elastic/elasticsearch/pull/135987) +* Create new block when filter `OrdinalBytesRefBlock` [#136444](https://github.com/elastic/elasticsearch/pull/136444) (issue: [#136423](https://github.com/elastic/elasticsearch/issues/136423)) +* Fix FORK with union-types [#134033](https://github.com/elastic/elasticsearch/pull/134033) (issue: [#133973](https://github.com/elastic/elasticsearch/issues/133973)) +* Fix `AsyncOperator` status values and add emitted rows [#132738](https://github.com/elastic/elasticsearch/pull/132738) +* Fix a breaker bug [#136105](https://github.com/elastic/elasticsearch/pull/136105) (issues: [#135224](https://github.com/elastic/elasticsearch/issues/135224), [#135260](https://github.com/elastic/elasticsearch/issues/135260)) +* Fix alias id when drop all aggregates [#135247](https://github.com/elastic/elasticsearch/pull/135247) +* Fix async operator warnings not always sent when blocking [#132744](https://github.com/elastic/elasticsearch/pull/132744) (issues: [#130642](https://github.com/elastic/elasticsearch/issues/130642), [#132554](https://github.com/elastic/elasticsearch/issues/132554), [#132778](https://github.com/elastic/elasticsearch/issues/132778), [#130296](https://github.com/elastic/elasticsearch/issues/130296), [#132555](https://github.com/elastic/elasticsearch/issues/132555), [#131563](https://github.com/elastic/elasticsearch/issues/131563), [#131148](https://github.com/elastic/elasticsearch/issues/131148), [#132604](https://github.com/elastic/elasticsearch/issues/132604), [#128030](https://github.com/elastic/elasticsearch/issues/128030)) +* Fix bug in topn [#133601](https://github.com/elastic/elasticsearch/pull/133601) (issues: [#133600](https://github.com/elastic/elasticsearch/issues/133600), [#133574](https://github.com/elastic/elasticsearch/issues/133574), [#133607](https://github.com/elastic/elasticsearch/issues/133607)) +* Fix lookup index resolution when field-caps returns empty mapping [#132138](https://github.com/elastic/elasticsearch/pull/132138) (issue: [#132105](https://github.com/elastic/elasticsearch/issues/132105)) +* Fix projection generation when pruning left join [#135446](https://github.com/elastic/elasticsearch/pull/135446) +* Fix union types lost attributes in `StubRelation` for inlinestats [#135547](https://github.com/elastic/elasticsearch/pull/135547) +* Fix: prevent duplication of "invalid index name" string in the final exception error message [#130027](https://github.com/elastic/elasticsearch/pull/130027) +* Fixes `countDistinctWithConditions` in csv-spec tests [#135097](https://github.com/elastic/elasticsearch/pull/135097) (issue: [#134380](https://github.com/elastic/elasticsearch/issues/134380)) +* Handle right hand side of Inline Stats coming optimized with `LocalRelation` shortcut [#135011](https://github.com/elastic/elasticsearch/pull/135011) +* Limit when we push topn to lucene [#134497](https://github.com/elastic/elasticsearch/pull/134497) +* Make `ResolveUnionTypes` rule stateless [#136492](https://github.com/elastic/elasticsearch/pull/136492) +* Mark LOOKUP JOIN as `ExecutesOn.Any` by default [#133064](https://github.com/elastic/elasticsearch/pull/133064) +* Pass fix size instead of `maxPageSize` to `LuceneTopNOperator` scorer [#135767](https://github.com/elastic/elasticsearch/pull/135767) +* Replace any Attribute type when pushing down past Project [#135295](https://github.com/elastic/elasticsearch/pull/135295) (issue: [#134407](https://github.com/elastic/elasticsearch/issues/134407)) +* Telemetry with inlinestats [#134309](https://github.com/elastic/elasticsearch/pull/134309) +* Throw 4xx instead of 5xx for ESQL malformed query params [#134879](https://github.com/elastic/elasticsearch/pull/134879) (issue: [#134618](https://github.com/elastic/elasticsearch/issues/134618)) +* TopNOperator, release Row on failure [#130330](https://github.com/elastic/elasticsearch/pull/130330) (issue: [#130215](https://github.com/elastic/elasticsearch/issues/130215)) +* [main]Prepare Index Like fix for backport to 9.1 and 8.19 [#130947](https://github.com/elastic/elasticsearch/pull/130947) + +ILM+SLM: +* Add origin to client in SLM task [#135484](https://github.com/elastic/elasticsearch/pull/135484) +* Avoid counting snapshot failures twice in SLM [#136759](https://github.com/elastic/elasticsearch/pull/136759) +* Avoid running asynchronous ILM actions while ILM is stopped [#133683](https://github.com/elastic/elasticsearch/pull/133683) (issues: [#99859](https://github.com/elastic/elasticsearch/issues/99859), [#81234](https://github.com/elastic/elasticsearch/issues/81234), [#85097](https://github.com/elastic/elasticsearch/issues/85097)) +* Correctly update SLM stats with master shutdown [#134152](https://github.com/elastic/elasticsearch/pull/134152) +* Fix log formatting in `SnapshotLifecycleTask` [#136709](https://github.com/elastic/elasticsearch/pull/136709) + +Indices APIs: +* Reindex-from-remote: Validate basic auth params [#136501](https://github.com/elastic/elasticsearch/pull/136501) (issue: [#135925](https://github.com/elastic/elasticsearch/issues/135925)) +* Updating `TransportSimulateIndexTemplateAction.resolveTemplate()` to account for data stream overrides [#132131](https://github.com/elastic/elasticsearch/pull/132131) (issue: [#131425](https://github.com/elastic/elasticsearch/issues/131425)) + +Infra/Core: +* Fix offset handling in Murmur3Hasher [#133193](https://github.com/elastic/elasticsearch/pull/133193) + +Infra/Scripting: +* Fixed GeneralScriptException to return 400 http status code [#133659](https://github.com/elastic/elasticsearch/pull/133659) + +Ingest Node: +* Add support for flexible access pattern to `NormalizeForStreamProcessor` [#134524](https://github.com/elastic/elasticsearch/pull/134524) +* Fix append processor `ignore_empty_values` edge case [#136649](https://github.com/elastic/elasticsearch/pull/136649) +* Fixing conditional processor mutability bugs [#134936](https://github.com/elastic/elasticsearch/pull/134936) + +Logs: +* Fix logsdb settings provider mapping filters [#136119](https://github.com/elastic/elasticsearch/pull/136119) (issue: [#136107](https://github.com/elastic/elasticsearch/issues/136107)) + +Machine Learning: +* Add exception for perform embedding inference requests with query provided [#131641](https://github.com/elastic/elasticsearch/pull/131641) +* Adjust jinaai rerank response parser to handle document field as string or object [#136751](https://github.com/elastic/elasticsearch/pull/136751) +* Allow timeout during trained model download process [#129003](https://github.com/elastic/elasticsearch/pull/129003) +* Clean up inference indices on failed endpoint creation [#136577](https://github.com/elastic/elasticsearch/pull/136577) (issue: [#123726](https://github.com/elastic/elasticsearch/issues/123726)) +* Cohere service Model Id field is required [#136017](https://github.com/elastic/elasticsearch/pull/136017) +* Ensure queued `AbstractRunnables` are notified when executor stops [#135966](https://github.com/elastic/elasticsearch/pull/135966) (issue: [#134651](https://github.com/elastic/elasticsearch/issues/134651)) +* Fix model assignment error handling and assignment explanation generation [#133916](https://github.com/elastic/elasticsearch/pull/133916) +* Implementing latency improvements for EIS integration [#133861](https://github.com/elastic/elasticsearch/pull/133861) +* Improve memory estimation methods accuracy in `TrainedModelAssignmentRebalancer` and related classes [#133930](https://github.com/elastic/elasticsearch/pull/133930) +* Inference API disable partial search results [#132362](https://github.com/elastic/elasticsearch/pull/132362) +* Release cluster state [#136769](https://github.com/elastic/elasticsearch/pull/136769) (issue: [#123243](https://github.com/elastic/elasticsearch/issues/123243)) +* Remove rate limit field from services API for EIS [#135838](https://github.com/elastic/elasticsearch/pull/135838) +* Return 429 status when `RequestExecutorService` queue full [#134178](https://github.com/elastic/elasticsearch/pull/134178) +* Sync Inference with Trained Model stats [#130544](https://github.com/elastic/elasticsearch/pull/130544) (issue: [#130339](https://github.com/elastic/elasticsearch/issues/130339)) + +Mapping: +* Store full path in `_ignored` when ignoring dynamic array field [#136315](https://github.com/elastic/elasticsearch/pull/136315) +* [Downsampling++] Allow merging of passthrough mappers with object mappers under certain conditions [#135431](https://github.com/elastic/elasticsearch/pull/135431) + +Network: +* Fix `NullPointerException` in transport trace logger [#132243](https://github.com/elastic/elasticsearch/pull/132243) + +Search: +* Adjust date docvalue formatting to return 4xx instead of 5xx [#132414](https://github.com/elastic/elasticsearch/pull/132414) +* Apply source excludes early when retrieving the `_inference_fields` [#135897](https://github.com/elastic/elasticsearch/pull/135897) +* Correct exception for missing nested path [#132408](https://github.com/elastic/elasticsearch/pull/132408) +* Handle special regex cases for version fields [#132511](https://github.com/elastic/elasticsearch/pull/132511) +* Initialize `TermsEnum` eagerly [#136279](https://github.com/elastic/elasticsearch/pull/136279) +* Support returning default `index_options` for `semantic_text` fields when `include_defaults` is true [#129967](https://github.com/elastic/elasticsearch/pull/129967) +* Switch to Sending a Bad Request User When Function Score Query Generates Negative Scores [#133357](https://github.com/elastic/elasticsearch/pull/133357) (issue: [#133358](https://github.com/elastic/elasticsearch/issues/133358)) +* Tests for FORK's evaluation of field names used in `field_caps` resolve calls [#131723](https://github.com/elastic/elasticsearch/pull/131723) + +Security: +* Configurable HTTP read and connect timeouts for url based SAML metadata resolution [#136058](https://github.com/elastic/elasticsearch/pull/136058) +* Optimize Index Permission Automatons for Has Privileges [#136625](https://github.com/elastic/elasticsearch/pull/136625) + +TSDB: +* Fix warning when creating an OTel data stream [#133952](https://github.com/elastic/elasticsearch/pull/133952) (issue: [#132918](https://github.com/elastic/elasticsearch/issues/132918)) + +Transform: +* Fix stuck in STOPPING by retrying the startup task indefinitely until it succeeds [#132048](https://github.com/elastic/elasticsearch/pull/132048) (issue: [#128221](https://github.com/elastic/elasticsearch/issues/128221)) + +Vector Search: +* Bugfix 136545 [#136556](https://github.com/elastic/elasticsearch/pull/136556) +* Bugfix/disable matches highlight knn [#136563](https://github.com/elastic/elasticsearch/pull/136563) +* Cardinality Aggregator Throws `UnsupportedOperationException` When Field Type is Vector [#135994](https://github.com/elastic/elasticsearch/pull/135994) +* Fix _inference_fields handling on old indices [#136312](https://github.com/elastic/elasticsearch/pull/136312) (issue: [#136130](https://github.com/elastic/elasticsearch/issues/136130)) +* Have top level knn searches tracked in query stats [#132548](https://github.com/elastic/elasticsearch/pull/132548) + + + +## 9.0.8 [elasticsearch-9.0.8-release-notes] + +### Highlights [elasticsearch-9.0.8-highlights] + ::::{dropdown} Security advisory -The 9.1.5 release contains fixes for potential security vulnerabilities. Please see our [security advisory](https://discuss.elastic.co/c/announcements/security-announcements/31) for more details. +The 9.0.8 release contains fixes for potential security vulnerabilities. Please see our [security advisory](https://discuss.elastic.co/c/announcements/security-announcements/31) for more details. :::: -### Features and enhancements [elasticsearch-9.1.5-features-enhancements] +### Features and enhancements [elasticsearch-9.0.8-features-enhancements] Audit: * Change reindex to use ::es-redacted:: filtering [#135414](https://github.com/elastic/elasticsearch/pull/135414) @@ -47,19 +510,17 @@ Audit: Authorization: * [Island Browser] Add `manage`, `create_index`, `read`, `index`, `write`, `delete`, permission for third party agent indices `kibana_system` [#134636](https://github.com/elastic/elasticsearch/pull/134636) (issue: [#134136](https://github.com/elastic/elasticsearch/issues/134136)) +Infra/Plugins: +* Add Reason field to elastic-agent upgrade details metadata [#134711](https://github.com/elastic/elasticsearch/pull/134711) -### Fixes [elasticsearch-9.1.5-fixes] + +### Fixes [elasticsearch-9.0.8-fixes] Aggregations: * Propagates filter() to aggregation functions' surrogates [#134461](https://github.com/elastic/elasticsearch/pull/134461) (issue: [#134380](https://github.com/elastic/elasticsearch/issues/134380)) -Codec: -* Address es819 tsdb doc values format performance bug [#135505](https://github.com/elastic/elasticsearch/pull/135505) (issue: [#135340](https://github.com/elastic/elasticsearch/issues/135340)) - ES|QL: -* Ban Limit + `MvExpand` before remote Enrich [#135051](https://github.com/elastic/elasticsearch/pull/135051) * Fix async get results with inconsistent headers [#135078](https://github.com/elastic/elasticsearch/pull/135078) (issue: [#135042](https://github.com/elastic/elasticsearch/issues/135042)) -* Fix expiration time in ES|QL async [#135209](https://github.com/elastic/elasticsearch/pull/135209) (issue: [#135169](https://github.com/elastic/elasticsearch/issues/135169)) Engine: * Bypass MMap arena grouping as this has caused issues with too many regions being mapped [#135012](https://github.com/elastic/elasticsearch/pull/135012) @@ -68,9 +529,6 @@ Engine: Geo: * `CentroidCalculator` does not return negative summation weights [#135176](https://github.com/elastic/elasticsearch/pull/135176) (issue: [#131861](https://github.com/elastic/elasticsearch/issues/131861)) -Infra/Core: -* Bug fix: Facilitate second retrieval of the same value [#134790](https://github.com/elastic/elasticsearch/pull/134790) (issue: [#134770](https://github.com/elastic/elasticsearch/issues/134770)) - Infra/Node Lifecycle: * Fix systemd notify to use a shared arena [#135235](https://github.com/elastic/elasticsearch/pull/135235) @@ -81,29 +539,39 @@ Machine Learning: * Add .reindexed-v7-ml-anomalies-* to anomaly results template index pattern [#135270](https://github.com/elastic/elasticsearch/pull/135270) * Gracefully shutdown model deployment when node is removed from assignment routing [#134673](https://github.com/elastic/elasticsearch/pull/134673) * Reset health status on successful empty checkpoint [#135653](https://github.com/elastic/elasticsearch/pull/135653) (issue: [#135650](https://github.com/elastic/elasticsearch/issues/135650)) -* Tolerate mixed types in datafeed stats sort [#135096](https://github.com/elastic/elasticsearch/pull/135096) Mapping: -* Avoid holding references to `SearchExecutionContext` in `SourceConfirmedTextQuery` [#134887](https://github.com/elastic/elasticsearch/pull/134887) * Fix for creating semantic_text fields on pre-8.11 indices crashing Elasticsearch [#135845](https://github.com/elastic/elasticsearch/pull/135845) -* Fixed match only text block loader not working when a keyword multi field is present [#134582](https://github.com/elastic/elasticsearch/pull/134582) Search: * Fix KQL case-sensitivity for keyword fields in ES|QL [#135776](https://github.com/elastic/elasticsearch/pull/135776) (issue: [#135772](https://github.com/elastic/elasticsearch/issues/135772)) +* Prevent field caps from failing due to can match failure [#134134](https://github.com/elastic/elasticsearch/pull/134134) (issue: [#116106](https://github.com/elastic/elasticsearch/issues/116106)) Transform: * Fix a bug in the GET _transform API that incorrectly claims some Transform configurations are missing [#134963](https://github.com/elastic/elasticsearch/pull/134963) (issue: [#134263](https://github.com/elastic/elasticsearch/issues/134263)) * Prevent Transform from queuing too many PIT close requests by waiting for PIT to close before finishing the checkpoint [#134955](https://github.com/elastic/elasticsearch/pull/134955) (issue: [#134925](https://github.com/elastic/elasticsearch/issues/134925)) -## 9.0.8 [elasticsearch-9.0.8-release-notes] -### Highlights [elasticsearch-9.0.8-highlights] + +## 9.1.5 [elasticsearch-9.1.5-release-notes] + +### Highlights [elasticsearch-9.1.5-highlights] ::::{dropdown} Security advisory -The 9.0.8 release contains fixes for potential security vulnerabilities. Please see our [security advisory](https://discuss.elastic.co/c/announcements/security-announcements/31) for more details. +The 9.1.5 release contains fixes for potential security vulnerabilities. Please see our [security advisory](https://discuss.elastic.co/c/announcements/security-announcements/31) for more details. :::: -### Features and enhancements [elasticsearch-9.0.8-features-enhancements] +::::{dropdown} Prevent LIMIT + MV_EXPAND before remote ENRICH +Queries using LIMIT followed by MV_EXPAND before a remote ENRICH can produce incorrect results due to distributed execution semantics. +These queries are now unsupported and produce an error. Example: +[source,yaml] +---------------------------- +FROM *:events | SORT @timestamp | LIMIT 2 | MV_EXPAND ip | ENRICH _remote:clientip_policy ON ip +---------------------------- +To avoid this error, reorder your query, for example by moving ENRICH earlier in the pipeline. +:::: + +### Features and enhancements [elasticsearch-9.1.5-features-enhancements] Audit: * Change reindex to use ::es-redacted:: filtering [#135414](https://github.com/elastic/elasticsearch/pull/135414) @@ -111,17 +579,19 @@ Audit: Authorization: * [Island Browser] Add `manage`, `create_index`, `read`, `index`, `write`, `delete`, permission for third party agent indices `kibana_system` [#134636](https://github.com/elastic/elasticsearch/pull/134636) (issue: [#134136](https://github.com/elastic/elasticsearch/issues/134136)) -Infra/Plugins: -* Add Reason field to elastic-agent upgrade details metadata [#134711](https://github.com/elastic/elasticsearch/pull/134711) - -### Fixes [elasticsearch-9.0.8-fixes] +### Fixes [elasticsearch-9.1.5-fixes] Aggregations: * Propagates filter() to aggregation functions' surrogates [#134461](https://github.com/elastic/elasticsearch/pull/134461) (issue: [#134380](https://github.com/elastic/elasticsearch/issues/134380)) +Codec: +* Address es819 tsdb doc values format performance bug [#135505](https://github.com/elastic/elasticsearch/pull/135505) (issue: [#135340](https://github.com/elastic/elasticsearch/issues/135340)) + ES|QL: +* Ban Limit + `MvExpand` before remote Enrich [#135051](https://github.com/elastic/elasticsearch/pull/135051) * Fix async get results with inconsistent headers [#135078](https://github.com/elastic/elasticsearch/pull/135078) (issue: [#135042](https://github.com/elastic/elasticsearch/issues/135042)) +* Fix expiration time in ES|QL async [#135209](https://github.com/elastic/elasticsearch/pull/135209) (issue: [#135169](https://github.com/elastic/elasticsearch/issues/135169)) Engine: * Bypass MMap arena grouping as this has caused issues with too many regions being mapped [#135012](https://github.com/elastic/elasticsearch/pull/135012) @@ -130,6 +600,9 @@ Engine: Geo: * `CentroidCalculator` does not return negative summation weights [#135176](https://github.com/elastic/elasticsearch/pull/135176) (issue: [#131861](https://github.com/elastic/elasticsearch/issues/131861)) +Infra/Core: +* Bug fix: Facilitate second retrieval of the same value [#134790](https://github.com/elastic/elasticsearch/pull/134790) (issue: [#134770](https://github.com/elastic/elasticsearch/issues/134770)) + Infra/Node Lifecycle: * Fix systemd notify to use a shared arena [#135235](https://github.com/elastic/elasticsearch/pull/135235) @@ -140,13 +613,15 @@ Machine Learning: * Add .reindexed-v7-ml-anomalies-* to anomaly results template index pattern [#135270](https://github.com/elastic/elasticsearch/pull/135270) * Gracefully shutdown model deployment when node is removed from assignment routing [#134673](https://github.com/elastic/elasticsearch/pull/134673) * Reset health status on successful empty checkpoint [#135653](https://github.com/elastic/elasticsearch/pull/135653) (issue: [#135650](https://github.com/elastic/elasticsearch/issues/135650)) +* Tolerate mixed types in datafeed stats sort [#135096](https://github.com/elastic/elasticsearch/pull/135096) Mapping: +* Avoid holding references to `SearchExecutionContext` in `SourceConfirmedTextQuery` [#134887](https://github.com/elastic/elasticsearch/pull/134887) * Fix for creating semantic_text fields on pre-8.11 indices crashing Elasticsearch [#135845](https://github.com/elastic/elasticsearch/pull/135845) +* Fixed match only text block loader not working when a keyword multi field is present [#134582](https://github.com/elastic/elasticsearch/pull/134582) Search: * Fix KQL case-sensitivity for keyword fields in ES|QL [#135776](https://github.com/elastic/elasticsearch/pull/135776) (issue: [#135772](https://github.com/elastic/elasticsearch/issues/135772)) -* Prevent field caps from failing due to can match failure [#134134](https://github.com/elastic/elasticsearch/pull/134134) (issue: [#116106](https://github.com/elastic/elasticsearch/issues/116106)) Transform: * Fix a bug in the GET _transform API that incorrectly claims some Transform configurations are missing [#134963](https://github.com/elastic/elasticsearch/pull/134963) (issue: [#134263](https://github.com/elastic/elasticsearch/issues/134263)) @@ -2013,4 +2488,6 @@ Vector Search: * Return appropriate error on null dims update instead of npe [#125716](https://github.com/elastic/elasticsearch/pull/125716) Watcher: -* Watcher history index has too many indexed fields - [#117701](https://github.com/elastic/elasticsearch/pull/117701) (issue: [#71479](https://github.com/elastic/elasticsearch/issues/71479)) \ No newline at end of file +* Watcher history index has too many indexed fields - [#117701](https://github.com/elastic/elasticsearch/pull/117701) (issue: [#71479](https://github.com/elastic/elasticsearch/issues/71479)) + + From 4d9abe077b6a6ece7f4151de9a98a659178dc4f7 Mon Sep 17 00:00:00 2001 From: Charlotte Hoblik Date: Thu, 23 Oct 2025 09:19:40 +0200 Subject: [PATCH 2/3] fix breaking changes and deprications --- docs/release-notes/breaking-changes.md | 4 +--- docs/release-notes/deprecations.md | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/release-notes/breaking-changes.md b/docs/release-notes/breaking-changes.md index b5bd003b28a00..52293f6a6507d 100644 --- a/docs/release-notes/breaking-changes.md +++ b/docs/release-notes/breaking-changes.md @@ -12,10 +12,10 @@ If you are migrating from a version prior to version 9.0, you must first upgrade % ## Next version [elasticsearch-nextversion-breaking-changes] +## 9.2.0 [elasticsearch-9.2.0-breaking-changes] ```{applies_to} stack: ga 9.2.0 ``` -## 9.2.0 [elasticsearch-9.2.0-breaking-changes] Ingest Node: * Simulate API: Return 400 on invalid processor(s) [#130325](https://github.com/elastic/elasticsearch/pull/130325) (issue: [#120731](https://github.com/elastic/elasticsearch/issues/120731)) @@ -26,8 +26,6 @@ Mapping: Vector Search: * Enable `exclude_source_vectors` by default for new indices [#131907](https://github.com/elastic/elasticsearch/pull/131907) - - ## 9.0.8 [elasticsearch-9.0.8-breaking-changes] There are no breaking changes associated with this release. diff --git a/docs/release-notes/deprecations.md b/docs/release-notes/deprecations.md index 66bb6adcb1c84..6891445206565 100644 --- a/docs/release-notes/deprecations.md +++ b/docs/release-notes/deprecations.md @@ -16,16 +16,14 @@ To give you insight into what deprecated features you’re using, {{es}}: % ## Next version [elasticsearch-nextversion-deprecations] +## 9.2.0 [elasticsearch-9.2.0-deprecations] ```{applies_to} stack: ga 9.2.0 ``` -## 9.2.0 [elasticsearch-9.2.0-deprecations] Cluster Coordination: * Remove `PeerFinder` request timeout [#134365](https://github.com/elastic/elasticsearch/pull/134365) - - ## 9.0.8 [elasticsearch-9.0.8-deprecations] There are no deprecations associated with this release. From d0b8fe759e11cbf5f643adc0cc7504f5f2fc8c62 Mon Sep 17 00:00:00 2001 From: Charlotte Hoblik Date: Thu, 23 Oct 2025 09:29:51 +0200 Subject: [PATCH 3/3] fix index.md --- .../release-notes/changelog-bundles/9.1.5.yml | 7 ++-- .../release-notes/changelog-bundles/9.2.0.yml | 21 ++++++------ docs/release-notes/index.md | 32 +++++++++---------- 3 files changed, 32 insertions(+), 28 deletions(-) diff --git a/docs/release-notes/changelog-bundles/9.1.5.yml b/docs/release-notes/changelog-bundles/9.1.5.yml index 4c07b9dbc2cd9..c2f96e7659163 100644 --- a/docs/release-notes/changelog-bundles/9.1.5.yml +++ b/docs/release-notes/changelog-bundles/9.1.5.yml @@ -74,10 +74,11 @@ changelogs: body: |- Queries using LIMIT followed by MV_EXPAND before a remote ENRICH can produce incorrect results due to distributed execution semantics. These queries are now unsupported and produce an error. Example: - [source,yaml] - ---------------------------- + + ```yaml FROM *:events | SORT @timestamp | LIMIT 2 | MV_EXPAND ip | ENRICH _remote:clientip_policy ON ip - ---------------------------- + ``` + To avoid this error, reorder your query, for example by moving ENRICH earlier in the pipeline. pr: 135051 - pr: 135078 diff --git a/docs/release-notes/changelog-bundles/9.2.0.yml b/docs/release-notes/changelog-bundles/9.2.0.yml index 24dfa95bd78fa..5065b05479d9c 100644 --- a/docs/release-notes/changelog-bundles/9.2.0.yml +++ b/docs/release-notes/changelog-bundles/9.2.0.yml @@ -98,10 +98,11 @@ changelogs: title: Add remote index support to LOOKUP JOIN body: |- Queries containing LOOKUP JOIN now can be preformed on cross-cluster indices, for example: - [source,yaml] - ---------------------------- + + ```yaml FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100 - ---------------------------- + ``` + pr: 129013 - pr: 129090 summary: Enable force inference endpoint deleting for invalid models and after stopping model deployment fails @@ -1271,8 +1272,8 @@ changelogs: - all other restrictions that exist for `dense_vector` fields still apply To utilize the format, its just like any other: - [source,yaml] - ---------------------------- + + ```yaml PUT vectors { "mappings": { @@ -1281,10 +1282,11 @@ changelogs: } } } - ---------------------------- + ``` + Querying is just like any other field. - [source,yaml] - ---------------------------- + + ```yaml POST vectors/_search{ "query": { "knn": { @@ -1294,7 +1296,8 @@ changelogs: } } } - ---------------------------- + ``` + `num_candidates` can be used for tuning approximate nature of the search. Or, more granular control can be provided by setting `visit_percentage` directly. pr: 135299 diff --git a/docs/release-notes/index.md b/docs/release-notes/index.md index 4508b06fbb76d..d058f0bfb6d8b 100644 --- a/docs/release-notes/index.md +++ b/docs/release-notes/index.md @@ -56,8 +56,8 @@ Current restrictions: - all other restrictions that exist for `dense_vector` fields still apply To utilize the format, its just like any other: -[source,yaml] ----------------------------- + +```yaml PUT vectors { "mappings": { @@ -66,10 +66,11 @@ PUT vectors } } } ----------------------------- +``` + Querying is just like any other field. -[source,yaml] ----------------------------- + +```yaml POST vectors/_search{ "query": { "knn": { @@ -79,7 +80,8 @@ POST vectors/_search{ } } } ----------------------------- +``` + `num_candidates` can be used for tuning approximate nature of the search. Or, more granular control can be provided by setting `visit_percentage` directly. :::: @@ -105,10 +107,11 @@ for a variety of use cases. ::::{dropdown} Add remote index support to LOOKUP JOIN Queries containing LOOKUP JOIN now can be preformed on cross-cluster indices, for example: -[source,yaml] ----------------------------- + +```yaml FROM logs-*, remote:logs-* | LOOKUP JOIN clients on ip | SORT timestamp | LIMIT 100 ----------------------------- +``` + :::: ::::{dropdown} New lucene 10.3.0 release @@ -492,8 +495,6 @@ Vector Search: * Fix _inference_fields handling on old indices [#136312](https://github.com/elastic/elasticsearch/pull/136312) (issue: [#136130](https://github.com/elastic/elasticsearch/issues/136130)) * Have top level knn searches tracked in query stats [#132548](https://github.com/elastic/elasticsearch/pull/132548) - - ## 9.0.8 [elasticsearch-9.0.8-release-notes] ### Highlights [elasticsearch-9.0.8-highlights] @@ -551,8 +552,6 @@ Transform: * Fix a bug in the GET _transform API that incorrectly claims some Transform configurations are missing [#134963](https://github.com/elastic/elasticsearch/pull/134963) (issue: [#134263](https://github.com/elastic/elasticsearch/issues/134263)) * Prevent Transform from queuing too many PIT close requests by waiting for PIT to close before finishing the checkpoint [#134955](https://github.com/elastic/elasticsearch/pull/134955) (issue: [#134925](https://github.com/elastic/elasticsearch/issues/134925)) - - ## 9.1.5 [elasticsearch-9.1.5-release-notes] ### Highlights [elasticsearch-9.1.5-highlights] @@ -564,10 +563,11 @@ The 9.1.5 release contains fixes for potential security vulnerabilities. Please ::::{dropdown} Prevent LIMIT + MV_EXPAND before remote ENRICH Queries using LIMIT followed by MV_EXPAND before a remote ENRICH can produce incorrect results due to distributed execution semantics. These queries are now unsupported and produce an error. Example: -[source,yaml] ----------------------------- + +```yaml FROM *:events | SORT @timestamp | LIMIT 2 | MV_EXPAND ip | ENRICH _remote:clientip_policy ON ip ----------------------------- +``` + To avoid this error, reorder your query, for example by moving ENRICH earlier in the pipeline. ::::