Elasticsearch version 7.15.0
- Elasticsearch version 7.15.0

Elasticsearch version 7.15.0

Also see Breaking changes in 7.15.

Known issues

Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.
Aggregations: In Elasticsearch 7.14.0–7.15.0, when a cross-cluster search (CCS) request is proxied, the memory for the aggregations on the proxy node will not be freed. The trigger is cross-cluster search using aggregations where minimize roundtrips is not effective (for example, when minimize roundtrips is explicitly disabled, or implicitly disabled when using scroll, async and point-in-time searches).

This affects Kibana CCS aggregations because Kibana uses async search by default. This issue can also happen in all modes of remote connections configured for cross-cluster search (sniff and proxy). In sniff mode, we only connect to a subset of the remote nodes (by default 3). So if the remote node we want to send a request to is not one of those 3, we must send the request as a proxy request. The workaround is to periodically restart nodes with heap pressure.

We have fixed this issue in Elasticsearch 7.15.1 and later versions. For more details, see #78404.
Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)
The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

Breaking changes

Infra/Core

Remove quota-aware-fs plugin #76352 (issue: #70309)

Remove xpack actions for vector field #75017 (issues: #53076, #74974)

Deprecations

Store

Deprecate SimpleFS and replace it with NIOFS #75156

New features

EQL

Add multi-valued fields support. Part 1 #74567
Backwards compatibility support for multi value fields update #76605
Multi-value fields support #76610

Geo

Remove vector tile feature flag #76068

Infra/Node Lifecycle

Remove Node Shutdown API feature flag #76588 (issue: #70338)

Machine Learning

Add model_prune_window field to AD job config #75741 (issue: #1962)

Mapping

Unsigned longs should be compatible with index sorting #75599

Enhancements

Allocation

Clarify allocation explain if random shard chosen #75670
Prefer primaries in cluster allocation explain #76220

Authentication

Add kibana-system service account #76449
Support 2b and 2y prefixes in bcrypt #76083 (issues: #51132, #56086)
Support password_hash for Change Password API #75500

Authorization

Add threshold time logging for Authorized indices #75439
Fix privileges for GetRollupIndexCapabilities API #75614 (issue: #74779)
Granting kibana_system reserved role access to “all” privileges to .internal.alerts* index #76624

Cluster Coordination

Distinguish timeouts/failures in follower checker #75519
Indicate quorumness of discovered nodes earlier #74823
Warn on possible master service starvation #74820

Data streams

Add filter support to data stream aliases #74784 (issue: #66163)

EQL

Minimise CCS roundtrips #76076

Engine

Refresh after force-merge #76221 (issue: #74649)

Geo

Add mvt field type format to geo fields #75367 (issue: #73872)

ILM+SLM

Add phase configs to ILM telemetry #74575 (issue: #69152)
Add validation of the number_of_shards parameter in Shrink Action of ILM #74219 (issue: #72724)
Allow ILM move-to-step without action or name #75435 (issue: #58128)
Update ILM message for out of order phases error #75099 (issue: #70336)

Infra/CLI

In MultiCommand, show command list when required #75993

Infra/Logging

Adds minimal traceparent header support to Elasticsearch #74210

Infra/Plugins

Add support in plugins for a separate spi classloader #76288 (issue: #74448)

Infra/Resiliency

Print suggestion to view log on fatal error #75418 (issue: #61767)

Infra/Scripting

Add Fields API to aggregation scripts and field scripts #76325
Add ability to allow list instance methods on the script class #76045
Add fields api to runtime fields contexts #76265
Script: Def encoding parser #74840
Script: Fields API for Filter context #76119
Script: Fields API for Sort and Score scripts #75863 (issue: #61388)
Script: Mangle user function names #72892 (issue: #69742)
Script: ulong via fields API #76519 (issue: #64361)

Ingest

Add support for _meta field to ingest pipelines #75905
Configurable media_type for mustache template encoding on append processor #76210
Json processor: add_to_root_conflict_strategy option #74967
Json processor: allow duplicate keys #74956
Map iteration support for ForEach processor #64062 (issue: #55215)
Support for wildcards and override option for dot_expander processor #74601

License

Add persistent licensed feature tracking #76476

Machine Learning

Add datafeed_config field to anomaly detection job configs #74265
Adding datafeed_config to job in high level rest client #75338
Adding new p_value scoring heuristic to significant terms aggregation #75313
Delete expired annotations #75617 (issue: #75572)
Deleting a job now deletes the datafeed if necessary #76010
Integrating ML with the node shutdown API #75188
Throttle job audit msgs if delayed data occurs for consecutive buckets #75815 (issue: #73679)
Speed up training of regression and classification models on very large data sets #1941
Improve regression and classification training accuracy for small data sets #1960
Prune models for split fields (by, partition) that haven’t seen data updates for a given period of time #1962

Mapping

Handle runtime subfields when shadowing dynamic mappings #75595 (issue: #75454)

Network

Drop ReceiveTimeoutTransportException stack trace #75671
Ensure indexing_data is compressed appropriately #76321 (issue: #73497)
Respond with same compression scheme received #76372 (issue: #73497)

Other

Add resolve_during_rolling_upgrade field to deprecation issue #74226 (issue: #73091)

Packaging

Accept settings in snake case in Docker image #74327 (issue: #74036)
Change env var prefix in Docker settings support #76192 (issues: #74327, #76148)
Reduce default Docker image size in 7.x #75079
Tighten up write permissions in Docker image #70635

Query Languages

Introduce ParserUtils to consolidate code #76399

Recovery

Add peer recoveries using snapshot files when possible #76237 (issue: #73496)
Add peer recovery planners that take into account available snapshots #75840
Compute latest snapshot directly in TransportGetShardSnapshotAction #76254 (issue: #73496)
Keep track of data recovered from snapshots in RecoveryState #76499 (issue: #73496)

Reindex

Log reindex bulk request completion #75935

Add index disk usage API #74051 (issue: #74051)
Allow specifying index in pinned queries #74873 (issue: #67855)
Compress async search responses before storing #74766 (issue: #67594)
Emit multiple fields from a runtime field script #75108 (issue: #68203)
Replaced field_masking_span occurrences with respective ParseField #74718 (issue: #63527)
Search performance - better caching logic for queries on wildcard field #76035 (issue: #75848)
Support search slicing with point-in-time #74457 (issue: #65740)
Track Lucene field usage #74227 (issue: #73944)
Unsigned long should accept value with zero fraction #75646 (issue: #67565)

Security

A new search API for API keys - core search function #75335 (issues: #71023, #73705)
Add sort and pagination support for QueryApiKey API #76144 (issue: #71023)
HLRC support for query API key API #76520 (issues: #71023, #75335, #76144)
Query API key - Rest spec and yaml tests #76238 (issue: #71023)
Return file-backed service tokens from all nodes #75200 (issue: #74530)
OnChange listener now only runs when there is actual change #75207 (issue: #44701)

Snapshot/Restore

Add the ability to fetch the latest successful shard snapshot #75080 (issue: #73496)
Fix SnapshotInfo#toXContent delegation #75899
Implement Numeric Offset Parameter in Get Snapshots API #76233 (issue: #74350)
Include extra snapshot details in logs/APIs #75917
Prevent searchable snapshots indices to be shrunk/split #75227 (issue: #56595)
Return Total Result Count and Remaining Count in Get Snapshots Response #76150

Stats

Adding shard count to node stats api #75433
Aggregate data tier index stats separately from node stats #76322

Store

Replace internal usages of SimpleFS with NIOFS #74996

TLS

Add KeyUsage, ExtendedKeyUsage, CipherSuite & Protocol to SSL diagnos… #65634 (issue: #63784)
Solved issue “certutil: large passwords not set” #30944 #36689

Task Management

Adding the option to abort persistent tasks locally #74115
Include reason in cancellation exceptions #75332 (issue: #74825)

Transform

Align transform checkpoint range with date_histogram interval for better performance #74004 (issue: #62746)
Changing hard coded 10k page size limit to 65k #74651 (issue: #57719)
Improve performance by using point in time API for search #74984 (issue: #73481)
Optimize composite agg execution using ordered groupings #75424

Bug fixes

Aggregations

Fix docCountError calculation for multiple reduces #76391 (issues: #43874, #75667)
Fix wrong error upper bound when performing incremental reductions #43874 (issue: #40005)

Allocation

Active masternode crash caused by datanode Input/output error #76480 (issues: #1, #436)

Distributed

Wrap exception with SendRequestTransportException instead of TransportException #73551

Geo

Alias field does not work with geo_shape query #74895
Use contains relation for geo_point intersection over geo_point field #75272 (issue: #75103)

ILM+SLM

Changing wait_for_snapshot to check start time rather than finish time #75644

Indices APIs

Fix pending deletes timeout log message #75776

Infra/Core

Add system data streams to feature state snapshots #75902 (issue: #75860)
Fix for JarHell Bootstrap Check can yield false positives #76217 (issue: #75701)

Infra/Scripting

Fix the error message for instance methods that don’t exist #76512

Ingest

Execute EnrichPolicyRunner on a non dedicated master node #76881 (issue: #70436)

Machine Learning

Allow for larger models in the inference step for data frame analytics #76116 (issue: #76093)
Ignore exceptions while opening job after SIGTERM to JVM #75850
Prevent accidentally asking for more resources when scaling down and improve scaling size estimations #74691 (issue: #74709)
When setting upgrade mode ensure that internal actions don’t throw unnecessary permissions errors #75880
Fix potential “process stopped unexpectedly: Fatal error” for training regression and classification models #1997 (issue: #1956)

Network

Ensure replica requests are marked as index_data (#75008) #75014 (issue: #73497)
Fix memory/breaker leaks for outbound responses (#76474) #76536 (issues: #65921, #76474)

Remove invalid path from Open PIT rest spec #77609
Skip noop normalizing #74656

Security

Don’t remove warning headers on all failure #76434 (issue: #75739)

Snapshot/Restore

Fix Get Snapshots Request Cancellation with ignore_unavailable=true #78004 (issue: #77980)
Fix Spurious Warnings During Snapshot Delete #75911
Include full blob key in Azure read operations exceptions #75920 (issue: #75916)
Not throwing FileNotFoundException if BlobContainer has been deleted #75991

Transform

Fix potential deadlock when using stop_at_checkpoint #76034 (issue: #75846)

Upgrades

Ingest

Upgrade Tika to 1.27 for ingest #75191

Network

Upgrade to Netty 4.1.66 #76135

Query Languages

Upgrade ANTLR and move it into QL #76358 (issues: #74448, #76288, #76354)

Snapshot/Restore