Elasticsearch version 8.6.0

Elasticsearch version 8.6.0

Also see Breaking changes in 8.6.

Known issues

  • Orchestrators which use a settings.json file to configure Elasticsearch may encounter deadlocks during master elections (issue: #92812)

    To resolve the deadlock, remove the settings.json file and restart the affected node.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)

  • When the ingest attachment processor is used, the interaction of Tika with log4j 2.18.0 and higher (introduced in Elasticsearch 8.4.0) results in excessive logging. This logging is so excessive that it can lead to cluster instability, to the point where the cluster is unusable and nodes must be restarted. (issue: #91964). This issue is fixed in Elasticsearch 8.7.0 (#93878)

    To resolve the issue, upgrade to 8.7.0 or higher.

  • Shard rebalancing may temporarily unbalance cluster

    From 8.6.0 onwards the default shard rebalancing algorithm will compute the final desired balance and then make shard movements to reconcile the current state of the cluster with the desired state. However the order in which the shard movements take place may be skewed towards certain nodes, causing the cluster to become temporarily unbalanced while the reconciliation is ongoing. As always, once a node reaches a disk watermark it will not accept any additional shards, but this skew may result in nodes reaching their disk watermarks more often than expected in normal operation. Once the reconciliation process completes, the cluster will be balanced again.

    To avoid this problem, upgrade to 8.8.0 or later.

Bug fixes

Aggregations

  • GeoBoundsAggregations reject sub aggregations #91073 (issue: #91072)

Allocation

  • Avoid NPE when disassociateDeadNodes is executed for a node present in the desired balance #91659
  • Check NodesShutdownMetadata type before assuming restart #90807

Authentication

  • Fix time unit for connection request timeout of JWKs reload #92080
  • Improve performance for role mapping with DNs #92074
  • Improve robustness of JwkSet reloading #92081
  • Support stored authentication headers prior to version 6.7 #92221

Authorization

  • Make adding auth info to REST responses more robust #92168
  • Security remove datemath special handling #91047

Cluster Coordination

  • Fix TransportMasterNodeAction holding a CS reference needlessly #90724 (issue: #89220)
  • Include last-committed data in publication #92259 (issue: #90158)
  • Unsafe bootstrap memory optimization #92493

EQL

  • EQL sequences: support join on multi-values #89965

Graph

  • Fix potential issue with graph api’s timed out field in response #91006

Health

  • Don’t account for the unassigned reason when diagnosing NO_VALID_SHARD_COPY #92416
  • Fix NPE when evaluating the disk health for non-data nodes #92643
  • Use https in the short URLs for the shards_availability indicator #92310

Indices APIs

  • Trigger index settings providers when updating component templates #91615 (issue: #91592)

Infra/Core

  • Check reserved state in Metadata.isGlobalStateEquals #92124
  • Datastream unavailable exception metadata #91461
  • Fix BytesRefArray on append empty BytesRef #91364
  • Fix index expression options for requests with a single name or pattern #91231
  • Force init of Unbox in log4j #92377 (issue: #91964)
  • In file based settings, wait until security index is ready for role mappings #92173 (issue: #91939)
  • Index expression exclusions never trigger “not found” #90902
  • Update error states from inside the main state executor #90346 (issue: #90337)

Infra/Scripting

  • Fix compile with hex literals ending with d/f #91501 (issue: #88614)

Ingest Node

  • Fixing a race condition in EnrichCoordinatorProxyAction that can leave an item stuck in its queue #90688 (issue: #90598)

Machine Learning

  • Copy more settings when creating DF analytics destination index #91546 (issue: #89795)
  • Fix for No statistics error message #2410
  • Fix for No counts available error message #2414
  • Guard against input sequences that are too long for Question Answering models #91924
  • Improve performance of closing files before spawning #2424
  • Skip remote clusters when performing up front privileges validation for datafeeds #91895 (issue: #87832)
  • Support fields with commas in data frame analytics analyzed_fields #91710 (issue: #72541)
  • Validate rule filters are present on open anomaly detection api #92207

Mapping

  • Consolidate field name validation when parsing mappings and documents #91328
  • Fix handling empty key case in the terms aggregation #90822

Monitoring

  • Fix logstash loadavg (xpack cases) #90494
  • [Stack Monitoring] Update ES module mappings #90649

Network

  • Clean up on exception while chunking XContent #92024
  • Fix Chunked APIs sending incorrect responses to HEAD requests #92042 (issue: #92032)
  • Reject connection attempts while closing #92465

SQL

  • Fix NPE on logging when not tracking total hits #92425

Search

  • Allow different decay values depending on the score function #91195 (issue: #78887)
  • Fix timing bug with DFS profiling #92421

Snapshot/Restore

  • Simplify and optimize deduplication of RepositoryData for a non-caching repository instance #91851 (issue: #89952)

Store

  • Fix numOpenOutputs and modCount in ByteSizeCachingDirectory #92440 (issue: #92434)

Transform

  • Skip remote clusters when performing up front privileges validation #91788

Vector Search

  • Make knn search requests fully cancellable #90612

Deprecations

Allocation

  • Deprecate state field in /_cluster/reroute response #90399
  • Ensure balance threshold is at least 1 #92100

Ingest Node

  • Deprecate remove_binary default of false for ingest attachment processor #90460

Mapping

  • Deprecate silently ignoring type, fields, copy_to and boost in metadata field definition #90989 (issue: #35389)

Enhancements

Allocation

  • Clear up forecasted write load and shard size from previous write index during rollovers #91590
  • Forecast average shard size during rollovers #91561
  • Forecast write load during rollovers #91425
  • Improve shard balancing #91603
  • Introduce desired-balance allocator #91343
  • Limit shard realocation retries #90296
  • Prevalidate node removal API #88952
  • Set default cluster.routing.allocation.balance.disk_usage #91951
  • Store write load in the IndexMetadata during data streams rollovers #91019
  • Update the default cluster.routing.allocation.balance.disk_usage #92065
  • DesiredBalance: expose it via _internal/desired_balance #91038 (issue: #90583)

Authorization

  • [Fleet] Added logs-elastic_agent* read privileges to kibana_system #91701

CRUD

  • Keep track of average shard write load #90768 (issue: #90102)

Geo

  • Centroid aggregation for cartesian points and shapes #89216 (issue: #90156)
  • Improve H3#hexRing logic and add H3#areNeighborCells method #91140
  • Move SpatialUtils to geo library #88088 (issue: #86607)
  • Reduce number of object allocations in H3#geoToH3 and speed up computations #91492
  • Support cartesian_bounds aggregation on point and shape #91298 (issue: #90157)

ILM+SLM

  • ILM: Get policy support wildcard name #89238

Infra/Core

  • Handle APM global labels as affix setting #91438 (issue: #91278)
  • Improve date math exclusions in expressions #90298
  • Introduce a phase to use String.equals on constant strings, rather than def equality #91362 (issue: #91235)
  • More actionable error for ancient indices #91243
  • Operator/index templates #90143
  • Operator/ingest #89735
  • Transport threads and _hot_threads #90482 (issue: #90334)
  • Upgrade XContent to Jackson 2.14.0 and enable Fast Double Parser #90553

Infra/Plugins

  • Create placeholder plugin when loading stable plugins #90870
  • Example stable plugin #90805
  • Make extendedPlugins, HasNativeController and moduleName optional in plugin descriptor #90835
  • Rename NamedComponent name parameter to value #91306

Infra/Scripting

  • Use an explicit null check for null receivers in painless, rather than an NPE #91347 (issue: #91236)

Machine Learning

  • Add a filter parameter to frequent items #91137
  • Add a regex to the output of the categorize_text aggregation #90723
  • Add ability to filter and sort buckets by change_point numeric values #91299
  • Add api to update trained model deployment number_of_allocations #90728
  • Alias timestamp to @timestamp in anomaly detection results index #90812
  • Allow model_aliases to be used with Pytorch trained models #91296
  • Allow overriding timestamp field to null in file structure finder #90764
  • Audit a message every day the datafeed has seen no data #91774
  • Low priority trained model deployments #91234 (issue: #91024)
  • Provide additional information about anomaly score factors #90675

Mapping

  • Don’t create IndexCaps objects when recording unmapped fields #90806 (issue: #90796)
  • aggregate metric double add a max min validation #90381

Recovery

  • Remove resize index settings once shards are started #90391 (issue: #90127)

Rollup

  • Test downsample runtime fields and security #90593

Search

  • Add LimitedOffsetsEnum to Limited offset token #86110 (issue: #86109)
  • Add profiling and documentation for dfs phase #90536 (issue: #89713)
  • Bulk merge field-caps responses using mapping hash #86323
  • Enhance nested depth tracking when parsing queries #90425
  • Expose telemetry about search usage #91528
  • Return docs when using nested mappings in archive indices #90585 (issue: #90523)
  • Use IndexOrDocValues query for IP range queries #90303 (issue: #83658)

Snapshot/Restore

  • Increase snaphot pool max size to 10 #90282 (issue: #89608)
  • Tie snapshot speed to node bandwidth settings #91021 (issue: #57023)

Store

  • Allow plugins to wrap Lucene directories created by the IndexModule #91556

TLS

  • Add certificate start/expiry dates to SSL Diagnostic message #89461

TSDB

  • Generate index.routing_path from dynamic mapping templates #90552 (issue: #90528)
  • Support malformed numbers in synthetic _source #90428
  • Support synthetic _source for _doc_count field #91465
  • Synthetic _source: support field in many cases #89950
  • Synthetic _source: ignore_malformed for ip #90038
  • Synthetic _source: support wildcard field #90196

Transform

  • Add a health section to transform stats #90760
  • Support aggregate_metric_double field type in transform aggregations #91045

Vector Search

  • Add profiling information for knn vector queries #90200

New features

Distributed

  • Add “index” and “search” node roles with feature flag and setting #90993

EQL

Health

  • Use chunked encoding for RestGetHealthAction #91515 (issue: #90223)
  • [HealthAPI] Use the RestCancellableNodeClient infrastructure #91587

Machine Learning

  • Make categorize_text aggregation GA #88600

Vector Search

  • Add fielddata and scripting support for byte-sized vectors #91184
  • Add support for indexing byte-sized knn vectors #90774

Regressions

Infra/Core

  • Revert “Remove ImmutableOpenMap from snapshot services” #90287

Upgrades

Infra/Logging

Network

  • Upgrade to Netty 4.1.82.Final #90604
  • Upgrade to Netty 4.1.84 #91271

Snapshot/Restore

  • Upgrade GCS SDK to 2.13.1 #92327