Elasticsearch version 7.13.0

Elasticsearch version 7.13.0

Also see Breaking changes in 7.13.

Known issues

  • Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

    This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.

  • If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs to the maximum value of 512. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.

  • Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:

    1. PUT _cluster/settings
    2. {
    3.   "persistent" : {
    4.     "snapshot.max_concurrent_operations" : 1
    5.   }
    6. }

    This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #73456.

  • If local and remote clusters are on different patch releases, response serialization fails for requests to a remote cluster that use the field capabilities API with CCS. Because Kibana uses this API internally, requests to CCS index patterns will also fail.

    This issue is fixed in Elasticsearch version 7.13.3 and later. For more details, see #74504.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)

  • The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

Deprecations

Aggregations

  • Deprecate date aggregations on boolean fields #70030 (issue: #59255)

Authentication

  • Deprecate the behavior of implicitly disabling file/native realm #69320 (issue: #50892)

Features/Indices APIs

  • Deprecate legacy index template API endpoints #71309 (issue: #71307)

Infra/Core

  • Add migration check for legacy role settings #71197 (issues: #54998, #71143)
  • Add multiple data paths deprecation to the deprecation API #71209 (issue: #71205)
  • Deprecate multiple path.data entries #71207 (issue: #71205)

Infra/Settings

  • Add clear deprecation around legacy role settings #71143 (issue: #54998)
  • Add deprecation warning for default value of action.destructive_requires_name #70932 (issues: #66908, #67543)

New features

EQL

  • Adds fields request field to the EQL request #68962 (issue: #68115)

Features/ILM+SLM

  • Make all the shrink action steps retryable #70107 (issue: #48183)

Features/Ingest

  • Add support for validating IPv4/IPv6 addresses to Convert processor #69989 (issue: #36145)
  • Add registered domain processor #67611

Infra/Core

  • Add API for resetting state of a SystemIndexPlugin #69469

Machine Learning

  • Adds new trained model alias API to simplify trained model updates and deployments #68922

Search

Enhancements

Aggregations

  • Improve handling of incompatible after key types in composite aggregations #70839 (issue: #70480)
  • Increase search.max_bucket default value by one #70645 (issue: #57042)
  • Modest memory savings in a date_histogram followed by a terms aggregation #68592
  • Speed up top_metrics on hot shards #70579 (issue: #70453)
  • Speed up aggregations with sub-aggregations #69806 (issues: #63643, #68871)
  • Speed up terms aggregation when alone #69377 (issue: #68871)
  • Speed up terms aggregation when not force merged #71241 (issue: #71086)
  • Use #updateTop to speed up InternalComposite#reduce #71278

Allocation

  • Improve awareness allocation explanation #69371
  • Skip zone/host awareness with auto-expand replicas #69334 (issues: #2869, #54151)

Analysis

  • Allow updateable flag for keyword_marker filter #65457 (issue: #65355)

Audit

Authentication

  • Add a deprecation message if a REST wrapper implementing plugin presents #66827
  • Service Accounts - Authentication with file tokens #70543
  • Service Accounts - Fleet integration #70724
  • Service Accounts - Get service account API #71315
  • Service Accounts - Initial bootstrap plumbing to add essential classes #70391
  • Service Accounts - New CLI tool for managing file tokens #70454
  • Service Accounts - delete index backed service account token #71165
  • Service Accounts - token name in response to Authenticate API #71382
  • Support metadata on API keys #70292 (issue: #48182)

Authorization

  • Add read permissions for apm_user role to APM fleet indices #68749
  • Include role names in access denied errors #69318 (issue: #42166)

Autoscaling

  • Frozen tier autoscaling decider based on shards #71042

Cluster Coordination

  • Include node roles in cluster state JSON response #71386 (issue: #71385)
  • Remove node attributes from cluster membership messages #69811

Distributed

  • Add fleet polling API for global checkpoint #71093
  • Reduce size of MANAGEMENT threadpool on small node #71171 (issue: #70435)

EQL

  • Allow Unicode escape sequences in strings #70514 (issue: #62832)
  • Improve null handling in the optimizer #70557

Engine

  • Support include_unloaded_segments in node stats #69682

Features/ILM+SLM

  • Add max_single_primary_size as a condition for the ILM rollover action #68917 (issues: #63026, #67842)
  • Reject creating ILM policies with phase timings that are not greater than or equal to the previous phase #70089 (issue: #70032)
  • Switch built-in policies to max_primary_shard_size #69995 (issue: #63026)

Features/Indices APIs

  • Date math support for aliases #67226 (issue: #20367)
  • Introduce separate shard limit for frozen shards #71392 (issues: #34021, #71042)
  • Support specifying multiple templates names in delete component template api #70314 (issue: #69973)
  • Support specifying multiple templates names in delete composable index template api #70094 (issue: #69973)

Features/Ingest

  • Accept more ingest simulate params as integers or strings #66197 (issues: #23823, #65992)
  • Extract device type from user agent info #69322
  • Network direction processor additions #68712
  • Summary option for listing ingest pipelines without their definitions #69756 (issue: #31954)
  • MurmurHash3 support for fingerprint processor #70632 (issue: #69182)

Features/Java Low Level REST Client

  • Support new data roles #66947

Features/Stats

Features/Watcher

  • Migrate watcher to system indices infrastructure #67588 (issue: #61656)

Infra/Core

  • Manage Fleet system indices within Elasticsearch #70689
  • Support mixed node versions in system index descriptors #71144

Infra/Logging

  • Only install templates for deprecation indices from elected master node #70057 (issues: #69918, #70020)

Infra/Scripting

  • Add Runtime Fields Contexts to Painless Execute API #71374 (issue: #70467)
  • Add a new ANTLR lexer for Painless suggestions #70517
  • Improve null def access error message #69226 (issue: #53129)
  • Make the available ScriptContexts accessible from ScriptService #70465
  • Whitelist the CIDR convenience API #71258 (issue: #60668)

Query Languages

  • Adds runtime_mappings to EQL and SQL requests #71356 (issue: #68116)

Machine Learning

  • Add new delete trained model aliases API #69195
  • Add runtime mappings to data frame analytics source config #69183 (issue: #65056)
  • Adding new _preview endpoint for data frame analytics #69453
  • Adding support for composite aggregations in anomaly detection #69970
  • Allow datafeed and job configs for datafeed preview API #70836 (issue: #70264)
  • Improve messages related to assigning machine learning jobs #69752
  • Add put and delete trained model alias APIs to high-level REST client #69214
  • Speed up training of regression and classification models for data sets with many features #1746
  • Adjust the syscall filter to allow mremap and avoid spurious audit logging #1819
  • Avoid overfitting in final training by scaling regularizers to account for the difference in the number of training examples. This results in a better match between train and test errors for classification and regression and often slightly improved test errors #1755

Mapping

  • Add calculated numeric fields #69531
  • Allow specify dynamic templates in bulk request #69948 (issue: #61939)
  • Change default format for date_nanos field #70463 (issues: #67063, #69192)
  • Expose if a field is a metadata field in the field capabilities response #69977
  • Field capabilities index action should not fork its execution #69865
  • Improve error message for invalid field name #70972 (issue: #70960)
  • New queryable _tier metadata field #69288 (issue: #68135)
  • Output script stats for indexed fields #71219
  • Preserve half_float precision in fields API #70653 (issue: #70260)
  • Support fetching flattened subfields #70916 (issue: #70605)

Network

Ranking

  • Add access to dense_vector values #71313 (issue: #51964)
  • Make wildcard field use constant scoring queries for wildcard queries #70452 (issue: #69604)

Recovery

  • Fix retention lease expiry to not mark stale #68577

SQL

  • Removed the always on total hit tracking #70319 (issue: #52787)

Search

  • Add _size and _doc_count to fields output #70575 (issue: #63569)
  • Add positive_score_impact to rank_features type #69994 (issue: #68619)
  • Add earlier validation for some SearchSourceBuilder settings #69548 (issue: #54958)
  • Allow format sort values of date fields #70357 (issue: #69192)
  • Cancel searches earlier #69795
  • Close search contexts on reassigned shard #68539
  • Improve lookup for include_unmapped field pattern #69984 (issue: #69983)
  • Support fetching _tier field value #71379 (issues: #63569, #68135)

Security

  • Service Accounts - CLI to delete and list file tokens #71380
  • Warn users if security is implicitly disabled #70114

Snapshot/Restore

  • Add searchable snapshot stats for reads from Lucene #70464
  • Add support for range reads and retries to URL repositories #69521
  • Change tier preference for shared_cache searchable snapshots to frozen only #70786 (issue: #70341)
  • Enforce data_frozen for partial searchable snapshot _tier_preference #71155 (issues: #70786, #71014)
  • Forbid dedicated frozen nodes w/ unfrozen indices #71395
  • Include min/max/average file size in Searchable Snapshots Stats API #70294
  • Make searchable snapshot cache size effectively zero on non-frozen nodes #71134 (issues: #70341, #70846, #71013)
  • Skip TRANSLOG stage for searchable snapshots recovery stage #70311 (issue: #65531)
  • Use default application credentials for GCS repositories #71239

Transform

  • Add support for geo_line aggregation in pivot function #69299
  • Enhance transform role checks #70139 (issue: #69518)
  • Redirect transform actions to transform and remote_cluster_client node when needed #70125
  • Report warnings in _preview response #68396 (issue: #70059)

Bug fixes

Aggregations

  • Significant text aggregation - return empty results rather than error if field unmapped #70778 (issue: #69809)
  • Stop terms aggregation from losing buckets #70493 (issues: #68871, #70449)

Analysis

  • Ukrainian language plugin can fill up heap #71998

Authentication

  • Fix inconsistency of internal user checking #70123

Engine

  • Allow force-merges to run in parallel on a node #69416 (issue: #69327)

Features/Features

  • Unique names for bulk processor scheduler threads #69432 (issues: #1, #68470)

Features/Java High Level REST Client

  • Fix ignoring require_alias parameter in high level rest client #67865 (issue: #67819)

Features/Java Low Level REST Client

  • Fix Suppressing Interrupted Flag in Client #68999 (issue: #68525)

Geo

  • Fix overflow in GeoTileGridTiler #70222

Infra/Logging

Infra/Scripting

Machine Learning

  • Consider xpack.ml.max_ml_node_size in effective_model_memory_limit #70473 (issue: #70069)
  • Do not create machine learning annotations index in upgrade mode #71175
  • Do not track machine learning usage when collecting monitoring #71314
  • Ensure auc_roc curve is monotonic #70628
  • Exclude nested fields in data frame analytics #71400
  • Improve readability of messages written when assigning machine learning jobs to nodes #69629 (issue: #59602)
  • Ensure the same hyperparameters are chosen if classification or regression training is stopped and restarted, for example, if the node fails #1848
  • Fail gracefully if insufficient data is supplied for classification or regression training #1855
  • Fail gracefully on encountering unexpected state in restore from snapshot for anomaly detection #1872
  • Use appropriate memory ordering flags for aarch64 with string store to avoid excessive string duplication #1888
  • Fix autoscaling bug where many jobs take a long time to open #72423
  • Use appropriate master timeouts for master actions #72492
  • Fix empty overall_buckets response #72542
  • Check the out stream exists before consuming it #72455
  • Prevent data frame analytics freeze after loading data #72412

Mapping

  • Fix binary docvalue_fields with padding #70826 (issue: #70244)
  • Propagate index errors in field_caps #70245 (issue: #68994)

Search

  • Correct service time parameter in ARS formula #70283 (issue: #65838)
  • In ARS, correct default number of outstanding requests #71022 (issue: #70283)
  • Prevent aliased fields being used for index sorts #70879

Transforms