Elasticsearch version 7.16.0

Elasticsearch version 7.16.0

Also see Breaking changes in 7.16.

Deprecations

Allocation

  • Improve single-node watermark deprecation checks #80824 (issue: #80817)

Authentication

  • Adding a deprecation info API check for SAML nameid_format setting #77276 (issues: #42404, #44090)

Cluster Coordination

  • Adding a deprecation info check for cluster.join.timeout #76944 (issue: #42404)

Geo

ILM+SLM

  • Add deprecation info and warnings for an empty _tier_preference #79305 (issue: #76147)
  • Adding deprecation logging and deprecation info API for ILM freeze ation #77969 (issues: #42404, #70192)

Indices APIs

Infra/Logging

  • Change default deprecation logger level to CRITICAL #77030 (issue: #76754)
  • Deprecation logs indexing is enabled by default #78991 (issues: #76292, #78850)
  • Exposing the ability to log deprecated settings at non-critical level #79107 (issues: #77482, #78781)
  • REST API to reset deprecation indexing cache #78392 (issue: #78134)
  • Add category field to deprecation logs #77550 (issue: #67443)

Infra/Scripting

  • Script: Deprecate script context cache #79508 (issue: #62899)

Java High Level REST Client

  • Add deprecation headers to HLRC classes #79754

Machine Learning

  • Deprecate estimated_heap_memory_usage_bytes and replace with model_size_bytes #80545

Monitoring

  • Deprecate monitoring settings #79499
  • Deprecate create_legacy_templates setting on HTTP exporters #78564
  • Deprecate the monitoring ingest settings #77659 (issue: #77459)

Known issues

  • Painless: Any Painless script where a method call resolves a lambda argument to a UnaryOperator or a BinaryOperator type results in an IllegalArgumentException. This exception results in a compilation error when static types are used, or a runtime error for dynamic types. Use standard procedural statements as a workaround, such as if/else, while, and for statements.

    The following example is a UnaryOperator type that will result in a failure:

    1. List l = ['abc', '123']; l.replaceAll(f -> f.replace('abc', 'doremi');

    The following example is a possible workaround:

    1. List l = ['abc', '123'];
    2. for (int i = 0; i < l.size(); ++i) {
    3.     l.set(i, l.get(i).replace('abc', 'doremi'));
    4. }

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)

  • The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

New features

Data streams

  • API for adding and removing indices from a data stream #79279 (issue: #75852)

EQL

Machine Learning

  • Text/Log categorization multi-bucket aggregation #71752

SQL

  • Add ability to perform CCS through SQL querying #78903

Search

Enhancements

Aggregations

  • Adds support for the rate aggregation under a composite aggregation #76992 (issue: #76988)
  • Interrupt aggregation reduce phase if the search task is cancelled #71714 (issue: #71021)
  • Add deprecation check for adjacency matrix setting #77802 (issue: #46327)

Allocation

  • Replace RoutingTable#shardsWithState(...) with RoutingNodes#unassigned(...) #78931 (issue: #77466)
  • Reuse local node in async shard fetch responses #77991 (issue: #77266)

Audit

  • Add cluster UUID and name to audit logs #79401 (issue: #72408)

Authentication

  • Remove HTTPS check for API Keys & Service Accounts #76801

Authorization

  • Add privileges to kibana_system for upgrading endpoint hidden indices #80140
  • Allow fleet-server service account to set up Fleet #78192 (issue: #78078)
  • Grant additional privileges for endpoint transform indices to kibana_system #79619
  • Grant privileges required by package upgrade to kibana_system #78049 (issue: #77294)
  • Optimize FLS/DLS setup in IndicePermission authz #77832
  • Skip loading authorized indices if requests do not need them #78321
  • Superuser fastpath for indexAccessControl #78498
  • Update transform destination index privilege for kibana_system #79076
  • Use fixed size memory allocation in IndicesPermission #77748
  • Add proper permissions to fleet server for Endpoint response index #80231
  • Migrate custom role providers to licensed feature #79127

CCR

  • Add description to shard changes action request #80275 (issue: #79311)

Cluster Coordination

  • Add cluster state serialization stats #78816
  • Add timing stats to publication process #76771 (issue: #76625)
  • Get hot threads on lagging nodes #78879
  • Improve error message in 8.x to 7.x downgrade #78644 (issue: #78638)
  • Improve logging in LeaderChecker #78883
  • Mention “warn threshold” in master service slowlog #76815 (issue: #76625)
  • Recycle pages used by outgoing publications #77407 (issue: #77317)
  • Reuse previous indices lookup when possible #79004 (issues: #77888, #78980)
  • Validate PING and STATE connections at join time #77741

Composite aggregations

  • Support _first and _last ordering of missing values in composite aggregations #76740 (issues: #34550, #63523)

Data streams

Distributed

  • Add Fleet search API to wait on refreshes #73134 (issue: #71449)
  • Add support for superseding in CancellableSingleObjectCache #80199

EQL

  • Sequences will now support nano-timestamps #76953 (issue: #68812)

Engine

  • Apply the reader wrapper on can_match source #78988
  • Enable sort optimization in query Lucene changes #77907
  • Honor max segment size when setting only_expunge_deletes on force merge #77478 (issues: #61764, #77270)

Geo

  • Add support for metrics aggregations to mvt end point #78614 (issue: #77072)
  • Include _index property for each hit in _mvt response #77995 (issue: #77205)
  • Add track_total_hits support in mvt API #78074 (issue: #77222)

ILM+SLM

  • Improve LifecycleExecutionState parsing #77855 (issue: #77466)
  • Reduce the number of times that LifecycleExecutionState is parsed when running a policy #77863 (issue: #77466)
  • Add built-in ILM policies for common user use cases #76791
  • Allow for setting the total shards per node in the Allocate ILM action #76134

Indices APIs

  • Fleet: Add action_response into .fleet-actions-results mapping #79584
  • Store template’s mappings as bytes for disk serialization #78746
  • Allow indices lookup to be built lazily #78745 (issue: #77466)

Infra/Core

  • Add optional content checking to ResourceWatcher #79423
  • Add other time accounting in HotThreads #79392
  • Add upgrade preparation and completion callbacks to SystemIndexPlugin #78542
  • Cache index.hidden setting #78612 (issue: #77974)
  • Enable wait/blocked time accounting #77935 (issue: #72376)
  • Handle empty /proc/self/cgroup file #78659 (issue: #77833)
  • Implement framework for migrating system indices #78951
  • Require System Index Descriptors to allow a specific suffix #78355
  • Support mem type in nodes hot_threads API #72850 (issue: #70345)
  • Use enum field for HotThreads report type #77462

Infra/Node Lifecycle

  • Enable exit on out of memory error #71542

Infra/Scripting

  • Add a direct sub classes data structure to the Painless lookup #76955
  • Add ability to augment classes with fields from other classes in Painless #76628
  • Add dynamic (duck) type resolution to Painless static types #78575
  • Adds a lookup method to Painless for finding methods of all sub classes #77044
  • UnsignedLong field type converter #77271
  • Compile/cache eviction history metric placeholders #78257 (issue: #62899)

Infra/Settings

  • Add show command to the keystore CLI #76693 (issue: #57261)
  • Filtering setting deprecation info API messages based on a setting #78725

Ingest

  • Add enrich node cache #76800 (issue: #48988)
  • Add indices pipeline settings check when deleting a pipeline #77013
  • Allow range types to be used for enrich matching #76110
  • ECS support for Grok processor #76885 (issue: #66528)
  • Improving cache lookup to reduce recomputing / searches #77259
  • Make enrich policy execution cancelable #77188 (issue: #48988)
  • Optimistic concurrency control for updating ingest pipelines #78551 (issue: #77031)
  • Sync grok processor patterns with Logstash #76752
  • Updating ingest pipeline without changes is no-op #78196 (issue: #77382)

Java High Level REST Client

  • Add support for rest compatibility headers to the HLRC #78490 (issue: #77859)

License

  • Add license family attribute to feature usage tracking #76622
  • Add utility for tracking licensed persistent tasks #76672
  • Separate feature usage tracking for FLS and DLS #79152
  • Use a licensed feature per realm-type (+custom) #78810

Machine Learning

  • Add new normalize_above parameter to p_value significant terms heuristic #78833
  • Add new default char filter first_line_with_letters for machine learning categorization #77457
  • Add new defer_definition_decompression parameter to put trained model API #77189 (issue: #77132)
  • Enable ML on macOS on ARM #78203
  • Track feature usage for data frame analytics, inference, and anomaly jobs #76789
  • Speed up training of regression and classification models #2024
  • Improve concurrency for training regression and classification models #2031
  • Improve aspects of implementation of skip_model_update rule #2053
  • Make sure instrumentation captures the best hyperparameters found for training classification and regression models https://github.com/elastic/ml-cpp/pull/2057{#2057}

Mapping

  • Better error message for long keys in flattened fields #80433 (issue: #78248)
  • Add time_series_metric parameter #76766 (issue: #74014)
  • Add dimension mapping parameter #74450 (issue: #74014)

Monitoring

  • Remove license check for monitoring data retention #79010

Packaging

  • Switch to Ubuntu docker base image #80640
  • Use Cloudflare’s zlib in Docker images #81245 (issue: #81208)
  • Use almalinux as the Docker base image #80524 (issue: #76681)

Recovery

  • Add support for peer recoveries using snapshots after primary failovers #77420 (issue: #73496)
  • Respect generational files in recoveryDiff #77695 (issues: #55142, #55239)
  • Limit concurrent snapshot file restores in recovery per node #79316 (issue: #79044)

Search

  • Add _ignored and _routing metatada fields to fields API #78981 (issues: #75836, #78828)
  • Add _index and _version metatada fields to fields API #79042 (issues: #75836, #78828)
  • Add ability to retrieve _id via fields option #78828 (issue: #75836)
  • Add node-level field capabilities API requests #79212 (issues: #74648, #77047, #78647)
  • Add segment sorter for data streams #75195
  • Add sort optimization with after from Lucene #64292
  • Don’t always rewrite the Lucene query in search phases #79358
  • Expand segment sorter for all timeseries indices #78639 (issue: #75195)
  • Node level can match action #78765
  • Search - return ignored field values from fields API #78697 (issue: #74121)
  • Support request cache on frozen tier #77694 (issue: #75309)
  • Use search_coordination threadpool for field capabilities API requests #79378 (issue: #79212)
  • Create a sha-256 hash of the shard request cache key #74877 (issue: #74061)

Security

  • Add extensionName() to security extension #79329
  • Optimize StringMatcher for match-all patterns #77738
  • CreateApiKey response now returns the base64-encoded credentials #77351 (issue: #50235)

Snapshot/Restore

  • Add filtering by SLM policy to get snapshots API #77321
  • Add sort by shard count and failed shard count to get snapshots API #77011
  • Add descriptions to various tasks #76700
  • Add maintenance service to clean up unused docs in snapshot blob cache #77686
  • Add periodic maintenance task to clean up unused blob store cache docs #78438 (issue: #77686)
  • Filter Unneeded SnapshotInfo Instances Early in TransportGetSnapshotsAction #78032 (issue: #74350)
  • Implement exclude patterns for snapshot and repository names in get snapshots API #77308
  • Implement sort by repository name in get snapshots API #77049
  • Implement from_sort_value parameter in get snapshots API #77618

Stats

Transform

  • Add _meta field to TransformConfig #79003 (issue: #77506)
  • Add method to collect deprecations from a transform configuration #77565
  • Add transform upgrade endpoint #77566
  • Reduce indexes to query based on checkpoints #75839
  • Implement the ability to preview the existing transform #76697 (issue: #76427)

Bug fixes

Aggregations

  • Add extra round trip to aggregation tests #79638 (issue: #73680)
  • Fix rate aggregation with custom _doc_count #79346 (issue: #77734)
  • Fix several potential circuit breaker leaks in aggregators #79676
  • Scale doubles to floats when necessary to match the field #78344 (issue: #77033)
  • Support for aggregation names with dots in first element path of a pipeline aggregation #77481

Allocation

  • Make disk.threshold_enabled operator only #78822 (issue: #77846)

Authorization

  • Improve permission granting for index referred by multiple names #78902
  • Tighten API key behaviour with DLS and incompatible license #78378

CAT APIs

  • Adjust _cat/templates to not request all metadata #78829

CCR

  • Clear auto-follow errors on deleting pattern #80544 (issue: #77723)

CRUD

  • Use query param instead of a system property for opting in for new cluster health response code #79351 (issues: #70849, #78940)

Cluster Coordination

  • Avoid early release of local forking requests #77641 (issues: #77407, #77634)
  • Check for global blocks after IndexNotFoundException in TransportMasterNodeAction #78128 (issue: #70572)
  • Improve control of outgoing connection lifecycles #77295 (issue: #67873)
  • Only remove active peer on connection failure #79557 (issues: #77295, #79550)
  • Reduce merging in PersistedClusterStateService #79793 (issue: #77466)

CompositeAggs

Data streams

  • Add replicated field to get data stream API response #80988 (issue: #118899)
  • Correct check for write index and increment generation on all data stream backing index operations #79916
  • Fix data stream bug causing it to rollover to a non-existent Index #79759
  • Fix IndexNotFoundException error when handling remove alias action #80312
  • Fix data stream alias validation #81040 (issue: #80972)

Distributed

  • Fix Fleet search API with no checkpoints #79400
  • Modify Fleet search URLs to avoid URL collisions #79776

EQL

  • Add optional fields and limit joining keys on non-null values only #79677

Geo

  • Fix bug filtering collinear points on vertical lines #81155 (issues: #59501, #81076)
  • Spherical mercator transformation should handle properly out of bounds latitudes #81145 (issue: #81128)
  • Vector tiles: Add key bucket value to the aggregation layer #79634 (issue: #79585)

ILM+SLM

  • Prevent duplicate ILM cluster state updates from being created #78390 (issues: #77466, #78246)
  • Run ILM and SLM stopping cluster state updates at IMMEDIATE priority #80207 (issue: #80099)
  • Validate that snapshot repository exists for ILM policies during GenerateSnapshotNameStep #77657 (issue: #72957)

Indices APIs

  • Get templates APIs don’t support lists #78989 (issue: #78829)

Infra/Core

Infra/Logging

  • Disable deprecation log indexing until templates are loaded #80406 (issue: #80265)

Infra/Settings

  • Stricter UpdateSettingsRequest parsing on the REST layer #79228 (issue: #29268)
  • Fix flood stage with system indices #80674

Ingest

  • Addressing assertion failure, downgrading to exception - enrich #79717
  • Fix executing missing enrich policy bug #80728

Java High Level REST Client

  • Force typed keys in the HLRC get async search #78992 (issue: #77608)

Machine Learning

  • Address potential ML feature reset permissions bug #79179
  • Adjust ML memory tracker to reduce logging impact #78482
  • Audit job open failures and stop any corresponding datafeeds #80665 (issues: #48934, #80621)
  • Fix acceptable model snapshot versions in ML deprecation checker #81060 (issues: #79387, #81039, #119745)
  • Fix autoscaling capacity consistency #81181
  • Fix bug in inference stats persister for when feature reset is called #77213 (issue: #77182)
  • Fix datafeed preview with remote indices #81099 (issue: #77109)
  • Fix language identification bug when multi-languages are present #80675
  • Fix model snapshot sorting when sorting by min_version #80596 (issue: #80561)
  • ML legacy index templates that are no longer needed should be deleted #80874 (issue: #80876)
  • Need to tolerate .ml-config being an alias #80025
  • Parent datafeed actions to the datafeed’s persistent task #81143
  • Wait for .ml-state-write alias to be readable #79731 (issue: #79636)
  • Improve the estimates of hyperparameter importance in data frame analytics and avoid incorrectly stopping the hyperparameter search prematurely #2073
  • Fix numerical instability in hyperparameter optimization for training regression and classification models #2078
  • Fix numerical stability issues in time series modeling [#[2083](https://github.com/elastic/ml-cpp/pull/2083)\]

Mapping

  • Disable request cache for non-deterministic runtime fields #75054
  • Fix TextFieldMapper Retaining a Reference to its Builder #77251 (issue: #73845)

Packaging

  • Ensure LIBFFI_TMPDIR is exported by init script #80794

Recovery

  • Do not release snapshot file download permit during recovery retries #79409 (issue: #79316)

SQL

  • Fix NULLS FIRST/LAST for aggregations #77750 (issue: #34550)
  • Fix use of requestTimeout and pageTimeout query parameters #79360 (issue: #72151)
  • Swap JDBC page.timeout and query.timeout properties in query requests #79491 (issue: #79430)

Search

  • Minimize search source of shard level search requests #80634 (issue: #80187)
  • Prevent NullPointerException in SourceConfirmedTextQuery #80472 (issue: #80419)
  • _terms_enum API index_filter doesn’t work with _tier field on upgraded cluster #79553 (issue: #79200)
  • Undeprecate the auto complete thread pool #80204
  • Wildcard field regex query fix #78839 (issue: #78391)

Snapshot/Restore

  • Fix Queued Snapshot Clone not Starting after Data Node Drops Out #77111 (issue: #77101)
  • Fix Temporarily Leaking Shard Level Metadata Blobs in some Cases #76562
  • Fix after restore Lucene.pruneUnreferencedFiles() conditional #81047 (issues: #68821, #75308)
  • Improve handling of corrupt index.latest blob #77339
  • Submit GCS delete batch requests incrementally #80540
  • Fix repository-azure for empty settings on reload #79559

Transform

  • Fix transform feature reset permissions bug #79178
  • Respect timeout parameters in all APIs #79468 (issue: #79268)

Watcher

  • Fix watcher check that determines when to serialize indices options #78070 (issue: #78035)
  • Fix index action simulation when indexing several documents #76820 (issues: #66735, #74148)

Upgrades

Infra/Core

Watcher