Elasticsearch version 7.12.0
- Elasticsearch version 7.12.0

Elasticsearch version 7.12.0

Also see Breaking changes in 7.12.

Security updates

A document disclosure flaw was found in the Elasticsearch suggester and profile API when Document and Field Level Security are enabled. The suggester and profile API are normally disabled for an index when document level security is enabled on the index. Certain queries are able to enable the profiler and suggester which could lead to disclosing the existence of documents and fields the attacker should not be able to view. All versions of Elasticsearch before 6.8.15 and 7.11.2 are affected by this flaw. You must upgrade to 6.8.15 or 7.11.2 to obtain the fix. CVE-2021-22137

Known issues

Frozen tier: (Windows only) The frozen data tier relies on multiple caching mechanisms to speed up access to searchable snapshot files. One of these caches uses sparse files to avoid creating large files on disk when it is not strictly required. A bug prevented files from being created with the right options to enable sparse support on Windows, leading Elasticsearch to create potentially large files that can end up consuming all the disk space.

This issue is fixed in Elasticsearch versions 7.15.2 and later. There is no known workaround for earlier versions. Filesystems that enable sparse files by default are not affected. For more details, see #79371.
If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.
If autoscaling is enabled and job assignment takes an inordinate amount of time, scaling sizes may increase much higher than required. Elasticsearch 7.13.0 contains a fix for this. For more details, see #72509
Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:
```
PUT _cluster/settings
{
  "persistent" : {
    "snapshot.max_concurrent_operations" : 1
  }
}
```
This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #73456.
The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

Breaking changes

Mapping

Returning nested data in fields API #67432 (issue: #63709)

Query Languages

Fields API implementation in QL #68802 (issue: #67727)

SQL

Test and fix the NULL handling of the string functions #68379 (issue: #58907)

Breaking Java changes

Features/Java High Level REST Client

Move find file structure finder in REST High Level client to its new endpoint and plugin #67290

Deprecations

Distributed

Warn of change of default of wait_for_active_shards #67527 (issues: #66419, #67158, #67246, #67498)

Infra/Core

Add deprecation warning for removed strict_duplicate_detection setting #63500 (issues: #34588, #60222)

Geo

Deprecate GeoPolygon query in favour of GeoShape query #64227 (issue: #48928)

Machine Learning

Add deprecation check for job model snapshots that need upgrade #66062 (issue: #64154)

Security

Deprecate the id field for the InvalidateApiKey API #66317 (issue: #63224)

New features

Authorization

Add a cluster privilege to cancel tasks and delete async searches #68679 (issue: #67965)

EQL

Introduce ~ grammar for case-insensitive functions #67869 (issue: #67868)
Introduce like and regex keywords #68791 (issue: #68639)

Features/ILM+SLM

Add support for partial searchable snapshots to ILM #68714 (issues: #68509, #68605)
Add the frozen tier node role and ILM phase #68605 (issue: #60848)
Support max_single_primary_size in resize action and expose in ILM #67705 (issue: #65714)

Features/Ingest

Fingerprint ingest processor #68415 (issue: #53578)
Network community_id processor for ingest pipelines #66534 (issue: #55685)
Network direction processor #66644

Query Languages

Implement support for date_nanos Elasticsearch field type #67666 (issue: #38562)

SQL

Implement TO_CHAR() function #66486 (issue: #54964)

Add minimum compatibility version to SearchRequest #65896 (issue: #63304)
Sort field tiebreaker for point in time (PIT) readers #66093 (issue: #56828)

Snapshot/Restore

Add partial searchable snapshot support for a frozen tier #68509

Enhancements

Aggregations

Add multi_terms aggs #67597 (issue: #65623)
Lower contention on requests with many aggs #66895 (issue: #58647)

Allocation

Drop gateway.recover_after_nodes from default conf #68000 (issue: #53646)
Make InternalClusterInfoService async #66993

Authentication

Add grant-api-key to HLRC #68190

Authorization

Add more context to cluster access denied messages #66900 (issues: #42166, #60357)
Add more context to index access denied errors #60357 (issue: #42166)
Allow Kibana to create Fleet Server indices #68152
Phase 2 support for operator privileges: Cluster settings #66684
view_index_metadata and manage privs now grant access to field capabilities #67392 (issue: #66867)

Cluster Coordination

Clarify message emitted on cluster UUID mismatch #66915
Extend default probe connect/handshake timeouts #68059 (issue: #42636)
Skip cluster state serialization to closed channel #67413

EQL

Add result_position to correctness test #66761
Introduce EQL search status API #68065 (issue: #66955)

Engine

Assign id to searcher using ids of segments #66668 (issue: #63963)
Use merging fieldsreader when restoring versionmap during recovery #66944

Features/ILM+SLM

Make the rest of the forcemerge action steps retryable #66352 (issue: #48183)
Make the unfollow action and CCR related steps retryable #66356 (issue: #48183)
Skip unfollow action if the index is not a follower #68690

Features/Indices APIs

Add max_single_primary_size as a condition for the rollover index API #67842 (issue: #63026)
Make template conflict warning deterministic #67066 (issue: #66820)

Features/Ingest

Configurable MIME type for mustache template encoding on set processor #65314 (issue: #65115)

Features/Java High Level REST Client

Added support for passing http query parameters #67802 (issue: #53846)

Features/Java Low Level REST Client

Add client metadata header on RestClient requests #66303 (issue: #66189)

Features/Stats

Add processor architectures to cluster stats #68264
Make GET _cluster/stats cancellable #68676 (issue: #55550)

Geo

Add support for Spatial Relationships to geo_point field #67631 (issue: #52382)
Implement IndexOrDocValuesQuery for geo_shape field #64688

Highlighting

Add query param to limit highlighting to specified length #67325 (issue: #52155)

Infra/Core

Clarify bootstrap check failure messages #67501
Introduce “Feature States” for managing snapshots of system indices #63513 (issue: #61657)

Infra/Logging

Align JSON logs better with ECS #67266
Introduce deprecation categories #67443 (issues: #64824, #67266)

Infra/Plugins

Notify users to restart Elasticsearch after plugin installation #66723 (issue: #39487)

Infra/Scripting

Add OSS whitelist to execute API #67038 (issue: #67035)
Capture structured javadoc from stdlib #68782
Enforce GPLv2 for parsed stdlib docs #68601
Improve bad regex pattern syntax error #68520
Parse stdlib files for parameter names #67837
Whitelist JSON functions for ingest #67118

Infra/Settings

Add enumsetting to org.elasticsearch.common.settings.Setting #66826 (issue: #65335)
Support removing archived settings if cluster has read only blocks #64113

License

Remove grace period from license expiration check #67316

Machine Learning

Add new multi custom processor for data frame analytics and model inference #67362
Allow data frame analytics memory estimation on non-ML nodes #68146 (issue: #67840)
Automatic management for machine learning system indices #68044
Expand regression and classification hyperparameters #67950
Improve resuming a data frame analytics job stopped during inference #67623
Move find file structure to a new API endpoint #67123 (issue: #67001)
Preserve inference progress when starting data frame analytics #68222
Write notification messages indicating required capacity when a job is not assigned to a node #67181
Fix edge case which could cause spurious anomalies early in the learning process if the time series has non-diurnal seasonality #1634
Compute importance of hyperparameters optimized in the fine parameter tuning step #1627
Early stopping for the fine parameter tuning step of classification and regression model training #1676
Correct upgrade for pre-6.3 state for lat_long anomaly anomaly detectors #1681
Per tree feature bag to speed up training of regression and classification models and improve scalability for large numbers of features #1733

Mapping

Shortcut to avoid fmod #66909

Packaging

Tweaks to the Iron Bank docker context #66942

Performance

Add benchmark racing scripts #68369

Query Languages

Adapt nested fields extraction from fields API output to the new un-flattened structure #68745 (issue: #68722)
Simplify arithmetic operations in binary comps #66022 (issue: #65394)

Ranking

Add linear function to rank_feature query #67438 (issue: #49859)

Rollup

Remove the rollup thread pool #65958

SQL

Enhance error message on filtering check against aggs #68763 (issue: #57125)

Add grok and dissect methods to runtime fields #68088 (issue: #67825)
Allow deletion of async searches with the manage privilege #67965
Avoid duplicate serialization for TermsQueryBuilder #67223 (issue: #67132)
Early detection of circuit breaker exception in the coordinating node #67431 (issue: #62884)
Limit the depth of nested bool queries #66204 (issue: #55303)
Remove unnecessary optimizations for TermsSetQueryBuilder #67637 (issue: #67223)
Retry point in time on other copy when possible #66713 (issue: #61062)
Slightly speed up scripts #68478
Use a mapping snapshot for fetching nested docs #66877 (issue: #66295)

Security

Build complex automatons more efficiently #66724 (issue: #36062)

Snapshot/Restore

Add ClusterUUID to RepositoryData #68002
Allow searchable snapshots to work on source-only repositories #67975 (issue: #67936)
Avoid async cache-size fetch on partial shards #68644
Forbid removing write block from searchable snapshot index #68036
Ignore disk watermarks on partial shards #68673
Improve error message on incompatible repo format #67776
Introduce repository UUIDs #67829 (issue: #66431)
Introduce repository test kit/analyser #67247
Permit rename of repository of searchable snapshot #67968 (issue: #66431)
Reject remounting snapshot of a searchable snapshot #68816 (issue: #68792)

Task Management

Remove parent-task bans on channels disconnect #66066 (issues: #56620, #65443)

Transform

Add support for search-time runtime fields #67643 (issue: #65147)
Automatic management for transform system indices #68590
Implement retention policy to delete data from a transform #67832 (issue: #67916)
Refactor cat transform to show more useful information #68232 (issues: #66367, #66718)
Report last search time in transform stats #66718 (issues: #66367, #66410, #66718)

Bug fixes

Aggregations

Remove toString->parse roundtrip for missing value in terms agg #67954 (issue: #67197)

Engine

Fail force-merges on read-only engines #64756

Features/Features

Prevent unnecessary system index access warnings in Deprecation Info API #67312 (issue: #66063)

Features/ILM+SLM

Avoid exception in in SetStepInfoUpdateTask#onFailure #67553
Fix issues with CS Handling in ILM async steps #68361

Features/Java Low Level REST Client

Fix wrong totalIterationCount when running BulkIndexBenchmark #64515 (issue: #61451)

Features/Monitoring

Move monitoring collection timeouts to coordinator #67084 (issues: #60188, #66993)

Features/Ingest

Fix ingest java week based year defaulting #65717

Geo

Fix indexing of degenerated rectangles #67702 (issue: #67695)

Infra/Core

Allow the *,-* (“no-index”) pattern for destructive actions when destructive_requires_name is true #68021 (issue: #67958)

Infra/Scripting

Augmentation.join can’t handle empty strings at the start #68251 (issue: #33434)

Machine Learning

Make find_structure validate lines_to_sample at least 2 #68199 (issue: #68128)
Return 400 status for invalid charset in find_structure #68201 (issue: #68130)
Return status 400 for grok_pattern errors in find_structure #68205 (issue: #68132)
Fix a source of instability in time series modeling for anomaly detection. This has been observed to cause spurious anomalies for a partition which no longer receives any data #1675
Ensure that we stop modeling seasonality for data which flatlines. This is important for count and sum detectors which treat empty buckets as zero. Before this change, we could sometimes detect spurious anomalies in realtime detection after a partition no longer received any data #1654

SQL

Fix ZonedDateTime with nanos serialisation #68253 (issue: #67666)

Handle ignored fields directly in SourceValueFetcher #68738

Snapshot/Restore

Make RepositoryData Parsing Stricter #67699 (issue: #67696)
« Elasticsearch version 7.12.1
Elasticsearch version 7.11.2 »