Elasticsearch version 7.4.0

Also see Breaking changes in 7.4.

Known issues

  • Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
  • Activating the search slow log should be avoided in this version. Any attempt to log a slow search can throw an AIOOBE due to a bug that performs concurrent modifications on a shared byte array. (issue: #48358)
  • Indices created in 6.x with date and date_nanos fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions.

Breaking changes

Infra/REST API

  • Update the schema for the REST API specification #42346 (issue: #35262)

Machine Learning

  • Improve progress reporting for data frame analytics #45856

Ranking

  • Forbid empty doc values on vector functions #43944

Search

  • Use float instead of double for query vectors. #46004

Snapshot/Restore

  • Provide an Option to Use Path-Style-Access with S3 Repo #41966 (issue: #41816)

Transforms

  • Combine task_state and indexer_state in _stats #45276 (issue: #45201)
  • Improve response format of transform stats endpoint #44350 (issue: #43767)

Breaking Java changes

Geo

  • Geo: Change order of parameter in Geometries to lon, lat #45332 (issue: #45048)

Network

  • Stop Recreating Wrapped Handlers in RestController #44964

Deprecations

Geo

Infra/Core

  • Bundle AdoptOpenJDK 13 #46860
  • Add deprecation check for pidfile setting #45939 (issues: #45938, #45940)
  • Deprecate the pidfile setting #45938
  • Add node.processors setting in favor of processors #45855
  • Deprecate setting processors to more than available #44889

Infra/Settings

Machine Learning

  • Only emit deprecation warning if there was actual change of a datafeed’s job_id. #44755
  • Deprecate the ability to update datafeed’s job_id. #44691 (issue: #44615)

New features

Aggregations

  • Support Range Fields in Histogram and Date Histogram #45395
  • Add Cumulative Cardinality agg (and Data Science plugin) #43661 (issue: #43550)

Analysis

  • Add support for inlined user dictionary in the Kuromoji plugin #45489 (issue: #25343)

Authentication

  • PKI realm authentication delegation #45906 (issue: #34396)
  • PKI Authentication Delegation in new endpoint #43796 (issue: #34396)

Authorization

  • Add granular privileges for API keys #42020

Features/ILM

Features/Watcher

  • Add max_iterations configuration to watcher action with foreach execution #45715 (issues: #41997, #45169)

Geo

  • [SPATIAL] New ShapeQueryBuilder for querying indexed cartesian geometry #45108 (issue: #44980)
  • [GEO] New ShapeFieldMapper for indexing cartesian geometries #44980
  • Add Circle Processor #43851 (issue: #43554)
  • New shape field type for indexing Cartesian Geometries #43644

Machine Learning

  • Allow the user to specify query in Evaluate Data Frame request #45775 (issue: #45729)
  • Call the new _estimate_memory_usage API endpoint on data frame analytics _start #45536 (issues: #44699, #45544)
  • HLRC for memory usage estimation API #45531 (issues: #44699, #45188)
  • Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 (issue: #44699)

Enhancements

Aggregations

  • Add more flexibility to MovingFunction window alignment #44360 (issue: #42181)
  • Optimize Min and Max BKD optimizations #44315 (issue: #44290)
  • Allow pipeline aggs to select specific buckets from multi-bucket aggs #44179

Allocation

  • Defer reroute when starting shards #44433 (issues: #42105, #42738)
  • Allow RerouteService to reroute at lower priority #44338
  • Auto-release of read-only-allow-delete block when disk utilization fa… #42559 (issue: #39334)

Analysis

  • Allow all token/char filters in normalizers #43803 (issue: #43758)

Authentication

  • Allow Transport Actions to indicate authN realm #45767 (issue: #45331)
  • Explicitly fail if a realm only exists in keystore #44471 (issue: #44207)

Authorization

  • Add manage_own_api_key cluster privilege #45897 (issue: #40031)
  • Consider owner flag when retrieving/invalidating keys with API key service #45421 (issue: #40031)
  • REST API changes for manage-own-api-key privilege #44936 (issue: #40031)
  • Simplify API key service API #44935 (issue: #40031)

CCR

  • Include leases in error message when operations no longer available #45681

CRUD

  • Return seq_no and primary_term for noop update #44603 (issue: #42497)

Cluster Coordination

  • Improve slow logging in MasterService #45086 (issue: #45007)
  • More logging for slow cluster state application #45007
  • Ignore unknown fields if overriding node metadata #44689
  • Allow pending tasks before state recovery #44685 (issue: #44652)

Distributed

  • Do not create engine under IndexShard#mutex #45263 (issue: #43699)

Docs Infrastructure

  • add clarification around TESTSETUP docu and error message #43306

Engine

  • Flush engine after big merge #46066
  • Do sync before closeIntoReader when rolling generation to improve index performance #45765 (issue: #45371)
  • Refactor index engines to manage readers instead of searchers #43860
  • Async IO Processor release before notify #43682
  • Enable indexing optimization using sequence numbers on replicas #43616 (issue: #34099)

Features/ILM

  • Add node setting for disabling SLM #46794 (issue: #38461)
  • Include in-progress snapshot for a policy with get SLM policy API #45245
  • Add option to filter ILM explain response #44777 (issue: #44189)
  • Expose index age in ILM explain output #44457 (issue: #38988)

Features/Indices APIs

Features/Ingest

  • Fix IngestService to respect original document content type #45799
  • Ingest Attachment: Upgrade tika to v1.22 #45575
  • Retrieve processors instead of checking existence #45354
  • Add ingest processor existence helper method #45156
  • Change the ingest simulate api to not include dropped documents #44161 (issue: #36150)

Features/Java High Level REST Client

  • Add XContentType as parameter to HLRC ART#createServerTestInstance #46036 (issue: #45970)
  • Add CloseIndexResponse to HLRC #44349 (issue: #39687)
  • Add mapper-extras and the RankFeatureQuery in the hlrc #43713 (issue: #43634)
  • removing background state update of Request object by RequestConverte… #40156 (issue: #39666)
  • Add delete aliases API to the high-level REST client #32909 (issue: #27205)

Features/Watcher

  • Add SSL/TLS settings for watcher email #45272 (issue: #30307)
  • Watcher reporting: add email warning if CSV attachment contains values that may be interperted as formulas #44460
  • Watcher add stopped listener #43939 (issue: #42409)
  • Improve CryptoService error message on missing secure file #43623 (issue: #43619)

Geo

  • Support WKT point conversion to geo_point type #44107 (issue: #41821)

Infra/Circuit Breakers

  • Fix G1 GC default IHOP #46169

Infra/Core

Infra/Packaging

  • Remove redundant Java check from Sys V init #45793 (issue: #45593)
  • Notify systemd when Elasticsearch is ready #44673

Infra/Plugins

  • Make plugin verification FIPS 140 compliant #44224 (issue: #41263)

Infra/Scripting

  • Whitelist randomUUID in Painless #45148 (issue: #39080)
  • Add missing ZonedDateTime methods for joda compat layer #44829 (issue: #44411)
  • Remove stale permissions from untrusted policy #44783

Infra/Settings

  • Add more meaningful keystore version mismatch errors #46291 (issue: #44624)
  • Lift the restrictions that uppercase is not allowed in Setting Name. #45222 (issue: #43835)
  • Normalize environment paths #45179 (issue: #45176)

Machine Learning

  • Support boolean fields for data frame analytics #46037
  • Add description to data frame analytics #45774
  • Add regression analysis to data frame analytics #45292
  • Introduce formal node ML role #45174 (issues: #29943, #43175)
  • Improve CSV header row detection in find_file_structure #45099 (issue: #45047)
  • Outlier detection should only fetch docs that have the analyzed fields #44944
  • Persist DatafeedTimingStats with RefreshPolicy.NONE by default #44940 (issue: #44792)
  • Add result_type field to TimingStats and DatafeedTimingStats documents #44812
  • Implement exponential average search time per hour statistics. #44683 (issue: #29857)
  • Add r_squared eval metric to regression #44248
  • Adds support for regression.mean_squared_error to eval API #44140
  • Add DatafeedTimingStats.average_search_time_per_bucket_ms and TimingStats.total_bucket_processing_time_ms stats #44125 (issue: #29857)
  • Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response #43045 (issue: #29857)

Network

  • Better logging for TLS message on non-secure transport channel #45835 (issue: #32688)
  • Asynchronously connect to remote clusters #44825 (issue: #40150)
  • Improve errors when TLS files cannot be read #44787 (issue: #43079)
  • Add per-socket keepalive options #44055
  • Move ConnectionManager to async APIs #42636

Ranking

  • Search enhancement: pinned queries #44345 (issue: #44074)
  • Fix parameter value for calling data.advanceExact #44205
  • Add l1norm and l2norm distances for vectors #44116 (issue: #37947)

Recovery

SQL

  • SQL: Support queries with HAVING over SELECT #46709 (issue: #37051)
  • SQL: Break TextFormatter/Cursor dependency #45613 (issue: #45516)
  • SQL: remove deprecated use of “interval” from date_histogram usage #45501 (issue: #43922)
  • SQL: use hasValue() methods from Elasticsearch’s InspectionHelper classes #44745 (issue: #35745)
  • Switch from using docvalue_fields to extracting values from _source #44062 (issue: #41852)

Search

  • Adds usage stats for vectors: #44512
  • Associate sub-requests to their parent task in multi search API #44492
  • Cancel search task on connection close #43332

Security

  • Set security index refresh interval to 1s #45434 (issue: #44934)

Snapshot/Restore

  • add disable_chunked_encoding configuration #44052
  • Repository Cleanup Endpoint #43900

Task Management

  • Remove task null check in TransportAction #45014
  • TaskListener#onFailure to accept Exception instead of Throwable #44946
  • Move child task cancellation to TaskManager #44573 (issue: #44494)

Transforms

  • Add update transform api endpoint #45154 (issue: #43438)
  • Add support for bucket_selector #44718 (issues: #43744, #44557)
  • Add force delete #44590 (issue: #43961)
  • Add dynamic cluster setting for failure retries #44577
  • Add optional defer_validation param to PUT #44455 (issue: #43439)
  • Add support for geo_bounds aggregation #44441
  • Add a frequency option to transform config, default 1m #44120

Bug fixes

Aggregations

  • Fix early termination of aggregators that run with breadth-first mode #44963 (issue: #44909)
  • Support BucketScript paths of type string and array. #44694 (issue: #44385)

Allocation

  • Avoid overshooting watermarks during relocation #46079 (issue: #45177)
  • Cluster health should await events plus other things #44348
  • Do not copy initial recovery filter during split #44053 (issue: #43955)

Analysis

  • Enable reloading of synonym_graph filters #45135 (issue: #45127)
  • Fix AnalyzeAction response serialization #44284 (issue: #44078)

Authentication

  • Fallback to realm authc if ApiKey fails #46538
  • Enforce realm name uniqueness #46253
  • Allow empty token endpoint for implicit flow #45038
  • Do not use scroll when finding duplicate API key #45026
  • Fix broken short-circuit in getUnlicensedRealms #44399
  • Fix X509AuthenticationToken principal #43932 (issues: #34396, #43796)

Authorization

  • Do not rewrite aliases on remove-index from aliases requests #46989
  • Give kibana user privileges to create APM agent config index #46765 (issue: #45610)
  • Add manage_own_api_key cluster privilege #45696 (issue: #40031)
  • Sparse role queries can throw an NPE #45053

CCR

  • Clean up ShardFollowTasks for deleted indices #44702 (issue: #34404)
  • Skip update if leader and follower settings identical #44535 (issue: #44521)
  • Avoid stack overflow in auto-follow coordinator #44421 (issue: #43251)
  • Avoid NPE when checking for CCR index privileges #44397 (issue: #44172)

CRUD

Cluster Coordination

  • Assert no exceptions during state application #47090 (issue: #47038)
  • Avoid counting votes from master-ineligible nodes #43688

Distributed

  • Fix false positive out of sync warning in synced-flush #46576 (issues: #28464, #30244)
  • Suppress warning logs from background sync on relocated primary #46247 (issues: #40800, #42241)
  • Ensure AsyncTask#isScheduled remain false after close #45687 (issue: #45576)
  • Update translog checkpoint after marking operations as persisted #45634 (issue: #29161)
  • Fix clock used in update requests #45262 (issue: #45254)
  • Restore DefaultShardOperationFailedException’s reason during deserialization #45203
  • Use IndicesModule named writables in elasticsearch-shard tool #45036 (issue: #44628)

Engine

  • Handle delete document level failures #46100 (issue: #46083)
  • Handle no-op document level failures #46083
  • Remove leniency during replay translog in peer recovery #44989
  • Throw TranslogCorruptedException in more cases #44217
  • Fail engine if hit document failure on replicas #43523 (issues: #40435, #43228)

Features/ILM

  • Handle partial failure retrieving segments in SegmentCountStep #46556
  • Fixes for API specification #46522
  • Fix SnapshotLifecycleMetadata xcontent serialization #46500 (issue: #46499)
  • Updated slm API spec parameters and URL #44797
  • Fix swapped variables in error message #44300

Features/Indices APIs

Features/Ingest

  • Allow dropping documents with auto-generated ID #46773 (issue: #46678)

Features/Java High Level REST Client

  • [HLRC] Send min_score as query string parameter to the count API #46829 (issue: #46474)
  • HLRC multisearchTemplate forgot params #46492 (issue: #46488)
  • terminateAfter added to the RequestConverter #46474 (issue: #46446)
  • [Closes #44045] Added slices parameter when submitting reindex request via Java high level REST client #45690 (issue: #44045)
  • HLRC: Fix + Not Correctly Encoded in GET Req. #33164 (issue: #33077)

Features/Watcher

  • Fix class used to initialize logger in Watcher #46467
  • Fix wrong URL encoding in watcher HTTP client #45894 (issue: #44970)
  • Fix watcher HttpClient URL creation #45207 (issue: #44970)
  • Log write failures for watcher history document. #44129

Geo

  • Geo: fix geo query decomposition #44924 (issue: #44891)
  • Geo: add validator that only checks altitude #43893

Highlighting

  • Fix highlighting for script_score query #46507 (issue: #46471)

Infra/Core

  • Always check that cgroup data is present #45606 (issue: #45396)
  • Safe publication of DelayedAllocationService and SnapshotShardsService #45517 (issue: #38560)
  • Add default CLI JVM options #44545 (issues: #219, #42021)
  • Fix decimal point parsing for date_optional_time #43859 (issue: #43730)

Infra/Logging

  • Fix types field in JSON Search Slow Logs #44641
  • Add types field to JSON slow logs in 7.x #44592 (issues: #41354, #44178)

Infra/Packaging

  • Add destructiveDistroTest meta task #45762
  • Use bundled JDK in Sys V init #45593 (issue: #45542)
  • Restore setting up temp dir for windows service #44541

Infra/Plugins

  • Do not checksum all bytes at once in plugin install #44649 (issue: #44545)

Infra/REST API

  • Improve error message when index settings are not a map #45588 (issue: #45126)
  • Add is_write_index column to cat.aliases #44772
  • Fix URL documentation in API specs #44487

Infra/Scripting

  • Fix bugs in Painless SCatch node #45880
  • Fix JodaCompatibleZonedDateTime casts in Painless #44874

Infra/Settings

  • bug fix about elasticsearch.common.settings.Settings.processSetting #44047 (issue: #43791)

Machine Learning

  • Fix two datafeed flush lockup bugs #46982
  • Throw an error when a datafeed needs CCS but it is not enabled for the node #46044 (issue: #46025)
  • Handle “null” value of Estimate memory usage API response gracefully. #45726 (issue: #44699)
  • Remove timeout on waiting for data frame analytics result processor to complete #45724 (issue: #45723)
  • Check dest index is empty when starting data frame analytics #45094
  • Catch any error thrown while closing data frame analytics process #44958
  • Treat PostDataActionResponse.DataCounts.bucketCount as incremental rather than absolute (total). #44803 (issue: #44792)
  • Treat big changes in searchCount as significant and persist the document after such changes #44413 (issues: #44196, #44335)
  • Update .ml-config mappings before indexing job, datafeed or data frame analytics config #44216 (issue: #44263)
  • Wait for .ml-config primary before assigning persistent tasks #44170 (issue: #44156)
  • Fix ML memory tracker lockup when inner step fails #44158 (issue: #44156)
  • Fix datafeed checks when a concrete remote index is present #43923 (issue: #42113)
  • Rename outlier detection method values knn and tnn to distance_kth_nn and distance_knn respectively to match the API. #598
  • Fix occasional (non-deterministic) reinitialisation of modeling for the lat_long function. #641

Mapping

  • Make sure to validate the type before attempting to merge a new mapping. #45157 (issues: #29316, #43012)
  • Ensure field caps doesn’t error on rank feature fields. #44370 (issue: #44330)
  • Prevent types deprecation warning for indices.exists requests #43963 (issue: #43905)
  • Add include_type_name in indices.exitst REST API spec #43910 (issue: #43905)

Network

Recovery

  • Never release store using CancellableThreads #45409 (issues: #45136, #45237)
  • Remove leniency in reset engine from translog #44711

Rollup

  • Fix Rollup job creation to work with templates #43943

SQL

  • SQL: Properly handle indices with no/empty mapping #46775 (issue: #46757)
  • SQL: improve ResultSet behavior when no rows are available #46753 (issue: #46750)
  • SQL: use the correct data type for types conversion #46574 (issue: #46090)
  • SQL: Fix issue with common type resolution #46565 (issue: #46551)
  • SQL: fix scripting for grouped by datetime functions #46421 (issue: #40241)
  • SQL: Use null schema response #46386 (issue: #46381)
  • SQL: Fix issue with IIF function when condition folds #46290 (issue: #46268)
  • SQL: Fix issue with DataType for CASE with NULL #46173 (issue: #46032)
  • SQL: adds format parameter to range queries for constant date comparisons #45326 (issue: #45139)
  • SQL: uniquely named inner_hits sections for each nested field condition #45039 (issues: #33080, #44544)
  • SQL: fix URI path being lost in case of hosted ES scenario #44776 (issue: #44721)
  • SQL: change the size of the list of concrete indices when resolving multiple indices #43878 (issue: #43876)
  • SQL: handle double quotes escaping #43829 (issue: #43810)
  • SQL: add pretty printing to JSON format #43756
  • SQL: handle SQL not being available in a more graceful way #43665 (issue: #41279)

Search

  • Multi-get requests should wait for search active #46283 (issue: #27500)
  • Ensure top docs optimization is fully disabled for queries with unbounded max scores. #46105 (issue: #45933)
  • Disallow partial results when shard unavailable #45739 (issue: #42612)
  • Prevent Leaking Search Tasks on Exceptions in FetchSearchPhase and DfsQueryPhase #45500
  • Fix an NPE when requesting inner hits and _source is disabled. #44836 (issue: #43517)
  • Don’t use index_phrases on graph queries #44340 (issue: #43976)

Security

  • Initialize document subset bit set cache used for DLS #46211 (issue: #45147)
  • Fix .security-* indices auto-create #44918
  • SecurityIndexManager handle RuntimeException while reading mapping #44409
  • Do not swallow I/O exception getting authentication #44398 (issues: #44172, #44397)
  • Use system context for looking up connected nodes #43991 (issue: #43974)

Snapshot/Restore

  • Fix Bug in Snapshot Status Response Timestamps #46919 (issue: #46913)
  • GCS deleteBlobsIgnoringIfNotExists should catch StorageException #46832 (issue: #46772)
  • Fix TransportSnapshotsStatusAction ThreadPool Use #45824
  • Stop Executing SLM Policy Transport Action on Snapshot Pool #45727 (issue: #45594)
  • Check again on-going snapshots/restores of indices before closing #43873
  • Make Timestamps Returned by Snapshot APIs Consistent #43148 (issue: #43074)
  • Recursively Delete Unreferenced Index Directories #42189 (issue: #13159)

Task Management

  • Catch AllocatedTask registration failures #45300

Transforms

  • Use field_caps API for mapping deduction #46703 (issue: #46694)
  • Fix off-by-one error in checkpoint operations_behind #46235
  • Moves failure state transition for MT safety #45676 (issue: #45664)
  • Fix _start?force=true bug #45660
  • Fix failure state transitions and race condition #45627 (issues: #45562, #45609)
  • Fix starting a batch data frame after stopping at runtime #45340 (issues: #44219, #45339)
  • Fix null aggregation handling in indexer #45061 (issue: #44906)
  • Unify validation exceptions between PUT and _preview #44983 (issue: #44953)
  • Treat bulk index failures as an indexing failure #44351 (issue: #44101)
  • Prevent task from attempting to run when failed #44239 (issue: #44121)
  • Respond with 409 status code when failing _stop #44231 (issue: #44103)
  • Add index validations to _start data frame transform #44191 (issue: #44104)
  • Data frame task failure does not make a 500 response #44058 (issue: #44011)
  • Audit message missing for autostop #43984 (issue: #43977)

Regressions

Aggregations

  • Implement rounding optimization for fixed offset timezones #46670 (issue: #45702)

Upgrades

Infra/Core

Infra/Packaging

Network

  • Upgrade to Netty 4.1.38 #45132

Search

  • Upgrade to lucene snapshot 8.3.0-snapshot-8dd116a6158 #45604 (issue: #43976)