Elasticsearch version 7.8.0

Also see Breaking changes in 7.8.

Breaking changes

Aggregations

• value_count aggregation optimization #54854

Features/Indices APIs

• Add auto create action #55858

Mapping

• Disallow changing enabled on the root mapper #54463 (issue: #33933)
• Fix updating include_in_parent/include_in_root of nested field #54386 (issue: #53792)

Deprecations

Authentication

• Deprecate the kibana reserved user; introduce kibana_system user #54967

Cluster Coordination

• Voting config exclusions should work with absent nodes #50836 (issue: #47990)

Features/Features

• Add node local storage deprecation check #54383 (issue: #54374)

Features/Indices APIs

• Deprecate local parameter for get field mapping request #55014

Infra/Core

• Deprecate node local storage setting #54374

Infra/Plugins

• Add xpack setting deprecations to deprecation API #56290 (issue: #54745)
• Deprecate disabling basic-license features #54816 (issue: #54745)
• Deprecated xpack “enable” settings should be no-ops #55416 (issues: #54745, #54816)
• Make xpack.ilm.enabled setting a no-op #55592 (issues: #54745, #54816, #55416)
• Make xpack.monitoring.enabled setting a no-op #55617 (issues: #54745, #54816, #55416, #55461, #55592)
• Restore xpack.ilm.enabled and xpack.slm.enabled settings #57383 (issues: #54745, #55416, #55592)

New features

Aggregations

• Add Student’s t-test aggregation support #54469 (issue: #53692)
• Add support for filters to t-test aggregation #54980 (issue: #53692)
• Histogram field type support for Sum aggregation #55681 (issue: #53285)
• Histogram field type support for ValueCount and Avg aggregations #55933 (issue: #53285)

Features/Indices APIs

• Add simulate template composition API _index_template/_simulate_index/{name} #55686 (issue: #53101)

Geo

• Add geo_bounds aggregation support for geo_shape #55328
• Add geo_shape support for geotile_grid and geohash_grid #55966
• Add geo_shape support for the geo_centroid aggregation #55602
• Add new point field #53804

SQL

• Implement DATETIME_FORMAT function for date/time formatting #54832 (issue: #53714)
• Implement DATETIME_PARSE function for parsing strings #54960 (issue: #53714)
• Implement scripting inside aggs #55241 (issues: #29980, #36865, #37271)

Enhancements

Aggregations

• Aggs must specify a field or script (or both) #52226
• Expose aggregation usage in Feature Usage API #55732 (issue: #53746)
• Reduce memory for big aggregations run against many shards #54758
• Save memory in on aggs in async search #55683

Allocation

• Disk decider respect watermarks for single data node #55805
• Improve same-shard allocation explanations #56010

Analysis

• Add preserve_original setting in ngram token filter #55432
• Add preserve_original setting in edge ngram token filter #55766 (issue: #55767)
• Add pre-configured “lowercase” normalizer #53882 (issue: #53872)

Audit

• Update the audit logfile list of system users #55578 (issue: #37924)

Authentication

• Let realms gracefully terminate the authN chain #55623

Authorization

• Add reserved_ml_user and reserved_ml_admin kibana privileges #54713

Autoscaling

• Rollover: refactor out cluster state update #53965

CRUD

• Avoid holding onto bulk items until all completed #54407

Cluster Coordination

• Add voting config exclusion add and clear API spec and integration test cases #55760 (issue: #48131)

Features/CAT APIs

• Add support for V2 index templates to /_cat/templates #55829 (issue: #53101)

Features/Indices APIs

• Add HLRC support for simulate index template api #55936 (issue: #53101)
• Add prefer_v2_templates flag and index setting #55411 (issue: #53101)
• Add warnings/errors when V2 templates would match same indices as V1 #54367 (issue: #53101)
• Disallow merging existing mapping field definitions in templates #57701 (issues: #55607, #55982, #57393)
• Emit deprecation warning if multiple v1 templates match with a new index #55558 (issue: #53101)
• Guard adding the index.prefer_v2_templates settings for pre-7.8 nodes #55546 (issues: #53101, #55411, #55539)
• Handle merging dotted object names when merging V2 template mappings #55982 (issue: #53101)
• Throw exception on duplicate mappings metadata fields when merging templates #57835 (issue: #57701)
• Update template v2 api rest spec #55948 (issue: #53101)
• Use V2 index templates during index creation #54669 (issue: #53101)
• Use V2 templates when reading duplicate aliases and ingest pipelines #54902 (issue: #53101)
• Validate V2 templates more strictly #56170 (issues: #43737, #46045, #53101, #53970)

Features/Java High Level REST Client

• Enable support for decompression of compressed response within RestHighLevelClient #53533

Features/Stats

• Fix available / total disk cluster stats #32480 (issue: #32478)

Features/Watcher

• Delay warning about missing x-pack #54265 (issue: #40898)

Geo

• Add geo_shape mapper supporting doc-values in Spatial Plugin #55037 (issue: #53562)

Infra/Core

• Decouple Environment from DiscoveryNode #54373
• Ensure that the output of node roles are sorted #54376 (issue: #54370)
• Reintroduce system index APIs for Kibana #54858 (issues: #52385, #53912)
• Schedule commands in current thread context #54187 (issue: #17143)
• Start resource watcher service early #54993 (issue: #54867)

Infra/Packaging

• Make Windows JAVA_HOME handling consistent with Linux #55261 (issue: #55134)

Infra/REST API

• Add validation to the usage service #54617

Infra/Scripting

• Scripting: stats per context in nodes stats #54008 (issue: #50152)

Machine Learning

• Add effective max model memory limit to ML info #55529 (issue: #63942)
• Add loss_function to regression #56118
• Add new inference_config field to trained model config #54421
• Add failed_category_count to model_size_stats #55716 (issue: #1130)
• Add prediction_field_type to inference config #55128
• Allow a certain number of ill-formatted rows when delimited format is specified #55735 (issue: #38890)
• Apply default timeout in StopDataFrameAnalyticsAction.Request #55512
• Create an annotation when a model snapshot is stored #53783 (issue: #52149)
• Do not execute ML CRUD actions when upgrade mode is enabled #54437 (issue: #54326)
• Make find_file_structure recognize Kibana CSV report timestamps #55609 (issue: #55586)
• More advanced model snapshot retention options #56125 (issue: #52150)
• Return assigned node in start/open job/datafeed response #55473 (issue: #54067)
• Skip daily maintenance activity if upgrade mode is enabled #54565 (issue: #54326)
• Start gathering and storing inference stats #53429
• Unassign data frame analytics tasks in SetUpgradeModeAction #54523 (issue: #54326)
• Speed up anomaly detection for the lat_long function #1102
• Reduce CPU scheduling priority of native analysis processes to favor the ES JVM when CPU is constrained. This change is implemented only for Linux and macOS, not for Windows #1109
• Take training_percent into account when estimating memory usage for classification and regression #1111
• Support maximize minimum recall when assigning class labels for multiclass classification #1113
• Improve robustness of anomaly detection to bad input data #1114
• Add new num_matches and preferred_to_categories fields to category output #1062
• Add mean squared logarithmic error (MSLE) for regression #1101
• Add pseudo-Huber loss for regression #1168
• Reduce peak memory usage and memory estimates for classification and regression #1125.)
• Reduce variability of classification and regression results across our target operating systems #1127
• Switch data frame analytics model memory estimates from kilobytes to megabytes #1126 (issue: #54506)
• Add a machine learning native code build for Linux on AArch64 #1132, #1135
• Improve data frame analytics runtime by optimising memory alignment for intrinsic operations #1142
• Fix spurious anomalies for count and sum functions after no data are received for long periods of time #1158
• Improve false positive rates from periodicity test for time series anomaly detection #1177
• Break progress reporting of data frame analyses into multiple phases #1179
• Really centre the data before training for classification and regression begins. This means we can choose more optimal smoothing bias and should reduce the number of trees #1192

Mapping

• Merge V2 index/component template mappings in specific manner #55607 (issue: #53101)

Recovery

• Avoid copying file chunks in peer covery #56072 (issue: #55353)
• Retry failed peer recovery due to transient errors #55353

SQL

• Add BigDecimal support to JDBC #56015 (issue: #43806)
• Drop BASE TABLE type in favour for just TABLE #54836
• Relax version lock between server and clients #56148

Search

• Consolidate DelayableWriteable #55932
• Exists queries to MatchNoneQueryBuilder when the field is unmapped #54857
• Rewrite wrapper queries to match_none if possible #55271
• SearchService#canMatch takes into consideration the alias filter #55120 (issue: #55090)

Snapshot/Restore

• Add GCS support for searchable snapshots #55403
• Allocate searchable snapshots with the balancer #54889 (issues: #50999, #54729)
• Allow bulk snapshot deletes to abort #56009 (issue: #55773)
• Allow deleting multiple snapshots at once #55474
• Allow searching of snapshot taken while indexing #55511 (issue: #50999)
• Allow to prewarm the cache for searchable snapshot shards #55322
• Enable prewarming by default for searchable snapshots #56201 (issue: #55952)
• Permit searches to be concurrent to prewarming #55795
• Reduce contention in CacheFile.fileLock() method #55662
• Require soft deletes for searchable snapshots #55453
• Searchable Snapshots should respect max_restore_bytes_per_sec #55952
• Update the HDFS version used by HDFS Repo #53693
• Use streaming reads for GCS #55506 (issue: #55505)
• Use workers to warm cache parts #55793 (issue: #55322)

Task Management

• Add indexName in update-settings task name #55714
• Add scroll info to search task description #54606
• Broadcast cancellation to only nodes have outstanding child tasks #54312 (issues: #50990, #51157)
• Support hierarchical task cancellation #54757 (issue: #50990)

Transform

• Add throttling #56007 (issue: #54862)

Bug fixes

Aggregations

• Add analytics plugin usage stats to _xpack/usage #54911 (issue: #54847)
• Aggregation support for Value Scripts that change types #54830 (issue: #54655)
• Allow terms agg to default to depth first #54845
• Clean up how pipeline aggs check for multi-bucket #54161 (issue: #53215)
• Fix auto_date_histogram serialization bug #54447 (issues: #54382, #54429)
• Fix error massage for unknown value type #55821 (issue: #55727)
• Fix scripted metric in CCS #54776 (issue: #54758)
• Use Decimal formatter for Numeric ValuesSourceTypes #54366 (issue: #54365)

Allocation

• Fix Broken ExistingStoreRecoverySource Deserialization #55657 (issue: #55513)

Features/ILM+SLM

• ILM stop step execution if writeIndex is false #54805

Features/Indices APIs

• Fix NPE in MetadataIndexTemplateService#findV2Template #54945
• Fix creating filtered alias using now in a date_nanos range query failed #54785 (issue: #54315)
• Fix simulating index templates without specified index #56295 (issues: #53101, #56255)
• Validate non-negative priorities for V2 index templates #56139 (issue: #53101)

Features/Watcher

• Ensure watcher email action message ids are always unique #56574

Infra/Core

• Add generic Set support to streams #54769 (issue: #54708)

Machine Learning

• Fix GET _ml/inference so size param is respected #57303 (issue: #57298)
• Fix file structure finder multiline merge max for delimited formats #56023
• Validate at least one feature is available for DF analytics #55876 (issue: #55593)
• Trap and fail if insufficient features are supplied to data frame analyses. Otherwise, classification and regression got stuck at zero analyzing progress #1160 (issue: #55593)
• Make categorization respect the model_memory_limit #1167 (issue: #1130)
• Respect user overrides for max_trees for classification and regression #1185
• Reset memory status from soft_limit to ok when pruning is no longer required #1193 (issue: #1131)
• Fix restore from training state for classification and regression #1197
• Improve the initialization of seasonal components for anomaly detection #1201 (issue: #1178)

Network

• Fix issue with pipeline releasing bytes early #54458
• Handle TLS file updates during startup #54999 (issue: #54867)

SQL

• Fix DATETIME_PARSE behaviour regarding timezones #56158 (issue: #54960)

Search

• Don’t expand default_field in query_string before required #55158 (issue: #53789)
• Fix time_zone for query_string and date fields #55881 (issue: #55813)

Security

• Fix certutil http for empty password with JDK 11 and lower #55437 (issue: #55386)

Transform

• Fix count when matching exact ids #56544 (issue: #56196)
• Fix http status code when bad scripts are provided #56117 (issue: #55994)

Regressions

Infra/Scripting

• Don’t double-wrap expression values #54432 (issue: #53661)