Elasticsearch version 7.11.0

Elasticsearch version 7.11.0

Also see Breaking changes in 7.11.

Known issues

  • Integration with Active Directory realms and LDAP realms is impacted by an issue that prevents Elasticsearch from starting. If you have configured an Active Directory or LDAP realm, then Elasticseach will fail to start with an error message indicating that Could not initialize class com.unboundid.util.Debug. This exception is fatal. If you encounter this during an upgrade, because Elasticsearch failed during node construction, you can safely downgrade to your previous version of Elasticsearch. Note: these downgrade instructions only apply if you encounter this specific error message; in general, downgrades are not safe. Elasticsearch 7.11.1 will contain a fix for this issue. For more details, see #68838.
  • If autoscaling is enabled for machine learning, the administrator of the cluster should increase the cluster setting xpack.ml.max_open_jobs to the maximum value of 512. This allows autoscaling to run reliably as it relies on assigning jobs only via memory. Having xpack.ml.max_open_jobs as a small number may cause autoscaling to behave unexpectedly.
  • If autoscaling is enabled and job assignment takes an inordinate amount of time, scaling sizes may increase much higher than required. Elasticsearch 7.13.0 contains a fix for this. For more details, see #72509

  • Snapshot and restore: If an index is deleted while the cluster is concurrently taking more than one snapshot then there is a risk that one of the snapshots may never complete and also that some shard data may be lost from the repository, causing future restore operations to fail. To mitigate this problem, set snapshot.max_concurrent_operations: 1 to prevent concurrent snapshot operations:

    1. PUT _cluster/settings
    2. {
    3.   "persistent" : {
    4.     "snapshot.max_concurrent_operations" : 1
    5.   }
    6. }

    This issue is fixed in Elasticsearch versions 7.13.1 and later. It is not possible to repair a repository once it is affected by this issue, so you must restore the repository from a backup, or clear the repository by executing DELETE _snapshot/<repository name>/*, or move to a fresh repository. For more details, see #73456.

  • Parsing a request when the last element in an array is filtered out (for instance using _source_includes) fails. This is due to a bug in Jackson parser. Fixed in Elasticsearch 8.6.1 (#91456)

  • The deprecated index.mapper.dynamic setting can break your cluster. It can only be set using the Update index settings API. Symptoms include nodes failing to start or shards failing to allocate. Do not use this setting in versions prior to 7.17.22. The bug is fixed in 7.17.22. (issue: #109160)

FIPS 140-2 compliance

If you don’t run Elasticsearch in FIPS 140-2 mode, skip this section.

In some cases, such as when using the Bouncy Castle FIPS module in approved only mode, Elasticsearch may raise unexpected exceptions that prevent startup.

In Elasticsearch 7.11.0, we added a new PBKDF2-compatible hashing algorithm to fix this. For more information, see the Elasticsearch FIPS 140-2 documentation.

Breaking changes

Machine Learning

  • Add new flag exclude_generated that removes generated fields in GET config APIs #63899 (issue: #63055)

Mapping

  • Distinguish between simple matches with and without the terms index #63945

Templates

  • Index templates with data stream enabled can no longer contain index aliases or component templates with index aliases. A bug in Elasticsearch 7.10 and 7.9 previously allowed this. #67886

Deprecations

Features/CAT APIs

  • Deprecate the local parameter of /_cat/indices #62198 (issue: #60718)
  • Deprecate the local parameter of /_cat/shards #62197 (issue: #60718)

Features/Indices APIs

Security

  • Deprecate certificate generation without a CA, and add self-signed option #64037 (issue: #61884)

New features

Aggregations

Audit

  • Introduce new audit record for security configuration changes via API #62916

Features/Ingest

Geo

Search

Transform

  • Implement latest function for Transform #65304 (issue: #65869)

Enhancements

Aggregations

  • Add hard_bounds support for histogram field-based histograms #64246 (issue: #62124)
  • Add support for histogram fields to rate aggregation #63289 (issue: #62939)
  • Add supports for upper and lower values on boxplot based on the IQR value #63617 (issue: #60466)
  • Add value_count mode to rate agg #63687 (issue: #63575)
  • Allow mixing set-based and regexp-based include and exclude #63325 (issue: #62246)
  • Consider query when optimizing date rounding #63403
  • Speed up date_histogram without children #63643 (issue: #61467)
  • Speed up interval rounding #63245
  • Use global ordinals in Cardinality aggregation when it makes sense #62560

Allocation

  • Add data tiers preference for snapshot blob cache system index #64004 (issue: #63204)

Audit

  • Log a warning when current license level does not permit auditing #65498

Authentication

  • Adding API for generating SAML SP metadata #64517 (issue: #49018)
  • Adds realm name OIDC _security/oidc/prepare and _security/oidc/authenticate APIs responses #64966 (issue: #53161)
  • Introduce an additional hasher (PBKDF2_STRETCH) #65328
  • Returning tokenGroups attribute as SID string instead of byte array #63509 (issue: #61173)

Authorization

  • Optimize role bitset in document level security #64673

CCR

  • Clarify puts of non-existent settings on followers #64560

Client

  • Adding authentication information to access token create APIs #62490 (issue: #59685)

EQL

  • Add ? character to : operator #65545 (issue: #65536)
  • Add wildcard functionality to : operator #65188 (issue: #65154)
  • Introduce list declaration for : operator #65230 (issue: #64388)

Engine

  • Add index commit id to searcher #63963
  • Realtime get from in-memory segment when possible #64504

Features/Data streams

  • Add custom metadata support to data steams #63991 (issue: #59195)
  • Alow data streams to be marked hidden #63987 (issue: #63889)
  • Include date in data stream backing index names #65205 (issue: #65012)
  • Migrate aliased indices to data stream #61525 (issue: #61046)
  • REST endpoint and tests for data stream migration #64415 (issue: #61046)

Features/Features

Features/ILM+SLM

  • Add validation of the SLM schedule frequency #64452 (issue: #55450)
  • Allow readonly in the hot phase for ILM policies #64381 (issues: #52073, #58289, #64008)
  • Allow shrink in the hot phase for ILM policies #64008 (issues: #52073, #56377)
  • Add support for the searchable_snapshot action in the hot phase #64883 (issue: #64656)
  • Validate policy and actions against current license #65412 (issue: #60208)
  • Make SLM Run Snapshot Deletes in Parallel #62284 (issue: #59655)
  • Use data stream for ILM history #64521

Features/Indices APIs

  • Add a template parameter to override auto_create_index value #61858 (issue: #20640)
  • Make legacy template noop updates not change cluster state #64493 (issues: #57662, #57851)
  • Mark component and composable index template APIs as stable #65013

Features/Ingest

  • Add copy_from parameter for set ingest processor #63540 (issues: #51046, #55682, #56985)
  • Add resource_name field to attachment ingest processor to aid file type recognition #64389
  • Fix IngestDocument.deepCopy to support sets #63067 (issue: #63066)

Features/Monitoring

  • Add action to decommission legacy monitoring cluster alerts #64373 (issue: #62668)

Features/Stats

  • Ignore system indices from mapping stats and analysis stats #65220
  • Update file extensions #62019

Geo

  • Enable geo_distance and geo_bounding_box queries on geo_shape field type #64224 (issue: #54628)
  • geo_point runtime field implementation #63164

Highlighting

  • Make runtime fields highlightable #65560

Infra/Core

  • Add deprecation check for Java version #64996 (issues: #40754, #40756, #41092)
  • Add infrastructure for managing system indices #65604 (issue: #61656)
  • Add stream serialisation for the OffsetTime values #63568 (issue: #63521)
  • Add validation in policy files for missing codebases #64841
  • Use different G1GC options for small heaps #59667

Infra/Plugins

  • Introduce licensed plugins #64850

Infra/Scripting

  • Move helper into painless #63439

Infra/Settings

  • Enhance byte-size setting validation #65363 (issue: #64428)

License

  • Adding a warning header when a license is about to expire #64948 (issue: #60562)

Machine Learning

  • Add new setting xpack.ml.use_auto_machine_memory_percent for auto calculating native memory percentage allowed for jobs #63887 (issue: #63795)
  • Add new snapshot upgrader API for upgrading older snapshots #64665 (issue: #64154)
  • Add search runtime_mappings to datafeed configuration #65606
  • Add total ML memory to ML info #65195 (issue: #64225)
  • Adding result_type and mlcategory fields to category definitions #63326 (issue: #60108)
  • Increase log level for forecast disk storage problems #64766 (issue: #58806)
  • Provide a way to revert an anomaly detection job to an empty snapshot #65431
  • During regression and classification training prefer smaller models if performance is similar #1516
  • Add a response mechanism for commands sent to the native controller #1520, #63542 (issue: #62823)
  • Speed up anomaly detection for seasonal data. This is particularly effective for jobs using longer bucket lengths #1549
  • Fix an edge case which could cause typical and model plot bounds to blow up to around max double #1551
  • Estimate upper bound of potential gains before splitting a decision tree node to avoid unnecessary computation #1537
  • Improvements to time series modeling particularly in relation to adaption to change #16141614)
  • Warn and error log throttling #1615
  • Soften the effect of fluctuations in anomaly detection job memory usage on node assignment and add assignment_memory_basis to model_size_stats #1623, #65561 (issue: #63163)

Mapping

  • Add xpack info and usage endpoints for runtime fields #65600 (issue: #59332)
  • Enable collapse on unsigned_long field #63495 (issue: #60050)
  • Ensure field types consistency on supporting text queries #63487
  • Introduce dynamic runtime setting #65489
  • Introduce runtime section in mappings #62906 (issue: #59332)
  • Improve error message in case of invalid dynamic templates #60870 (issue: #51233)
  • Support unmapped fields in search fields option #65386 (issue: #63690)

Network

  • Include trusted issuer details in SSL diagnostics #61702
  • Remote/info should fail on node without remote_cluster_client role #65289

Packaging

  • Autodetermine heap settings based on node roles and total system memory #65905
  • Clarify usage of jvm.options in its comments #61456 (issue: #51882)

Query Languages

SQL

  • Add method args to PERCENTILE / PERCENTILE_RANK #65026 (issue: #63567)
  • Enhance IN operator support for formatted date fields. Extend allowed conversions #63483 (issue: #58932)

Search

  • Add runtime_mappings to search request #64374 (issue: #59332)
  • Add default scripts for runtime geo-point fields #65586
  • Add support for runtime fields #61776 (issue: #59332)
  • Apply can match phase on coordinator when the min max field data is available at the coordinator #65583 (issue: #65689)
  • Cancel search on shard failure when partial results disallowed #63520 (issue: #60278)
  • Do not skip not available shard exception in search response #64337 (issue: #47700)

Snapshot/Restore

  • Add CacheFile#fsync() method to ensure cached data are written on disk #64201
  • Allow searchable snapshot cache service to periodically fsync cache files #64696 (issue: #64201)
  • Allow adding existing cache files in searchable snapshots cache service #65538
  • Allow creating a SparseFileTracker with ranges already present #65501
  • Expose the list of completed ranges in SparseFileTracker #64145
  • Limit the number of snapshots in a BlobStoreRepository #64461
  • Record timestamp field range in index metadata #65564
  • Upgrade Azure repository SDK to v12 #65140
  • Wait for prewarm when relocating searchable snapshot shards #65531

Task Management

  • Send ban parent per outstanding child connection #65443

Transform

  • Add new exclude_generated flag to GET transform #63093 (issue: #63055)
  • Add support for “missing” aggregation #63651
  • Add support for median absolute deviation #64634
  • Improve continuous transform date_histogram group_by with ingest timestamps #63315 (issues: #54254, #59061)
  • Use ISO dates in output instead of epoch millis #65584 (issue: #63787)

Bug fixes

Aggregations

  • Prevent some BigArray leaking #64744
  • Return -/+ Infinity when min/max bounds are set null #64091
  • Return an error when a rate aggregation cannot calculate bucket sizes #65429 (issue: #63703)

Authentication

  • Don’t invalidate refresh token a second time #64259 (issue: #56903)
  • Fix a NPE bug caused by code keep executing after listener returns #64762 (issue: #64692)

Cluster Coordination

  • Simplify ClusterStateUpdateTask timeout handling #64117 (issue: #64116)

Features/Features

  • Some template alias settings are not persisted on PUT template #63162

Features/Indices APIs

Features/Ingest

  • Fail with a better error when providing null as processor config #64565 (issue: #57793)

Features/Java Low Level REST Client

Geo

  • Compute polygon orientation using signed area #64095 (issue: #26286)
  • Fixes GeoLineAggregator bugs #65521 (issue: #65473)
  • Handle degenerated rectangles while indexing #64122

Infra/Core

  • Fix array handling in XContentMapValues.extractRawValues() #65193

Machine Learning

  • Add ignore_above for message.raw field in notifications index mappings #64455 (issue: #63888)
  • Fix edge case for data frame analytics where a field mapped as a keyword actually has boolean and string values in the _source #64826
  • Fix job ID in C++ logs for normalize and memory estimation #63874 (issues: #54636, #60395)
  • Truncate long audit messages #64849 (issue: #64570)
  • Fix potential cause for log errors from CXMeansOnline1d #1586
  • Fix scaling of some hyperparameters for Bayesian optimization #1612
  • Fix missing state in persist and restore for anomaly detection. This caused suboptimal modeling after a job was closed and reopened or failed over to a different node #1668

Mapping

  • Count only mapped fields towards docvalue_fields limit #63806 (issue: #63730)
  • Fix exception when building exists query for nested search-as-you-type field #64630 (issue: #64609)

Packaging

  • Fix plugin installation in Docker container as non-default user #63807
  • Rework Dockerfile instructions to save space #65308 (issue: #64274)

Search

  • Add explanation to runtime field query #63429
  • Correctly handle mixed object paths in XContentMapValues #65539 (issue: #65499)
  • Drop boost from runtime distance feature query #63949 (issue: #63767)
  • Handle range query edge case #63397 (issue: #63386)
  • Return partial failures if search was cancelled #64382 (issues: #63520, #64367)
  • _mtermvectors should continue request on item error #65324 (issue: #65228)

Security

  • Gracefully handle exceptions from security providers #65464

Snapshot/Restore

  • Fix inconsistent API behavior when cluster never had a snapshot repository configured #65535 (issue: #65511)
  • Fix two snapshot clone state machine bugs #65042
  • Correct the number of searchable snapshot prewarmers #63919 (issue: #59313)
  • Stricter check for snapshot restore version compatibility #65580 (issue: #65567)

Transform