Index Sorting

When creating a new index in Elasticsearch it is possible to configure how the Segments inside each Shard will be sorted. By default Lucene does not apply any sort. The index.sort.* settings define which fields should be used to sort the documents inside each Segment.

nested fields are not compatible with index sorting because they rely on the assumption that nested documents are stored in contiguous doc ids, which can be broken by index sorting. An error will be thrown if index sorting is activated on an index that contains nested fields.

For instance the following example shows how to define a sort on a single field:

  1. PUT my-index-000001
  2. {
  3.   "settings": {
  4.     "index": {
  5.       "sort.field": "date", 
  6.       "sort.order": "desc"  
  7.     }
  8.   },
  9.   "mappings": {
  10.     "properties": {
  11.       "date": {
  12.         "type": "date"
  13.       }
  14.     }
  15.   }
  16. }

This index is sorted by the date field

…​ in descending order.

It is also possible to sort the index by more than one field:

  1. PUT my-index-000001
  2. {
  3.   "settings": {
  4.     "index": {
  5.       "sort.field": [ "username", "date" ], 
  6.       "sort.order": [ "asc", "desc" ]       
  7.     }
  8.   },
  9.   "mappings": {
  10.     "properties": {
  11.       "username": {
  12.         "type": "keyword",
  13.         "doc_values": true
  14.       },
  15.       "date": {
  16.         "type": "date"
  17.       }
  18.     }
  19.   }
  20. }

This index is sorted by username first then by date

…​ in ascending order for the username field and in descending order for the date field.

Index sorting supports the following settings:

index.sort.field

The list of fields used to sort the index. Only boolean, numeric, date and keyword fields with doc_values are allowed here.

index.sort.order

The sort order to use for each field. The order option can have the following values:

  • asc: For ascending order
  • desc: For descending order.

index.sort.mode

Elasticsearch supports sorting by multi-valued fields. The mode option controls what value is picked to sort the document. The mode option can have the following values:

  • min: Pick the lowest value.
  • max: Pick the highest value.

index.sort.missing

The missing parameter specifies how docs which are missing the field should be treated. The missing value can have the following values:

  • _last: Documents without value for the field are sorted last.
  • _first: Documents without value for the field are sorted first.

Index sorting can be defined only once at index creation. It is not allowed to add or update a sort on an existing index. Index sorting also has a cost in terms of indexing throughput since documents must be sorted at flush and merge time. You should test the impact on your application before activating this feature.

Early termination of search request

By default in Elasticsearch a search request must visit every document that match a query to retrieve the top documents sorted by a specified sort. Though when the index sort and the search sort are the same it is possible to limit the number of documents that should be visited per segment to retrieve the N top ranked documents globally. For example, let’s say we have an index that contains events sorted by a timestamp field:

  1. PUT events
  2. {
  3.   "settings": {
  4.     "index": {
  5.       "sort.field": "timestamp",
  6.       "sort.order": "desc" 
  7.     }
  8.   },
  9.   "mappings": {
  10.     "properties": {
  11.       "timestamp": {
  12.         "type": "date"
  13.       }
  14.     }
  15.   }
  16. }

This index is sorted by timestamp in descending order (most recent first)

You can search for the last 10 events with:

  1. GET /events/_search
  2. {
  3.   "size": 10,
  4.   "sort": [
  5.     { "timestamp": "desc" }
  6.   ]
  7. }

Elasticsearch will detect that the top docs of each segment are already sorted in the index and will only compare the first N documents per segment. The rest of the documents matching the query are collected to count the total number of results and to build aggregations.

If you’re only looking for the last 10 events and have no interest in the total number of documents that match the query you can set track_total_hits to false:

  1. GET /events/_search
  2. {
  3.   "size": 10,
  4.   "sort": [ 
  5.       { "timestamp": "desc" }
  6.   ],
  7.   "track_total_hits": false
  8. }

The index sort will be used to rank the top documents and each segment will early terminate the collection after the first 10 matches.

This time, Elasticsearch will not try to count the number of documents and will be able to terminate the query as soon as N documents have been collected per segment.

  1. {
  2.   "_shards": ...
  3.    "hits" : {  
  4.       "max_score" : null,
  5.       "hits" : []
  6.   },
  7.   "took": 20,
  8.   "timed_out": false
  9. }

The total number of hits matching the query is unknown because of early termination.

Aggregations will collect all documents that match the query regardless of the value of track_total_hits