Collapse search results

You can use the collapse parameter to collapse search results based on field values. The collapsing is done by selecting only the top sorted document per collapse key.

For example, the following search collapses results by user.id and sorts them by http.response.bytes.

GET /my-index-000001/_search
{
  "query": {
    "match": {
      "message": "GET /search"
    }
  },
  "collapse": {
    "field": "user.id"                
  },
  "sort": [ "http.response.bytes" ],  
  "from": 10                          
}

The total number of hits in the response indicates the number of matching documents without collapsing. The total number of distinct group is unknown.

The field used for collapsing must be a single valued keyword or numeric field with doc_values activated

The collapsing is applied to the top hits only and does not affect aggregations.

Expand collapse results

It is also possible to expand each collapsed top hits with the inner_hits option.

GET /my-index-000001/_search
{
  "query": {
    "match": {
      "message": "GET /search"
    }
  },
  "collapse": {
    "field": "user.id",                       
    "inner_hits": {
      "name": "most_recent",                  
      "size": 5,                              
      "sort": [ { "@timestamp": "asc" } ]     
    },
    "max_concurrent_group_searches": 4        
  },
  "sort": [ "http.response.bytes" ]
}

See inner hits for the complete list of supported options and the format of the response.

It is also possible to request multiple inner_hits for each collapsed hit. This can be useful when you want to get multiple representations of the collapsed hits.

GET /my-index-000001/_search
{
  "query": {
    "match": {
      "message": "GET /search"
    }
  },
  "collapse": {
    "field": "user.id",                      
      "inner_hits": [
      {
        "name": "largest_responses",         
        "size": 3,
        "sort": [ "http.response.bytes" ]
      },
      {
        "name": "most_recent",               
        "size": 3,
        "sort": [ { "@timestamp": "asc" } ]
      }
    ]
  },
  "sort": [ "http.response.bytes" ]
}

The expansion of the group is done by sending an additional query for each inner_hit request for each collapsed hit returned in the response. This can significantly slow things down if you have too many groups and/or inner_hit requests.

The max_concurrent_group_searches request parameter can be used to control the maximum number of concurrent searches allowed in this phase. The default is based on the number of data nodes and the default search thread pool size.

collapse cannot be used in conjunction with scroll, rescore or search after.

Second level of collapsing

Second level of collapsing is also supported and is applied to inner_hits.

For example, the following search collapses results by geo.country_name. Within each geo.country_name, inner hits are collapsed by user.id.

GET /my-index-000001/_search
{
  "query": {
    "match": {
      "message": "GET /search"
    }
  },
  "collapse": {
    "field": "geo.country_name",
    "inner_hits": {
      "name": "by_location",
      "collapse": { "field": "user.id" },
      "size": 3
    }
  }
}

Response:

{
  ...
  "hits": [
    {
      "_index": "my-index-000001",
      "_type": "_doc",
      "_id": "9",
      "_score": ...,
      "_source": {...},
      "fields": { "geo": { "country_name": [ "UK" ] }},
      "inner_hits": {
        "by_location": {
          "hits": {
            ...,
            "hits": [
              {
                ...
                "fields": { "user": "id": { [ "user124" ] }}
              },
              {
                ...
                "fields": { "user": "id": { [ "user589" ] }}
              },
              {
                ...
                "fields": { "user": "id": { [ "user001" ] }}
              }
            ]
          }
        }
      }
    },
    {
      "_index": "my-index-000001",
      "_type": "_doc",
      "_id": "1",
      "_score": ..,
      "_source": {...
      },
      "fields": { "geo": { "country_name": [ "Canada" ] }},
      "inner_hits": {
        "by_location": {
          "hits": {
            ...,
            "hits": [
              {
                ...
                "fields": { "user": "id": { [ "user444" ] }}
              },
              {
                ...
                "fields": { "user": "id": { [ "user1111" ] }
              },
              {
                ...
                  "fields": { "user": "id": { [ "user999" ] }}
              }
            ]
          }
        }
      }
    },
    ...
  ]
}

Second level of collapsing doesn’t allow inner_hits.