Retrieve a runtime field

Retrieve a runtime field

Use the fields parameter on the _search API to retrieve the values of runtime fields. Runtime fields won’t display in _source, but the fields API works for all fields, even those that were not sent as part of the original _source.

Define a runtime field to calculate the day of week

For example, the following request adds a runtime field called day_of_week. The runtime field includes a script that calculates the day of the week based on the value of the @timestamp field. We’ll include "dynamic":"runtime" in the request so that new fields are added to the mapping as runtime fields.

  1. PUT my-index-000001/
  2. {
  3.   "mappings": {
  4.     "dynamic": "runtime",
  5.     "runtime": {
  6.       "day_of_week": {
  7.         "type": "keyword",
  8.         "script": {
  9.           "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))"
  10.         }
  11.       }
  12.     },
  13.     "properties": {
  14.       "@timestamp": {"type": "date"}
  15.     }
  16.   }
  17. }

Ingest some data

Let’s ingest some sample data, which will result in two indexed fields: @timestamp and message.

  1. POST /my-index-000001/_bulk?refresh
  2. { "index": {}}
  3. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  4. { "index": {}}
  5. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  6. { "index": {}}
  7. { "@timestamp": "2020-04-30T14:30:17-05:00", "message" : "40.135.0.0 - - [2020-04-30T14:30:17-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  8. { "index": {}}
  9. { "@timestamp": "2020-04-30T14:30:53-05:00", "message" : "232.0.0.0 - - [2020-04-30T14:30:53-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  10. { "index": {}}
  11. { "@timestamp": "2020-04-30T14:31:12-05:00", "message" : "26.1.0.0 - - [2020-04-30T14:31:12-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  12. { "index": {}}
  13. { "@timestamp": "2020-04-30T14:31:19-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:19-05:00] \"GET /french/splash_inet.html HTTP/1.0\" 200 3781"}
  14. { "index": {}}
  15. { "@timestamp": "2020-04-30T14:31:27-05:00", "message" : "252.0.0.0 - - [2020-04-30T14:31:27-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  16. { "index": {}}
  17. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_brdl.gif HTTP/1.0\" 304 0"}
  18. { "index": {}}
  19. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_arw.gif HTTP/1.0\" 304 0"}
  20. { "index": {}}
  21. { "@timestamp": "2020-04-30T14:31:32-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:32-05:00] \"GET /images/nav_bg_top.gif HTTP/1.0\" 200 929"}
  22. { "index": {}}
  23. { "@timestamp": "2020-04-30T14:31:43-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:43-05:00] \"GET /french/images/nav_venue_off.gif HTTP/1.0\" 304 0"}

Search for the calculated day of week

The following request uses the search API to retrieve the day_of_week field that the original request defined as a runtime field in the mapping. The value for this field is calculated dynamically at query time without reindexing documents or indexing the day_of_week field. This flexibility allows you to modify the mapping without changing any field values.

  1. GET my-index-000001/_search
  2. {
  3.   "fields": [
  4.     "@timestamp",
  5.     "day_of_week"
  6.   ],
  7.   "_source": false
  8. }

The previous request returns the day_of_week field for all matching documents. We can define another runtime field called client_ip that also operates on the message field and will further refine the query:

  1. PUT /my-index-000001/_mapping
  2. {
  3.   "runtime": {
  4.     "client_ip": {
  5.       "type": "ip",
  6.       "script" : {
  7.       "source" : "String m = doc[\"message\"].value; int end = m.indexOf(\" \"); emit(m.substring(0, end));"
  8.       }
  9.     }
  10.   }
  11. }

Run another query, but search for a specific IP address using the client_ip runtime field:

  1. GET my-index-000001/_search
  2. {
  3.   "size": 1,
  4.   "query": {
  5.     "match": {
  6.       "client_ip": "211.11.9.0"
  7.     }
  8.   },
  9.   "fields" : ["*"]
  10. }

This time, the response includes only two hits. The value for day_of_week (Sunday) was calculated at query time using the runtime script defined in the mapping, and the result includes only documents matching the 211.11.9.0 IP address.

  1. {
  2.   ...
  3.   "hits" : {
  4.     "total" : {
  5.       "value" : 2,
  6.       "relation" : "eq"
  7.     },
  8.     "max_score" : 1.0,
  9.     "hits" : [
  10.       {
  11.         "_index" : "my-index-000001",
  12.         "_type" : "_doc",
  13.         "_id" : "oWs5KXYB-XyJbifr9mrz",
  14.         "_score" : 1.0,
  15.         "_source" : {
  16.           "@timestamp" : "2020-06-21T15:00:01-05:00",
  17.           "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  18.         },
  19.         "fields" : {
  20.           "@timestamp" : [
  21.             "2020-06-21T20:00:01.000Z"
  22.           ],
  23.           "client_ip" : [
  24.             "211.11.9.0"
  25.           ],
  26.           "message" : [
  27.             "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  28.           ],
  29.           "day_of_week" : [
  30.             "Sunday"
  31.           ]
  32.         }
  33.       }
  34.     ]
  35.   }
  36. }