EQL search

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Event Query Language (EQL) is a query language for event-based, time series data, such as logs.

Advantages of EQL

• EQL lets you express relationships between events.
  Many query languages allow you to match only single events. EQL lets you match a sequence of events across different event categories and time spans.
• EQL has a low learning curve.
  EQL syntax looks like other query languages. It lets you write and read queries intuitively, which makes for quick, iterative searching.
• We designed EQL for security use cases.
  While you can use EQL for any event-based data, we created EQL for threat hunting. EQL not only supports indicator of compromise (IOC) searching but makes it easy to describe activity that goes beyond IOCs.

Required fields

While no schema is required to use EQL in Elasticsearch, we recommend using the Elastic Common Schema (ECS). EQL search is designed to work with core ECS fields by default.

EQL assumes each document in a data stream or index corresponds to an event. To run an EQL search, each document must contain a timestamp and event category field.

EQL uses the @timestamp and event.category fields from the ECS as the default timestamp and event category fields. If your documents use a different timestamp or event category field, you must specify it in the search request. See Specify a timestamp or event category field.

Run an EQL search

You can use the EQL search API to run an EQL search. For supported query syntax, see Syntax reference.

The following request searches my-index-000001 for events with an event.category of process and a process.name of regsvr32.exe. Each document in my-index-000001 includes a @timestamp and event.category field.

GET /my-index-000001/_eql/search
{
  "query": """
    process where process.name == "regsvr32.exe"
  """
}

The API returns the following response. Matching events are included in the hits.events property. These events are sorted by timestamp, converted to milliseconds since the Unix epoch, in ascending order.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "my-index-000001",
        "_type": "_doc",
        "_id": "OQmfCaduce8zoHT93o4H",
        "_score": null,
        "_source": {
          "@timestamp": "2099-12-07T11:07:09.000Z",
          "event": {
            "category": "process",
            "id": "aR3NWVOs",
            "sequence": 4
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      },
      {
        "_index": "my-index-000001",
        "_type": "_doc",
        "_id": "xLkCaj4EujzdNSxfYLbO",
        "_score": null,
        "_source": {
          "@timestamp": "2099-12-07T11:07:10.000Z",
          "event": {
            "category": "process",
            "id": "GTSmSqgz0U",
            "sequence": 6,
            "type": "termination"
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      }
    ]
  }
}

Search for a sequence of events

You can use EQL’s sequence syntax to search for an ordered series of events.

The following EQL search request matches a sequence that:

1. Starts with an event with:

   • An event.category of process
   • A process.name of regsvr32.exe

2. Followed by an event with:

   • An event.category of file
   • A file.name that contains the substring scrobj.dll

GET /my-index-000001/_eql/search
{
  "query": """
    sequence
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}

The API returns the following response. Matching sequences are included in the hits.sequences property.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "my-index-000001",
            "_type": "_doc",
            "_id": "OQmfCaduce8zoHT93o4H",
            "_version": 1,
            "_seq_no": 3,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2099-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 4
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_type": "_doc",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_version": 1,
            "_seq_no": 4,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2099-12-07T11:07:10.000Z",
              "event": {
                "category": "file",
                "id": "tZ1NWVOs",
                "sequence": 5
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              },
              "file": {
                "path": "C:\\Windows\\System32\\scrobj.dll",
                "name": "scrobj.dll"
              }
            }
          }
        ]
      }
    ]
  }
}

You can use the with maxspan keywords to constrain a sequence to a specified timespan.

The following EQL search request adds with maxspan=1h to the previous query. This ensures all events in a matching sequence occur within 1h (one hour) of the first event’s timestamp.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}

You can further constrain matching event sequences using the by keyword.

The following EQL search request adds by process.pid to each event item. This ensures events matching the sequence share the same process.pid field value.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ] by process.pid
      [ file where stringContains(file.name, "scrobj.dll") ] by process.pid
  """
}

Because the process.pid field is shared across all events in the sequence, it can be included using sequence by. The following query is equivalent to the previous one.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}

The API returns the following response. The hits.sequences.join_keys property contains the shared process.pid value for each matching event.

{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          2012
        ],
        "events": [
          {
            "_index": "my-index-000001",
            "_type": "_doc",
            "_id": "OQmfCaduce8zoHT93o4H",
            "_version": 1,
            "_seq_no": 3,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2099-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 4
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_type": "_doc",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_version": 1,
            "_seq_no": 4,
            "_primary_term": 1,
            "_score": null,
            "_source": {
              "@timestamp": "2099-12-07T11:07:10.000Z",
              "event": {
                "category": "file",
                "id": "tZ1NWVOs",
                "sequence": 5
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              },
              "file": {
                "path": "C:\\Windows\\System32\\scrobj.dll",
                "name": "scrobj.dll"
              }
            }
          }
        ]
      }
    ]
  }
}

You can use the until keyword to specify an expiration event for sequences. Matching sequences must end before this event.

The following request adds until [ process where event.type == "termination" ] to the previous query. This ensures matching sequences end before a process event with an event.type of termination.

GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
    until [ process where event.type == "termination" ]
  """
}

Specify a timestamp or event category field

To run an EQL search, each searched document must contain a timestamp and event category field. The EQL search API uses the @timestamp and event.category fields from the Elastic Common Schema (ECS) by default. If your documents use a different timestamp or event category field, you must specify it in the search request using the timestamp_field or event_category_field parameters.

The event category field is typically mapped as a field type in the keyword family. The timestamp field is typically mapped as a date or date_nanos field.

You cannot use a nested field or the sub-fields of a nested field as the timestamp or event category field. See EQL search on nested fields.

The following request uses the timestamp_field parameter to specify file.accessed as the timestamp field. The request also uses the event_category_field parameter to specify file.type as the event category field.

GET /my-index-000001/_eql/search
{
  "timestamp_field": "file.accessed",
  "event_category_field": "file.type",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}

Filter using query DSL

You can use the filter parameter to specify an additional query using query DSL. This query filters the documents on which the EQL query runs.

The following request uses a range query to filter my-index-000001 to only documents with a file.size value greater than 1 but less than 1000000 bytes. The EQL query in query parameter then runs on these filtered documents.

GET /my-index-000001/_eql/search
{
  "filter": {
    "range" : {
      "file.size" : {
        "gte" : 1,
        "lte" : 1000000
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}

Run a case-sensitive EQL search

By default, matching for EQL queries is case-insensitive. You can use the case_sensitive parameter to toggle case sensitivity on or off.

The following search request contains a query that matches process events with a process.executable containing System32.

Because case_sensitive is true, this query only matches process.executable values containing System32 with the exact same capitalization. A process.executable value containing system32 or SYSTEM32 would not match this query.

GET /my-index-000001/_eql/search
{
  "keep_on_completion": true,
  "case_sensitive": true,
  "query": """
    process where stringContains(process.executable, "System32")
  """
}

Run an async EQL search

EQL searches are designed to run on large volumes of data quickly, often returning results in milliseconds. For this reason, EQL searches are synchronous by default. The search request waits for complete results before returning a response.

However, complete results can take longer for searches across:

• Frozen indices
• Multiple clusters
• Many shards

To avoid long waits, you can use the wait_for_completion_timeout parameter to run an asynchronous, or async, EQL search.

Set wait_for_completion_timeout to a duration you’d like to wait for complete search results. If the search request does not finish within this period, the search becomes async and returns a response that includes:

• A search ID, which can be used to monitor the progress of the async search.
• An is_partial value of true, meaning the response does not contain complete search results.
• An is_running value of true, meaning the search is async and ongoing.

The async search continues to run in the background without blocking other requests.

The following request searches the frozen-my-index-000001 index, which has been frozen for storage and is rarely searched.

Because searches on frozen indices are expected to take longer to complete, the request contains a wait_for_completion_timeout parameter value of 2s (two seconds). If the request does not return complete results in two seconds, the search becomes async and returns a search ID.

GET /frozen-my-index-000001/_eql/search
{
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

After two seconds, the request returns the following response. Note is_partial and is_running properties are true, indicating an async search.

{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": true,
  "is_running": true,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}

You can use the the search ID and the get async EQL search API to check the progress of an async search.

The get async EQL search API also accepts a wait_for_completion_timeout parameter. If ongoing search does not complete during this period, the response returns an is_partial value of true and no search results.

The following get async EQL search API request checks the progress of the previous async EQL search. The request specifies a wait_for_completion_timeout query parameter value of 2s (two seconds).

GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s

The request returns the following response. Note is_partial and is_running are false, indicating the async search has finished and the search results in the hits property are complete.

{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": false,
  "is_running": false,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}

Change the search retention period

By default, the EQL search API stores async searches for five days. After this period, any searches and their results are deleted. You can use the keep_alive parameter to change this retention period.

In the following EQL search request, the keep_alive parameter is 2d (two days). If the search becomes async, its results are stored on the cluster for two days. After two days, the async search and its results are deleted, even if it’s still ongoing.

GET /my-index-000001/_eql/search
{
  "keep_alive": "2d",
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

You can use the get async EQL search API‘s `keep_alive`parameter to later change the retention period. The new retention period starts after the get request executes.

The following request sets the keep_alive query parameter to 5d (five days). The async search and its results are deleted five days after the get request executes.

GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d

You can use the delete async EQL search API to manually delete an async EQL search before the keep_alive period ends. If the search is still ongoing, this cancels the search request.

The following request deletes an async EQL search and its results.

DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d

Store synchronous EQL searches

By default, the EQL search API only stores async searches that cannot be completed within the period set by wait_for_completion_timeout.

To save the results of searches that complete during this period, set the keep_on_completion parameter to true.

In the following search request, keep_on_completion is true. This means the search results are stored on the cluster, even if the search completes within the 2s (two-second) period set by the wait_for_completion_timeout parameter.

GET /my-index-000001/_eql/search
{
  "keep_on_completion": true,
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}

The API returns the following response. A search ID is provided in the id property. is_partial and is_running are false, indicating the EQL search was synchronous and returned complete results in hits.

{
  "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  "is_partial": false,
  "is_running": false,
  "took": 52,
  "timed_out": false,
  "hits": ...
}

You can use the search ID and the get async EQL search API to retrieve the same results later.

GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=

Saved synchronous searches are still subject to the retention period set by the keep_alive parameter. After this period, the search and its results are deleted.

You can also manually delete saved synchronous searches using the delete async EQL search API.