Quick start

Quick start

This guide helps beginners learn how to:

  • Install and run Elasticsearch in a test environment
  • Add data to Elasticsearch
  • Search and sort data
  • Extract fields from unstructured content during a search

Run Elasticsearch

The simplest way to set up Elasticsearch is to create a managed deployment with Elasticsearch Service on Elastic Cloud. If you prefer to manage your own test environment, you can install and run Elasticsearch using Docker.

Elasticsearch Service Self-managed

  1. Sign up for a free trial.
  2. Follow the on-screen steps to create your first project.
  3. Click Continue to open Kibana.
  4. Click Explore on my own.

Install and run Elasticsearch

  1. Install and start Docker Desktop.

  2. Run:

    1. docker network create elastic
    2. docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.26
    3. docker run --name es01-test --net elastic -p 127.0.0.1:9200:9200 -p 127.0.0.1:9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.17.26

Install and run Kibana

To analyze, visualize, and manage Elasticsearch data using an intuitive UI, install Kibana.

  1. In a new terminal session, run:

    1. docker pull docker.elastic.co/kibana/kibana:7.17.26
    2. docker run --name kib01-test --net elastic -p 127.0.0.1:5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.17.26

  2. To access Kibana, go to http://localhost:5601

Send requests to Elasticsearch

You send data and other requests to Elasticsearch using REST APIs. This lets you interact with Elasticsearch using any client that sends HTTP requests, such as curl. You can also use Kibana’s console to send requests to Elasticsearch.

Elasticsearch Service Self-managed

Use Kibana

  1. Open Kibana’s main menu and go to Dev Tools > Console.

    Kibana Console

  2. Run the following example API request in the console:

    1. GET /

Use curl

To communicate with Elasticsearch using curl or another client, you need your cluster’s endpoint.

  1. Open Kibana’s main menu and click Manage this deployment.
  2. From your deployment menu, go to the Elasticsearch page. Click Copy endpoint.

  3. To submit an example API request, run the following curl command in a new terminal session. Replace <password> with the password for the elastic user. Replace <elasticsearch_endpoint> with your endpoint.

    1. curl -u elastic:<password> <elasticsearch_endpoint>/

Use Kibana

  1. Open Kibana’s main menu and go to Dev Tools > Console.

    Kibana Console

  2. Run the following example API request in the console:

    1. GET /

Use curl

To submit an example API request, run the following curl command in a new terminal session.

  1. curl -X GET http://localhost:9200/

Add data

You add data to Elasticsearch as JSON objects called documents. Elasticsearch stores these documents in searchable indices.

For time series data, such as logs and metrics, you typically add documents to a data stream made up of multiple auto-generated backing indices.

A data stream requires an index template that matches its name. Elasticsearch uses this template to configure the stream’s backing indices. Documents sent to a data stream must have a @timestamp field.

Add a single document

Submit the following indexing request to add a single log entry to the logs-my_app-default data stream. Since logs-my_app-default doesn’t exist, the request automatically creates it using the built-in logs-*-* index template.

  1. POST logs-my_app-default/_doc
  2. {
  3.   "@timestamp": "2099-05-06T16:21:15.000Z",
  4.   "event": {
  5.     "original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
  6.   }
  7. }

The response includes metadata that Elasticsearch generates for the document:

  • The backing _index that contains the document. Elasticsearch automatically generates the names of backing indices.
  • A unique _id for the document within the index.
  1. {
  2.   "_index": ".ds-logs-my_app-default-2099-05-06-000001",
  3.   "_type": "_doc",
  4.   "_id": "gl5MJXMBMk1dGnErnBW8",
  5.   "_version": 1,
  6.   "result": "created",
  7.   "_shards": {
  8.     "total": 2,
  9.     "successful": 1,
  10.     "failed": 0
  11.   },
  12.   "_seq_no": 0,
  13.   "_primary_term": 1
  14. }

Add multiple documents

Use the _bulk endpoint to add multiple documents in one request. Bulk data must be newline-delimited JSON (NDJSON). Each line must end in a newline character (\n), including the last line.

  1. PUT logs-my_app-default/_bulk
  2. { "create": { } }
  3. { "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
  4. { "create": { } }
  5. { "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }

Search data

Indexed documents are available for search in near real-time. The following search matches all log entries in logs-my_app-default and sorts them by @timestamp in descending order.

  1. GET logs-my_app-default/_search
  2. {
  3.   "query": {
  4.     "match_all": { }
  5.   },
  6.   "sort": [
  7.     {
  8.       "@timestamp": "desc"
  9.     }
  10.   ]
  11. }

By default, the hits section of the response includes up to the first 10 documents that match the search. The _source of each hit contains the original JSON object submitted during indexing.

  1. {
  2.   "took": 2,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 1,
  6.     "successful": 1,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 3,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": null,
  16.     "hits": [
  17.       {
  18.         "_index": ".ds-logs-my_app-default-2099-05-06-000001",
  19.         "_type": "_doc",
  20.         "_id": "PdjWongB9KPnaVm2IyaL",
  21.         "_score": null,
  22.         "_source": {
  23.           "@timestamp": "2099-05-08T16:25:42.000Z",
  24.           "event": {
  25.             "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638"
  26.           }
  27.         },
  28.         "sort": [
  29.           4081940742000
  30.         ]
  31.       },
  32.       ...
  33.     ]
  34.   }
  35. }

Get specific fields

Parsing the entire _source is unwieldy for large documents. To exclude it from the response, set the _source parameter to false. Instead, use the fields parameter to retrieve the fields you want.

  1. GET logs-my_app-default/_search
  2. {
  3.   "query": {
  4.     "match_all": { }
  5.   },
  6.   "fields": [
  7.     "@timestamp"
  8.   ],
  9.   "_source": false,
  10.   "sort": [
  11.     {
  12.       "@timestamp": "desc"
  13.     }
  14.   ]
  15. }

The response contains each hit’s fields values as a flat array.

  1. {
  2.   ...
  3.   "hits": {
  4.     ...
  5.     "hits": [
  6.       {
  7.         "_index": ".ds-logs-my_app-default-2099-05-06-000001",
  8.         "_type": "_doc",
  9.         "_id": "PdjWongB9KPnaVm2IyaL",
  10.         "_score": null,
  11.         "fields": {
  12.           "@timestamp": [
  13.             "2099-05-08T16:25:42.000Z"
  14.           ]
  15.         },
  16.         "sort": [
  17.           4081940742000
  18.         ]
  19.       },
  20.       ...
  21.     ]
  22.   }
  23. }

Search a date range

To search across a specific time or IP range, use a range query.

  1. GET logs-my_app-default/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "@timestamp": {
  6.         "gte": "2099-05-05",
  7.         "lt": "2099-05-08"
  8.       }
  9.     }
  10.   },
  11.   "fields": [
  12.     "@timestamp"
  13.   ],
  14.   "_source": false,
  15.   "sort": [
  16.     {
  17.       "@timestamp": "desc"
  18.     }
  19.   ]
  20. }

You can use date math to define relative time ranges. The following query searches for data from the past day, which won’t match any log entries in logs-my_app-default.

  1. GET logs-my_app-default/_search
  2. {
  3.   "query": {
  4.     "range": {
  5.       "@timestamp": {
  6.         "gte": "now-1d/d",
  7.         "lt": "now/d"
  8.       }
  9.     }
  10.   },
  11.   "fields": [
  12.     "@timestamp"
  13.   ],
  14.   "_source": false,
  15.   "sort": [
  16.     {
  17.       "@timestamp": "desc"
  18.     }
  19.   ]
  20. }

Extract fields from unstructured content

You can extract runtime fields from unstructured content, such as log messages, during a search.

Use the following search to extract the source.ip runtime field from event.original. To include it in the response, add source.ip to the fields parameter.

  1. GET logs-my_app-default/_search
  2. {
  3.   "runtime_mappings": {
  4.     "source.ip": {
  5.       "type": "ip",
  6.       "script": """
  7.         String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
  8.         if (sourceip != null) emit(sourceip);
  9.       """
  10.     }
  11.   },
  12.   "query": {
  13.     "range": {
  14.       "@timestamp": {
  15.         "gte": "2099-05-05",
  16.         "lt": "2099-05-08"
  17.       }
  18.     }
  19.   },
  20.   "fields": [
  21.     "@timestamp",
  22.     "source.ip"
  23.   ],
  24.   "_source": false,
  25.   "sort": [
  26.     {
  27.       "@timestamp": "desc"
  28.     }
  29.   ]
  30. }

Combine queries

You can use the bool query to combine multiple queries. The following search combines two range queries: one on @timestamp and one on the source.ip runtime field.

  1. GET logs-my_app-default/_search
  2. {
  3.   "runtime_mappings": {
  4.     "source.ip": {
  5.       "type": "ip",
  6.       "script": """
  7.         String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
  8.         if (sourceip != null) emit(sourceip);
  9.       """
  10.     }
  11.   },
  12.   "query": {
  13.     "bool": {
  14.       "filter": [
  15.         {
  16.           "range": {
  17.             "@timestamp": {
  18.               "gte": "2099-05-05",
  19.               "lt": "2099-05-08"
  20.             }
  21.           }
  22.         },
  23.         {
  24.           "range": {
  25.             "source.ip": {
  26.               "gte": "192.0.2.0",
  27.               "lte": "192.0.2.240"
  28.             }
  29.           }
  30.         }
  31.       ]
  32.     }
  33.   },
  34.   "fields": [
  35.     "@timestamp",
  36.     "source.ip"
  37.   ],
  38.   "_source": false,
  39.   "sort": [
  40.     {
  41.       "@timestamp": "desc"
  42.     }
  43.   ]
  44. }

Aggregate data

Use aggregations to summarize data as metrics, statistics, or other analytics.

The following search uses an aggregation to calculate the average_response_size using the http.response.body.bytes runtime field. The aggregation only runs on documents that match the query.

  1. GET logs-my_app-default/_search
  2. {
  3.   "runtime_mappings": {
  4.     "http.response.body.bytes": {
  5.       "type": "long",
  6.       "script": """
  7.         String bytes=grok('%{COMMONAPACHELOG}').extract(doc[ "event.original" ].value)?.bytes;
  8.         if (bytes != null) emit(Integer.parseInt(bytes));
  9.       """
  10.     }
  11.   },
  12.   "aggs": {
  13.     "average_response_size":{
  14.       "avg": {
  15.         "field": "http.response.body.bytes"
  16.       }
  17.     }
  18.   },
  19.   "query": {
  20.     "bool": {
  21.       "filter": [
  22.         {
  23.           "range": {
  24.             "@timestamp": {
  25.               "gte": "2099-05-05",
  26.               "lt": "2099-05-08"
  27.             }
  28.           }
  29.         }
  30.       ]
  31.     }
  32.   },
  33.   "fields": [
  34.     "@timestamp",
  35.     "http.response.body.bytes"
  36.   ],
  37.   "_source": false,
  38.   "sort": [
  39.     {
  40.       "@timestamp": "desc"
  41.     }
  42.   ]
  43. }

The response’s aggregations object contains aggregation results.

  1. {
  2.   ...
  3.   "aggregations" : {
  4.     "average_response_size" : {
  5.       "value" : 12368.0
  6.     }
  7.   }
  8. }

Explore more search options

To keep exploring, index more data to your data stream and check out Common search options.

Clean up

When you’re done, delete your test data stream and its backing indices.

  1. DELETE _data_stream/logs-my_app-default

You can also delete your test deployment.

Elasticsearch Service Self-managed

Click Delete deployment from the deployment overview page and follow the prompts.

To stop your Elasticsearch and Kibana Docker containers, run:

  1. docker stop es01-test
  2. docker stop kib01-test

To remove the containers and their network, run:

  1. docker network rm elastic
  2. docker rm es01-test
  3. docker rm kib01-test

What’s next?