Ingest processor reference
- Ingest processor reference

Ingest processor reference

An ingest pipeline is made up of a sequence of processors that are applied to documents as they are ingested into an index. Each processor performs a specific task, such as filtering, transforming, or enriching data.

Each successive processor depends on the output of the previous processor, so the order of processors is important. The modified documents are indexed into Elasticsearch after all processors are applied.

Elasticsearch includes over 40 configurable processors. The subpages in this section contain reference documentation for each processor. To get a list of available processors, use the nodes info API.

resp = client.nodes.info(
    node_id="ingest",
    filter_path="nodes.*.ingest.processors",
)
print(resp)

response = client.nodes.info(
  node_id: 'ingest',
  filter_path: 'nodes.*.ingest.processors'
)
puts response

const response = await client.nodes.info({
  node_id: "ingest",
  filter_path: "nodes.*.ingest.processors",
});
console.log(response);

GET _nodes/ingest?filter_path=nodes.*.ingest.processors

Ingest processors by category

We’ve categorized the available processors on this page and summarized their functions. This will help you find the right processor for your use case.

Data enrichment processors

General outcomes

append processor

Appends a value to a field.

date_index_name processor

Points documents to the right time-based index based on a date or timestamp field.

enrich processor

Enriches documents with data from another index.

Refer to Enrich your data for detailed examples of how to use the enrich processor to add data from your existing indices to incoming documents during ingest.

inference processor

Uses machine learning to classify and tag text fields.

Specific outcomes

attachment processor

Parses and indexes binary data, such as PDFs and Word documents.

circle processor

Converts a location field to a Geo-Point field.

community_id processor

Computes the Community ID for network flow data.

fingerprint processor

Computes a hash of the document’s content.

geo_grid processor

Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape.

geoip processor

Adds information about the geographical location of an IPv4 or IPv6 address from a Maxmind database.

ip_location processor

Adds information about the geographical location of an IPv4 or IPv6 address from an ip geolocation database.

network_direction processor

Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.

registered_domain processor

Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN).

set_security_user processor

Sets user-related details (such as username, roles, email, full_name,metadata, api_key, realm and authentication_type) from the current authenticated user to the current document by pre-processing the ingest.

uri_parts processor

Parses a Uniform Resource Identifier (URI) string and extracts its components as an object.

urldecode processor

URL-decodes a string.

user_agent processor

Parses user-agent strings to extract information about web clients.

Data transformation processors

General outcomes

convert processor

Converts a field in the currently ingested document to a different type, such as converting a string to an integer.

dissect processor

Extracts structured fields out of a single text field within a document. Unlike the grok processor, dissect does not use regular expressions. This makes the dissect’s a simpler and often faster alternative.

grok processor

Extracts structured fields out of a single text field within a document, using the Grok regular expression dialect that supports reusable aliased expressions.

gsub processor

Converts a string field by applying a regular expression and a replacement.

redact processor

Uses the Grok rules engine to obscure text in the input document matching the given Grok patterns.

rename processor

Renames an existing field.

set processor

Sets a value on a field.