description: Send logs to Splunk HTTP Event Collector

Splunk

Splunk output plugin allows to ingest your records into a Splunk Enterprise service through the HTTP Event Collector (HEC) interface.

To get more details about how to setup the HEC in Splunk please refer to the following documentation: Splunk / Use the HTTP Event Collector

Configuration Parameters

Key Description default
Host IP address or hostname of the target Splunk service. 127.0.0.1
Port TCP port of the target Splunk service. 8088
Splunk_Token Specify the Authentication Token for the HTTP Event Collector interface.
Splunk_Send_Raw

When enabled, the record keys and values are set in the top level of the map instead of under the event key.

note: refer to the Sending Raw Events section below for more details to make this option work properly.

 Off
HTTP_User Optional username for Basic Authentication on HEC
HTTP_Passwd Password for user defined in HTTP_User

TLS / SSL

Splunk output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the TLS/SSL section.

Getting Started

In order to insert records into a Splunk service, you can run the plugin from the command line or through the configuration file:

Command Line

The splunk plugin, can read the parameters from the command line in two ways, through the -p argument (property), e.g:

  1. $ fluent-bit -i cpu -t cpu -o splunk -p host=127.0.0.1 -p port=8088 \
  2.   -p tls=on -p tls.verify=off -m '*'

Configuration File

In your main configuration file append the following Input & Output sections:

  1. [INPUT]
  2.     Name  cpu
  3.     Tag   cpu
  5. [OUTPUT]
  6.     Name        splunk
  7.     Match       *
  8.     Host        127.0.0.1
  9.     Port        8088
  10.     TLS         On
  11.     TLS.Verify  Off
  12.     Message_Key my_key

Data format

By default, the Splunk output plugin nests the record under the event key in the payload sent to the HEC. It will also append the time of the record to a top level time key.

If you would like to customize any of the Splunk event metadata, such as the host or target index, you can set Splunk_Send_Raw On in the plugin configuration, and add the metadata as keys/values in the record. Note: with Splunk_Send_Raw enabled, you are responsible for creating and populating the event section of the payload.

For example, to add a custom index and hostname:

  1. [INPUT]
  2.     Name  cpu
  3.     Tag   cpu
  5. # nest the record under the 'event' key
  6. [FILTER]
  7.     Name nest
  8.     Match *
  9.     Operation nest
  10.     Wildcard *
  11.     Nest_under event
  13. # add event metadata
  14. [FILTER]
  15.     Name      modify
  16.     Match     *
  17.     Add index my-splunk-index
  18.     Add host  my-host
  20. [OUTPUT]
  21.     Name        splunk
  22.     Match       *
  23.     Host        127.0.0.1
  24.     Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
  25.     Splunk_Send_Raw On

This will create a payload that looks like:

  1. {
  2.     "time": "1535995058.003385189",
  3.     "index": "my-splunk-index",
  4.     "host": "my-host",
  5.     "event": {
  6.         "cpu_p":0.000000,
  7.         "user_p":0.000000,
  8.         "system_p":0.000000
  9.     }
  10. }

For more information on the Splunk HEC payload format and all event meatadata Splunk accepts, see here: http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC

Sending Raw Events

If the option splunk_send_raw has been enabled, the user must take care to put all log details in the event field, and only specify fields known to Splunk in the top level event, if there is a mismatch, Splunk will return a HTTP error 400.

Consider the following example:

splunk_send_raw off

  1. {"time": ..., "event": {"k1": "foo", "k2": "bar", "index": "applogs"}}

splunk_send_raw on

  1. {"time": .., "k1": "foo", "k2": "bar", "index": "applogs"}

For up to date information about the valid keys in the top level object, refer to the Splunk documentation:

http://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHEC

Splunk Metric Index

With Splunk version 8.0> you can also use the Fluent Bit Splunk output plugin to send data to metric indices. This allows you to perform visualizations, metric queries, and analysis with other metrics you may be collecting. This is based off of Splunk 8.0 support of multi metric support via single JSON payload, more details can be found on Splunk’s documentation page

Sending to a Splunk Metric index requires the use of Splunk_send_raw option being enabled and formatting the message properly. This includes three specific operations

  • Nest metric events under a “fields” property
  • Add metric_name: to all metrics
  • Add index, source, sourcetype as fields in the message

Example Configuration

The following configuration gathers CPU metrics, nests the appropriate field, adds the required identifiers and then sends to Splunk.

  1. [INPUT]
  2.     name cpu
  3.     tag cpu
  5. # Move CPU metrics to be nested under "fields" and 
  6. # add the prefix "metric_name:" to all metrics
  7. # NOTE: you can change Wildcard field to only select metric fields    
  8. [FILTER]
  9.     Name nest
  10.     Match cpu
  11.     Wildcard *
  12.     Operation nest
  13.     Nest_under fields
  14.     Add_Prefix metric_name:
  16. # Add index, source, sourcetype
  17. [FILTER]
  18.     Name    modify
  19.     Match   cpu
  20.     Set index cpu-metrics 
  21.     Set source fluent-bit
  22.     Set sourcetype custom
  24. # ensure splunk_send_raw is on
  25. [OUTPUT]
  26.     name splunk 
  27.     match *
  28.     host <HOST>
  29.     port 8088
  30.     splunk_send_raw on
  31.     splunk_token f9bd5bdb-c0b2-4a83-bcff-9625e5e908db 
  32.     tls on
  33.     tls.verify off