Configuring Data Prepper

This is the reference for Data Prepper configuration files (data-prepper-config.yaml). These are general Data Prepper configurations independent from pipelines.

Data Prepper server options

OptionRequiredTypeDescription
sslNoBooleanIndicates whether TLS should be used for server APIs. Defaults to true.
keyStoreFilePathNoStringPath to a .jks or .p12 keystore file. Required if ssl is true.
keyStorePasswordNoStringPassword for keystore. Optional, defaults to empty string.
privateKeyPasswordNoStringPassword for private key within keystore. Optional, defaults to empty string.
serverPortNoIntegerPort number to use for server APIs. Defaults to 4900.
metricRegistriesNoListMetrics registries for publishing the generated metrics. Currently supports Prometheus and CloudWatch. Defaults to Prometheus.
metricTagsNoMapKey-value pairs as common metric tags to metric registries. The maximum number of pairs is three. Note that serviceName is a reserved tag key with DataPrepper as default tag value. Alternatively, administrators can set this value through the environment variable DATAPREPPER_SERVICE_NAME. If serviceName is defined in metricTags, that value overwrites those set through the above methods.
authenticationNoObjectAuthentication configuration. Valid option is http_basic with username and password properties. If not defined, the server does not perform authentication.
processorShutdownTimeoutNoDurationTime given to processors to clear any in-flight data and gracefully shut down. Default is 30s.
sinkShutdownTimeoutNoDurationTime given to sinks to clear any in-flight data and gracefully shut down. Default is 30s.
peer_forwarderNoObjectPeer forwarder configurations. See Peer forwarder options for more details.

Peer forwarder options

The following section details various configuration options for peer forwarder.

General options for peer forwarding

OptionRequiredTypeDescription
portNoIntegerThe peer forwarding server port. Valid options are between 0 and 65535. Defaults is 4994.
request_timeoutNoIntegerRequest timeout for the peer forwarder HTTP server in milliseconds. Default is 10000.
server_thread_countNoIntegerNumber of threads used by the peer forwarder server. Default is 200.
client_thread_countNoIntegerNumber of threads used by the peer forwarder client. Default is 200.
max_connection_countNoIntegerMaximum number of open connections for the peer forwarder server. Default is 500.
max_pending_requestsNoIntegerMaximum number of allowed tasks in ScheduledThreadPool work queue. Default is 1024.
discovery_modeNoStringPeer discovery mode to use. Valid options are local_node, static, dns, or aws_cloud_map. Defaults to local_node, which processes events locally.
static_endpointsConditionallyListA list containing endpoints of all Data Prepper instances. Required if discovery_mode is set to static.
domain_nameConditionallyStringA single domain name to query DNS against. Typically, used by creating multiple DNS A Records for the same domain. Required if discovery_mode is set to dns.
aws_cloud_map_namespace_nameConditionallyStringCloud Map namespace when using AWS Cloud Map service discovery. Required if discovery_mode is set to aws_cloud_map.
aws_cloud_map_service_nameConditionallyStringCloud Map service name when using AWS Cloud Map service discovery. Required if discovery_mode is set to aws_cloud_map.
aws_cloud_map_query_parametersNoMapKey-value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned.
buffer_sizeNoIntegerMax number of unchecked records the buffer accepts. Number of unchecked records is the sum of the number of records written into the buffer and the num of in-flight records not yet checked by the Checkpointing API. Default is 512.
batch_sizeNoIntegerMax number of records the buffer returns on read. Default is 48.
aws_regionConditionallyStringAWS region to use ACM, S3 or AWS Cloud Map. Required if use_acm_certificate_for_ssl is set to true or ssl_certificate_file and ssl_key_file is AWS S3 path or discovery_mode is set to aws_cloud_map.
drain_timeoutNoDurationWait time for the peer forwarder to complete processing data before shutdown. Default is 10s.

TLS/SSL options for peer forwarder

OptionRequiredTypeDescription
sslNoBooleanEnables TLS/SSL. Default is true.
ssl_certificate_fileConditionallyStringSSL certificate chain file path or AWS S3 path. S3 path example s3://<bucketName>/<path>. Required if ssl is true and use_acm_certificate_for_ssl is false. Defaults to config/default_certificate.pem which is the default certificate file. Read more about how the certificate file is generated here.
ssl_key_fileConditionallyStringSSL key file path or AWS S3 path. S3 path example s3://<bucketName>/<path>. Required if ssl is true and use_acm_certificate_for_ssl is false. Defaults to config/default_private_key.pem which is the default private key file. Read more about how the default private key file is generated here.
ssl_insecure_disable_verificationNoBooleanDisables the verification of server’s TLS certificate chain. Default is false.
ssl_fingerprint_verification_onlyNoBooleanDisables the verification of server’s TLS certificate chain and instead verifies only the certificate fingerprint. Default is false.
use_acm_certificate_for_sslNoBooleanEnables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is false.
acm_certificate_arnConditionallyStringACM certificate ARN. The ACM certificate takes preference over S3 or a local file system certificate. Required if use_acm_certificate_for_ssl is set to true.
acm_private_key_passwordNoStringACM private key password that decrypts the private key. If not provided, Data Prepper generates a random password.
acm_certificate_timeout_millisNoIntegerTimeout in milliseconds for ACM to get certificates. Default is 120000.
aws_regionConditionallyStringAWS region to use ACM, S3 or AWS Cloud Map. Required if use_acm_certificate_for_ssl is set to true or ssl_certificate_file and ssl_key_file is AWS S3 path or discovery_mode is set to aws_cloud_map.

Authentication options for peer forwarder

OptionRequiredTypeDescription
authenticationNoMapAuthentication method to use. Valid options are mutual_tls (use mTLS) or unauthenticated (no authentication). Default is unauthenticated.

Configuring Data Prepper - 图1