Configuration

Prometheus is configured via command-line flags and a configuration file. Whilethe command-line flags configure immutable system parameters (such as storagelocations, amount of data to keep on disk and in memory, etc.), theconfiguration file defines everything related to scraping jobs and theirinstances, as well aswhich rule files to load.

To view all available command-line flags, run ./prometheus -h.

Prometheus can reload its configuration at runtime. If the new configurationis not well-formed, the changes will not be applied.A configuration reload is triggered by sending a SIGHUP to the Prometheus process orsending a HTTP POST request to the /-/reload endpoint (when the —web.enable-lifecycle flag is enabled).This will also reload any configured rule files.

Configuration file

To specify which configuration file to load, use the —config.file flag.

The file is written in YAML format,defined by the scheme described below.Brackets indicate that a parameter is optional. For non-list parameters thevalue is set to the specified default.

Generic placeholders are defined as follows:

  • <boolean>: a boolean that can take the values true or false
  • <duration>: a duration matching the regular expression [0-9]+(ms|[smhdwy])
  • <labelname>: a string matching the regular expression [a-zA-Z][a-zA-Z0-9]*
  • <labelvalue>: a string of unicode characters
  • <filename>: a valid path in the current working directory
  • <host>: a valid string consisting of a hostname or IP followed by an optional port number
  • <path>: a valid URL path
  • <scheme>: a string that can take the values http or https
  • <string>: a regular string
  • <secret>: a regular string that is a secret, such as a password
  • <tmpl_string>: a string which is template-expanded before usageThe other placeholders are specified separately.

A valid example file can be found here.

The global configuration specifies parameters that are valid in all other configurationcontexts. They also serve as defaults for other configuration sections.

  1. global:
  2.   # How frequently to scrape targets by default.
  3.   [ scrape_interval: <duration> | default = 1m ]
  5.   # How long until a scrape request times out.
  6.   [ scrape_timeout: <duration> | default = 10s ]
  8.   # How frequently to evaluate rules.
  9.   [ evaluation_interval: <duration> | default = 1m ]
  11.   # The labels to add to any time series or alerts when communicating with
  12.   # external systems (federation, remote storage, Alertmanager).
  13.   external_labels:
  14.     [ <labelname>: <labelvalue> ... ]
  16. # Rule files specifies a list of globs. Rules and alerts are read from
  17. # all matching files.
  18. rule_files:
  19.   [ - <filepath_glob> ... ]
  21. # A list of scrape configurations.
  22. scrape_configs:
  23.   [ - <scrape_config> ... ]
  25. # Alerting specifies settings related to the Alertmanager.
  26. alerting:
  27.   alert_relabel_configs:
  28.     [ - <relabel_config> ... ]
  29.   alertmanagers:
  30.     [ - <alertmanager_config> ... ]
  32. # Settings related to the remote write feature.
  33. remote_write:
  34.   [ - <remote_write> ... ]
  36. # Settings related to the remote read feature.
  37. remote_read:
  38.   [ - <remote_read> ... ]

<scrape_config>

A scrape_config section specifies a set of targets and parameters describing howto scrape them. In the general case, one scrape configuration specifies a singlejob. In advanced configurations, this may change.

Targets may be statically configured via the static_configs parameter ordynamically discovered using one of the supported service-discovery mechanisms.

Additionally, relabel_configs allow advanced modifications to anytarget and its labels before scraping.

  1. # The job name assigned to scraped metrics by default.
  2. job_name: <job_name>
  4. # How frequently to scrape targets from this job.
  5. [ scrape_interval: <duration> | default = <global_config.scrape_interval> ]
  7. # Per-scrape timeout when scraping this job.
  8. [ scrape_timeout: <duration> | default = <global_config.scrape_timeout> ]
  10. # The HTTP resource path on which to fetch metrics from targets.
  11. [ metrics_path: <path> | default = /metrics ]
  13. # honor_labels controls how Prometheus handles conflicts between labels that are
  14. # already present in scraped data and labels that Prometheus would attach
  15. # server-side ("job" and "instance" labels, manually configured target
  16. # labels, and labels generated by service discovery implementations).
  17. #
  18. # If honor_labels is set to "true", label conflicts are resolved by keeping label
  19. # values from the scraped data and ignoring the conflicting server-side labels.
  20. #
  21. # If honor_labels is set to "false", label conflicts are resolved by renaming
  22. # conflicting labels in the scraped data to "exported_<original-label>" (for
  23. # example "exported_instance", "exported_job") and then attaching server-side
  24. # labels.
  25. #
  26. # Setting honor_labels to "true" is useful for use cases such as federation and
  27. # scraping the Pushgateway, where all labels specified in the target should be
  28. # preserved.
  29. #
  30. # Note that any globally configured "external_labels" are unaffected by this
  31. # setting. In communication with external systems, they are always applied only
  32. # when a time series does not have a given label yet and are ignored otherwise.
  33. [ honor_labels: <boolean> | default = false ]
  35. # honor_timestamps controls whether Prometheus respects the timestamps present
  36. # in scraped data.
  37. #
  38. # If honor_timestamps is set to "true", the timestamps of the metrics exposed
  39. # by the target will be used.
  40. #
  41. # If honor_timestamps is set to "false", the timestamps of the metrics exposed
  42. # by the target will be ignored.
  43. [ honor_timestamps: <boolean> | default = true ]
  45. # Configures the protocol scheme used for requests.
  46. [ scheme: <scheme> | default = http ]
  48. # Optional HTTP URL parameters.
  49. params:
  50.   [ <string>: [<string>, ...] ]
  52. # Sets the `Authorization` header on every scrape request with the
  53. # configured username and password.
  54. # password and password_file are mutually exclusive.
  55. basic_auth:
  56.   [ username: <string> ]
  57.   [ password: <secret> ]
  58.   [ password_file: <string> ]
  60. # Sets the `Authorization` header on every scrape request with
  61. # the configured bearer token. It is mutually exclusive with `bearer_token_file`.
  62. [ bearer_token: <secret> ]
  64. # Sets the `Authorization` header on every scrape request with the bearer token
  65. # read from the configured file. It is mutually exclusive with `bearer_token`.
  66. [ bearer_token_file: /path/to/bearer/token/file ]
  68. # Configures the scrape request's TLS settings.
  69. tls_config:
  70.   [ <tls_config> ]
  72. # Optional proxy URL.
  73. [ proxy_url: <string> ]
  75. # List of Azure service discovery configurations.
  76. azure_sd_configs:
  77.   [ - <azure_sd_config> ... ]
  79. # List of Consul service discovery configurations.
  80. consul_sd_configs:
  81.   [ - <consul_sd_config> ... ]
  83. # List of DNS service discovery configurations.
  84. dns_sd_configs:
  85.   [ - <dns_sd_config> ... ]
  87. # List of EC2 service discovery configurations.
  88. ec2_sd_configs:
  89.   [ - <ec2_sd_config> ... ]
  91. # List of OpenStack service discovery configurations.
  92. openstack_sd_configs:
  93.   [ - <openstack_sd_config> ... ]
  95. # List of file service discovery configurations.
  96. file_sd_configs:
  97.   [ - <file_sd_config> ... ]
  99. # List of GCE service discovery configurations.
  100. gce_sd_configs:
  101.   [ - <gce_sd_config> ... ]
  103. # List of Kubernetes service discovery configurations.
  104. kubernetes_sd_configs:
  105.   [ - <kubernetes_sd_config> ... ]
  107. # List of Marathon service discovery configurations.
  108. marathon_sd_configs:
  109.   [ - <marathon_sd_config> ... ]
  111. # List of AirBnB's Nerve service discovery configurations.
  112. nerve_sd_configs:
  113.   [ - <nerve_sd_config> ... ]
  115. # List of Zookeeper Serverset service discovery configurations.
  116. serverset_sd_configs:
  117.   [ - <serverset_sd_config> ... ]
  119. # List of Triton service discovery configurations.
  120. triton_sd_configs:
  121.   [ - <triton_sd_config> ... ]
  123. # List of labeled statically configured targets for this job.
  124. static_configs:
  125.   [ - <static_config> ... ]
  127. # List of target relabel configurations.
  128. relabel_configs:
  129.   [ - <relabel_config> ... ]
  131. # List of metric relabel configurations.
  132. metric_relabel_configs:
  133.   [ - <relabel_config> ... ]
  135. # Per-scrape limit on number of scraped samples that will be accepted.
  136. # If more than this number of samples are present after metric relabelling
  137. # the entire scrape will be treated as failed. 0 means no limit.
  138. [ sample_limit: <int> | default = 0 ]

Where <job_name> must be unique across all scrape configurations.

<tls_config>

A tls_config allows configuring TLS connections.

  1. # CA certificate to validate API server certificate with.
  2. [ ca_file: <filename> ]
  4. # Certificate and key files for client cert authentication to the server.
  5. [ cert_file: <filename> ]
  6. [ key_file: <filename> ]
  8. # ServerName extension to indicate the name of the server.
  9. # https://tools.ietf.org/html/rfc4366#section-3.1
  10. [ server_name: <string> ]
  12. # Disable validation of the server certificate.
  13. [ insecure_skip_verify: <boolean> ]

<azure_sd_config>

Azure SD configurations allow retrieving scrape targets from Azure VMs.

The following meta labels are available on targets during relabeling:

  • __meta_azure_machine_id: the machine ID
  • __meta_azure_machine_location: the location the machine runs in
  • __meta_azure_machine_name: the machine name
  • __meta_azure_machine_os_type: the machine operating system
  • __meta_azure_machine_private_ip: the machine's private IP
  • __meta_azure_machine_public_ip: the machine's public IP if it exists
  • __meta_azure_machine_resource_group: the machine's resource group
  • _meta_azure_machine_tag<tagname>: each tag value of the machine
  • __meta_azure_machine_scale_set: the name of the scale set which the vm is part of (this value is only set if you are using a scale set)
  • __meta_azure_subscription_id: the subscription ID
  • __meta_azure_tenant_id: the tenant IDSee below for the configuration options for Azure discovery:
  1. # The information to access the Azure API.
  2. # The Azure environment.
  3. [ environment: <string> | default = AzurePublicCloud ]
  5. # The authentication method, either OAuth or ManagedIdentity.
  6. # See https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview
  7. [ authentication_method: <string> | default = OAuth]
  8. # The subscription ID. Always required.
  9. subscription_id: <string>
  10. # Optional tenant ID. Only required with authentication_method OAuth.
  11. [ tenant_id: <string> ]
  12. # Optional client ID. Only required with authentication_method OAuth.
  13. [ client_id: <string> ]
  14. # Optional client secret. Only required with authentication_method OAuth.
  15. [ client_secret: <secret> ]
  17. # Refresh interval to re-read the instance list.
  18. [ refresh_interval: <duration> | default = 300s ]
  20. # The port to scrape metrics from. If using the public IP address, this must
  21. # instead be specified in the relabeling rule.
  22. [ port: <int> | default = 80 ]

<consul_sd_config>

Consul SD configurations allow retrieving scrape targets from Consul'sCatalog API.

The following meta labels are available on targets during relabeling:

  • __meta_consul_address: the address of the target
  • __meta_consul_dc: the datacenter name for the target
  • _meta_consul_tagged_address<key>: each node tagged address key value of the target
  • _meta_consul_metadata<key>: each node metadata key value of the target
  • __meta_consul_node: the node name defined for the target
  • __meta_consul_service_address: the service address of the target
  • __meta_consul_service_id: the service ID of the target
  • _meta_consul_service_metadata<key>: each service metadata key value of the target
  • __meta_consul_service_port: the service port of the target
  • __meta_consul_service: the name of the service the target belongs to
  • __meta_consul_tags: the list of tags of the target joined by the tag separator
  1. # The information to access the Consul API. It is to be defined
  2. # as the Consul documentation requires.
  3. [ server: <host> | default = "localhost:8500" ]
  4. [ token: <secret> ]
  5. [ datacenter: <string> ]
  6. [ scheme: <string> | default = "http" ]
  7. [ username: <string> ]
  8. [ password: <secret> ]
  10. tls_config:
  11.   [ <tls_config> ]
  13. # A list of services for which targets are retrieved. If omitted, all services
  14. # are scraped.
  15. services:
  16.   [ - <string> ]
  18. # See https://www.consul.io/api/catalog.html#list-nodes-for-service to know more
  19. # about the possible filters that can be used.
  21. # An optional list of tags used to filter nodes for a given service. Services must contain all tags in the list.
  22. tags:
  23.   [ - <string> ]
  25. # Node metadata used to filter nodes for a given service.
  26. [ node_meta:
  27.   [ <name>: <value> ... ] ]
  29. # The string by which Consul tags are joined into the tag label.
  30. [ tag_separator: <string> | default = , ]
  32. # Allow stale Consul results (see https://www.consul.io/api/features/consistency.html). Will reduce load on Consul.
  33. [ allow_stale: <bool> ]
  35. # The time after which the provided names are refreshed.
  36. # On large setup it might be a good idea to increase this value because the catalog will change all the time.
  37. [ refresh_interval: <duration> | default = 30s ]

Note that the IP number and port used to scrape the targets is assembled as<meta_consul_address>:<metaconsulservice_port>. However, in someConsul setups, the relevant address is in meta_consul_service_address.In those cases, you can use the relabelfeature to replace the special address label.

The relabeling phase is the preferred and more powerfulway to filter services or nodes for a service based on arbitrary labels. Forusers with thousands of services it can be more efficient to use the Consul APIdirectly which has basic support for filtering nodes (currently by nodemetadata and a single tag).

<dns_sd_config>

A DNS-based service discovery configuration allows specifying a set of DNSdomain names which are periodically queried to discover a list of targets. TheDNS servers to be contacted are read from /etc/resolv.conf.

This service discovery method only supports basic DNS A, AAAA and SRV recordqueries, but not the advanced DNS-SD approach specified inRFC6763.

During the relabeling phase, the meta label__meta_dns_name is available on each target and is set to therecord name that produced the discovered target.

  1. # A list of DNS domain names to be queried.
  2. names:
  3.   [ - <domain_name> ]
  5. # The type of DNS query to perform.
  6. [ type: <query_type> | default = 'SRV' ]
  8. # The port number used if the query type is not SRV.
  9. [ port: <number>]
  11. # The time after which the provided names are refreshed.
  12. [ refresh_interval: <duration> | default = 30s ]

Where <domain_name> is a valid DNS domain name.Where <query_type> is SRV, A, or AAAA.

<ec2_sd_config>

EC2 SD configurations allow retrieving scrape targets from AWS EC2instances. The private IP address is used by default, but may be changed tothe public IP address with relabeling.

The following meta labels are available on targets during relabeling:

  • __meta_ec2_availability_zone: the availability zone in which the instance is running
  • __meta_ec2_instance_id: the EC2 instance ID
  • __meta_ec2_instance_state: the state of the EC2 instance
  • __meta_ec2_instance_type: the type of the EC2 instance
  • __meta_ec2_owner_id: the ID of the AWS account that owns the EC2 instance
  • __meta_ec2_platform: the Operating System platform, set to 'windows' on Windows servers, absent otherwise
  • __meta_ec2_primary_subnet_id: the subnet ID of the primary network interface, if available
  • __meta_ec2_private_dns_name: the private DNS name of the instance, if available
  • __meta_ec2_private_ip: the private IP address of the instance, if present
  • __meta_ec2_public_dns_name: the public DNS name of the instance, if available
  • __meta_ec2_public_ip: the public IP address of the instance, if available
  • __meta_ec2_subnet_id: comma separated list of subnets IDs in which the instance is running, if available
  • _meta_ec2_tag<tagkey>: each tag value of the instance
  • __meta_ec2_vpc_id: the ID of the VPC in which the instance is running, if availableSee below for the configuration options for EC2 discovery:
  1. # The information to access the EC2 API.
  3. # The AWS region. If blank, the region from the instance metadata is used.
  4. [ region: <string> ]
  6. # Custom endpoint to be used.
  7. [ endpoint: <string> ]
  9. # The AWS API keys. If blank, the environment variables `AWS_ACCESS_KEY_ID`
  10. # and `AWS_SECRET_ACCESS_KEY` are used.
  11. [ access_key: <string> ]
  12. [ secret_key: <secret> ]
  13. # Named AWS profile used to connect to the API.
  14. [ profile: <string> ]
  16. # AWS Role ARN, an alternative to using AWS API keys.
  17. [ role_arn: <string> ]
  19. # Refresh interval to re-read the instance list.
  20. [ refresh_interval: <duration> | default = 60s ]
  22. # The port to scrape metrics from. If using the public IP address, this must
  23. # instead be specified in the relabeling rule.
  24. [ port: <int> | default = 80 ]
  26. # Filters can be used optionally to filter the instance list by other criteria.
  27. # Available filter criteria can be found here:
  28. # https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html
  29. # Filter API documentation: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_Filter.html
  30. filters:
  31.   [ - name: <string>
  32.       values: <string>, [...] ]

The relabeling phase is the preferred and more powerfulway to filter targets based on arbitrary labels. For users with thousands ofinstances it can be more efficient to use the EC2 API directly which hassupport for filtering instances.

<openstack_sd_config>

OpenStack SD configurations allow retrieving scrape targets from OpenStack Novainstances.

One of the following <openstack_role> types can be configured to discover targets:

hypervisor

The hypervisor role discovers one target per Nova hypervisor node. The targetaddress defaults to the host_ip attribute of the hypervisor.

The following meta labels are available on targets during relabeling:

  • __meta_openstack_hypervisor_host_ip: the hypervisor node's IP address.
  • __meta_openstack_hypervisor_name: the hypervisor node's name.
  • __meta_openstack_hypervisor_state: the hypervisor node's state.
  • __meta_openstack_hypervisor_status: the hypervisor node's status.
  • __meta_openstack_hypervisor_type: the hypervisor node's type.

instance

The instance role discovers one target per network interface of Novainstance. The target address defaults to the private IP address of the networkinterface.

The following meta labels are available on targets during relabeling:

  • __meta_openstack_address_pool: the pool of the private IP.
  • __meta_openstack_instance_flavor: the flavor of the OpenStack instance.
  • __meta_openstack_instance_id: the OpenStack instance ID.
  • __meta_openstack_instance_name: the OpenStack instance name.
  • __meta_openstack_instance_status: the status of the OpenStack instance.
  • __meta_openstack_private_ip: the private IP of the OpenStack instance.
  • __meta_openstack_project_id: the project (tenant) owning this instance.
  • __meta_openstack_public_ip: the public IP of the OpenStack instance.
  • _meta_openstack_tag<tagkey>: each tag value of the instance.
  • __meta_openstack_user_id: the user account owning the tenant.See below for the configuration options for OpenStack discovery:
  1. # The information to access the OpenStack API.
  3. # The OpenStack role of entities that should be discovered.
  4. role: <openstack_role>
  6. # The OpenStack Region.
  7. region: <string>
  9. # identity_endpoint specifies the HTTP endpoint that is required to work with
  10. # the Identity API of the appropriate version. While it's ultimately needed by
  11. # all of the identity services, it will often be populated by a provider-level
  12. # function.
  13. [ identity_endpoint: <string> ]
  15. # username is required if using Identity V2 API. Consult with your provider's
  16. # control panel to discover your account's username. In Identity V3, either
  17. # userid or a combination of username and domain_id or domain_name are needed.
  18. [ username: <string> ]
  19. [ userid: <string> ]
  21. # password for the Identity V2 and V3 APIs. Consult with your provider's
  22. # control panel to discover your account's preferred method of authentication.
  23. [ password: <secret> ]
  25. # At most one of domain_id and domain_name must be provided if using username
  26. # with Identity V3. Otherwise, either are optional.
  27. [ domain_name: <string> ]
  28. [ domain_id: <string> ]
  30. # The project_id and project_name fields are optional for the Identity V2 API.
  31. # Some providers allow you to specify a project_name instead of the project_id.
  32. # Some require both. Your provider's authentication policies will determine
  33. # how these fields influence authentication.
  34. [ project_name: <string> ]
  35. [ project_id: <string> ]
  37. # The application_credential_id or application_credential_name fields are
  38. # required if using an application credential to authenticate. Some providers
  39. # allow you to create an application credential to authenticate rather than a
  40. # password.
  41. [ application_credential_name: <string> ]
  42. [ application_credential_id: <string> ]
  44. # The application_credential_secret field is required if using an application
  45. # credential to authenticate.
  46. [ application_credential_secret: <secret> ]
  48. # Whether the service discovery should list all instances for all projects.
  49. # It is only relevant for the 'instance' role and usually requires admin permissions.
  50. [ all_tenants: <boolean> | default: false ]
  52. # Refresh interval to re-read the instance list.
  53. [ refresh_interval: <duration> | default = 60s ]
  55. # The port to scrape metrics from. If using the public IP address, this must
  56. # instead be specified in the relabeling rule.
  57. [ port: <int> | default = 80 ]
  59. # TLS configuration.
  60. tls_config:
  61.   [ <tls_config> ]

<file_sd_config>

File-based service discovery provides a more generic way to configure static targetsand serves as an interface to plug in custom service discovery mechanisms.

It reads a set of files containing a list of zero or more<static_config>s. Changes to all defined files are detected via disk watchesand applied immediately. Files may be provided in YAML or JSON format. Onlychanges resulting in well-formed target groups are applied.

The JSON file must contain a list of static configs, using this format:

  1. [
  2.   {
  3.     "targets": [ "<host>", ... ],
  4.     "labels": {
  5.       "<labelname>": "<labelvalue>", ...
  6.     }
  7.   },
  8.   ...
  9. ]

As a fallback, the file contents are also re-read periodically at the specifiedrefresh interval.

Each target has a meta label __meta_filepath during therelabeling phase. Its value is set to thefilepath from which the target was extracted.

There is a list ofintegrations with thisdiscovery mechanism.

  1. # Patterns for files from which target groups are extracted.
  2. files:
  3.   [ - <filename_pattern> ... ]
  5. # Refresh interval to re-read the files.
  6. [ refresh_interval: <duration> | default = 5m ]

Where <filenamepattern> may be a path ending in .json, .yml or .yaml. The last path segmentmay contain a single * that matches any character sequence, e.g. my/path/tg*.json.

<gce_sd_config>

GCE SD configurations allow retrieving scrape targets from GCP GCE instances.The private IP address is used by default, but may be changed to the public IPaddress with relabeling.

The following meta labels are available on targets during relabeling:

  • __meta_gce_instance_id: the numeric id of the instance
  • __meta_gce_instance_name: the name of the instance
  • _meta_gce_label<name>: each GCE label of the instance
  • __meta_gce_machine_type: full or partial URL of the machine type of the instance
  • _meta_gce_metadata<name>: each metadata item of the instance
  • __meta_gce_network: the network URL of the instance
  • __meta_gce_private_ip: the private IP address of the instance
  • __meta_gce_project: the GCP project in which the instance is running
  • __meta_gce_public_ip: the public IP address of the instance, if present
  • __meta_gce_subnetwork: the subnetwork URL of the instance
  • __meta_gce_tags: comma separated list of instance tags
  • __meta_gce_zone: the GCE zone URL in which the instance is runningSee below for the configuration options for GCE discovery:
  1. # The information to access the GCE API.
  3. # The GCP Project
  4. project: <string>
  6. # The zone of the scrape targets. If you need multiple zones use multiple
  7. # gce_sd_configs.
  8. zone: <string>
  10. # Filter can be used optionally to filter the instance list by other criteria
  11. # Syntax of this filter string is described here in the filter query parameter section:
  12. # https://cloud.google.com/compute/docs/reference/latest/instances/list
  13. [ filter: <string> ]
  15. # Refresh interval to re-read the instance list
  16. [ refresh_interval: <duration> | default = 60s ]
  18. # The port to scrape metrics from. If using the public IP address, this must
  19. # instead be specified in the relabeling rule.
  20. [ port: <int> | default = 80 ]
  22. # The tag separator is used to separate the tags on concatenation
  23. [ tag_separator: <string> | default = , ]

Credentials are discovered by the Google Cloud SDK default client by lookingin the following places, preferring the first location found:

  • a JSON file specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  • a JSON file in the well-known path $HOME/.config/gcloud/application_default_credentials.json
  • fetched from the GCE metadata serverIf Prometheus is running within GCE, the service account associated with theinstance it is running on should have at least read-only permissions to thecompute resources. If running outside of GCE make sure to create an appropriateservice account and place the credential file in one of the expected locations.

<kubernetes_sd_config>

Kubernetes SD configurations allow retrieving scrape targets fromKubernetes' REST API and always staying synchronized withthe cluster state.

One of the following role types can be configured to discover targets:

node

The node role discovers one target per cluster node with the address defaultingto the Kubelet's HTTP port.The target address defaults to the first existing address of the Kubernetesnode object in the address type order of NodeInternalIP, NodeExternalIP,NodeLegacyHostIP, and NodeHostName.

Available meta labels:

  • __meta_kubernetes_node_name: The name of the node object.
  • _meta_kubernetes_node_label<labelname>: Each label from the node object.
  • _meta_kubernetes_node_labelpresent<labelname>: true for each label from the node object.
  • _meta_kubernetes_node_annotation<annotationname>: Each annotation from the node object.
  • _meta_kubernetes_node_annotationpresent<annotationname>: true for each annotation from the node object.
  • _meta_kubernetes_node_address<address_type>: The first address for each node address type, if it exists.In addition, the instance label for the node will be set to the node nameas retrieved from the API server.

service

The service role discovers a target for each service port for each service.This is generally useful for blackbox monitoring of a service.The address will be set to the Kubernetes DNS name of the service and respectiveservice port.

Available meta labels:

  • __meta_kubernetes_namespace: The namespace of the service object.
  • _meta_kubernetes_service_annotation<annotationname>: Each annotation from the service object.
  • _meta_kubernetes_service_annotationpresent<annotationname>: "true" for each annotation of the service object.
  • __meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
  • __meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
  • _meta_kubernetes_service_label<labelname>: Each label from the service object.
  • _meta_kubernetes_service_labelpresent<labelname>: true for each label of the service object.
  • __meta_kubernetes_service_name: The name of the service object.
  • __meta_kubernetes_service_port_name: Name of the service port for the target.
  • __meta_kubernetes_service_port_protocol: Protocol of the service port for the target.

pod

The pod role discovers all pods and exposes their containers as targets. For each declaredport of a container, a single target is generated. If a container has no specified ports,a port-free target per container is created for manually adding a port via relabeling.

Available meta labels:

  • __meta_kubernetes_namespace: The namespace of the pod object.
  • __meta_kubernetes_pod_name: The name of the pod object.
  • __meta_kubernetes_pod_ip: The pod IP of the pod object.
  • _meta_kubernetes_pod_label<labelname>: Each label from the pod object.
  • _meta_kubernetes_pod_labelpresent<labelname>: truefor each label from the pod object.
  • _meta_kubernetes_pod_annotation<annotationname>: Each annotation from the pod object.
  • _meta_kubernetes_pod_annotationpresent<annotationname>: true for each annotation from the pod object.
  • __meta_kubernetes_pod_container_init: true if the container is an InitContainer
  • __meta_kubernetes_pod_container_name: Name of the container the target address points to.
  • __meta_kubernetes_pod_container_port_name: Name of the container port.
  • __meta_kubernetes_pod_container_port_number: Number of the container port.
  • __meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
  • __meta_kubernetes_pod_ready: Set to true or false for the pod's ready state.
  • __meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknownin the lifecycle.
  • __meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
  • __meta_kubernetes_pod_host_ip: The current host IP of the pod object.
  • __meta_kubernetes_pod_uid: The UID of the pod object.
  • __meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
  • __meta_kubernetes_pod_controller_name: Name of the pod controller.

endpoints

The endpoints role discovers targets from listed endpoints of a service. For each endpointaddress one target is discovered per port. If the endpoint is backed by a pod, alladditional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

Available meta labels:

  • __meta_kubernetes_namespace: The namespace of the endpoints object.
  • __meta_kubernetes_endpoints_name: The names of the endpoints object.
  • For all targets discovered directly from the endpoints list (those not additionally inferredfrom underlying pods), the following labels are attached:
    • __meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
    • __meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
    • __meta_kubernetes_endpoint_ready: Set to true or false for the endpoint's ready state.
    • __meta_kubernetes_endpoint_port_name: Name of the endpoint port.
    • __meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
    • __meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
    • __meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
  • If the endpoints belong to a service, all labels of the role: service discovery are attached.
  • For all targets backed by a pod, all labels of the role: pod discovery are attached.

ingress

The ingress role discovers a target for each path of each ingress.This is generally useful for blackbox monitoring of an ingress.The address will be set to the host specified in the ingress spec.

Available meta labels:

  • __meta_kubernetes_namespace: The namespace of the ingress object.
  • __meta_kubernetes_ingress_name: The name of the ingress object.
  • _meta_kubernetes_ingress_label<labelname>: Each label from the ingress object.
  • _meta_kubernetes_ingress_labelpresent<labelname>: true for each label from the ingress object.
  • _meta_kubernetes_ingress_annotation<annotationname>: Each annotation from the ingress object.
  • _meta_kubernetes_ingress_annotationpresent<annotationname>: true for each annotation from the ingress object.
  • __meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLSconfig is set. Defaults to http.
  • __meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.See below for the configuration options for Kubernetes discovery:
  1. # The information to access the Kubernetes API.
  3. # The API server addresses. If left empty, Prometheus is assumed to run inside
  4. # of the cluster and will discover API servers automatically and use the pod's
  5. # CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
  6. [ api_server: <host> ]
  8. # The Kubernetes role of entities that should be discovered.
  9. role: <role>
  11. # Optional authentication information used to authenticate to the API server.
  12. # Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
  13. # mutually exclusive.
  14. # password and password_file are mutually exclusive.
  16. # Optional HTTP basic authentication information.
  17. basic_auth:
  18.   [ username: <string> ]
  19.   [ password: <secret> ]
  20.   [ password_file: <string> ]
  22. # Optional bearer token authentication information.
  23. [ bearer_token: <secret> ]
  25. # Optional bearer token file authentication information.
  26. [ bearer_token_file: <filename> ]
  28. # Optional proxy URL.
  29. [ proxy_url: <string> ]
  31. # TLS configuration.
  32. tls_config:
  33.   [ <tls_config> ]
  35. # Optional namespace discovery. If omitted, all namespaces are used.
  36. namespaces:
  37.   names:
  38.     [ - <string> ]

Where <role> must be endpoints, service, pod, node, oringress.

See this example Prometheus configuration filefor a detailed example of configuring Prometheus for Kubernetes.

You may wish to check out the 3rd party Prometheus Operator,which automates the Prometheus setup on top of Kubernetes.

<marathon_sd_config>

Marathon SD configurations allow retrieving scrape targets using theMarathon REST API. Prometheuswill periodically check the REST endpoint for currently running tasks andcreate a target group for every app that has at least one healthy task.

The following meta labels are available on targets during relabeling:

  • __meta_marathon_app: the name of the app (with slashes replaced by dashes)
  • __meta_marathon_image: the name of the Docker image used (if available)
  • __meta_marathon_task: the ID of the Mesos task
  • _meta_marathon_app_label<labelname>: any Marathon labels attached to the app
  • _meta_marathon_port_definition_label<labelname>: the port definition labels
  • _meta_marathon_port_mapping_label<labelname>: the port mapping labels
  • __meta_marathon_port_index: the port index number (e.g. 1 for PORT1)See below for the configuration options for Marathon discovery:
  1. # List of URLs to be used to contact Marathon servers.
  2. # You need to provide at least one server URL.
  3. servers:
  4.   - <string>
  6. # Polling interval
  7. [ refresh_interval: <duration> | default = 30s ]
  9. # Optional authentication information for token-based authentication
  10. # https://docs.mesosphere.com/1.11/security/ent/iam-api/#passing-an-authentication-token
  11. # It is mutually exclusive with `auth_token_file` and other authentication mechanisms.
  12. [ auth_token: <secret> ]
  14. # Optional authentication information for token-based authentication
  15. # https://docs.mesosphere.com/1.11/security/ent/iam-api/#passing-an-authentication-token
  16. # It is mutually exclusive with `auth_token` and other authentication mechanisms.
  17. [ auth_token_file: <filename> ]
  19. # Sets the `Authorization` header on every request with the
  20. # configured username and password.
  21. # This is mutually exclusive with other authentication mechanisms.
  22. # password and password_file are mutually exclusive.
  23. basic_auth:
  24.   [ username: <string> ]
  25.   [ password: <string> ]
  26.   [ password_file: <string> ]
  28. # Sets the `Authorization` header on every request with
  29. # the configured bearer token. It is mutually exclusive with `bearer_token_file` and other authentication mechanisms.
  30. # NOTE: The current version of DC/OS marathon (v1.11.0) does not support standard Bearer token authentication. Use `auth_token` instead.
  31. [ bearer_token: <string> ]
  33. # Sets the `Authorization` header on every request with the bearer token
  34. # read from the configured file. It is mutually exclusive with `bearer_token` and other authentication mechanisms.
  35. # NOTE: The current version of DC/OS marathon (v1.11.0) does not support standard Bearer token authentication. Use `auth_token_file` instead.
  36. [ bearer_token_file: /path/to/bearer/token/file ]
  38. # TLS configuration for connecting to marathon servers
  39. tls_config:
  40.   [ <tls_config> ]
  42. # Optional proxy URL.
  43. [ proxy_url: <string> ]

By default every app listed in Marathon will be scraped by Prometheus. If not allof your services provide Prometheus metrics, you can use a Marathon label andPrometheus relabeling to control which instances will actually be scraped.See the Prometheus marathon-sd configuration filefor a practical example on how to set up your Marathon app and your Prometheusconfiguration.

By default, all apps will show up as a single job in Prometheus (the one specifiedin the configuration file), which can also be changed using relabeling.

<nerve_sd_config>

Nerve SD configurations allow retrieving scrape targets from AirBnB's Nerve which are stored inZookeeper.

The following meta labels are available on targets during relabeling:

  • __meta_nerve_path: the full path to the endpoint node in Zookeeper
  • __meta_nerve_endpoint_host: the host of the endpoint
  • __meta_nerve_endpoint_port: the port of the endpoint
  • __meta_nerve_endpoint_name: the name of the endpoint
  1. # The Zookeeper servers.
  2. servers:
  3.   - <host>
  4. # Paths can point to a single service, or the root of a tree of services.
  5. paths:
  6.   - <string>
  7. [ timeout: <duration> | default = 10s ]

<serverset_sd_config>

Serverset SD configurations allow retrieving scrape targets from Serversets which arestored in Zookeeper. Serversets are commonlyused by Finagle andAurora.

The following meta labels are available on targets during relabeling:

  • __meta_serverset_path: the full path to the serverset member node in Zookeeper
  • __meta_serverset_endpoint_host: the host of the default endpoint
  • __meta_serverset_endpoint_port: the port of the default endpoint
  • _meta_serverset_endpoint_host<endpoint>: the host of the given endpoint
  • _meta_serverset_endpoint_port<endpoint>: the port of the given endpoint
  • __meta_serverset_shard: the shard number of the member
  • __meta_serverset_status: the status of the member
  1. # The Zookeeper servers.
  2. servers:
  3.   - <host>
  4. # Paths can point to a single serverset, or the root of a tree of serversets.
  5. paths:
  6.   - <string>
  7. [ timeout: <duration> | default = 10s ]

Serverset data must be in the JSON format, the Thrift format is not currently supported.

<triton_sd_config>

Triton SD configurations allow retrievingscrape targets from Container Monitordiscovery endpoints.

The following meta labels are available on targets during relabeling:

  • __meta_triton_groups: the list of groups belonging to the target joined by a comma separator
  • __meta_triton_machine_alias: the alias of the target container
  • __meta_triton_machine_brand: the brand of the target container
  • __meta_triton_machine_id: the UUID of the target container
  • __meta_triton_machine_image: the target containers image type
  • __meta_triton_server_id: the server UUID for the target container
  1. # The information to access the Triton discovery API.
  3. # The account to use for discovering new target containers.
  4. account: <string>
  6. # The DNS suffix which should be applied to target containers.
  7. dns_suffix: <string>
  9. # The Triton discovery endpoint (e.g. 'cmon.us-east-3b.triton.zone'). This is
  10. # often the same value as dns_suffix.
  11. endpoint: <string>
  13. # A list of groups for which targets are retrieved. If omitted, all containers
  14. # available to the requesting account are scraped.
  15. groups:
  16.   [ - <string> ... ]
  18. # The port to use for discovery and metric scraping.
  19. [ port: <int> | default = 9163 ]
  21. # The interval which should be used for refreshing target containers.
  22. [ refresh_interval: <duration> | default = 60s ]
  24. # The Triton discovery API version.
  25. [ version: <int> | default = 1 ]
  27. # TLS configuration.
  28. tls_config:
  29.   [ <tls_config> ]

<static_config>

A static_config allows specifying a list of targets and a common label setfor them. It is the canonical way to specify static targets in a scrapeconfiguration.

  1. # The targets specified by the static config.
  2. targets:
  3.   [ - '<host>' ]
  5. # Labels assigned to all metrics scraped from the targets.
  6. labels:
  7.   [ <labelname>: <labelvalue> ... ]

<relabel_config>

Relabeling is a powerful tool to dynamically rewrite the label set of a target beforeit gets scraped. Multiple relabeling steps can be configured per scrape configuration.They are applied to the label set of each target in order of their appearancein the configuration file.

Initially, aside from the configured per-target labels, a target's joblabel is set to the jobname value of the respective scrape configuration.The address label is set to the <host>:<port> address of the target.After relabeling, the instance label is set to the value of address by default ifit was not set during relabeling. The scheme and metrics_path labelsare set to the scheme and metrics path of the target respectively. The __param<name>label is set to the value of the first passed URL parameter called <name>.

Additional labels prefixed with _meta may be available during therelabeling phase. They are set by the service discovery mechanism that providedthe target and vary between mechanisms.

Labels starting with __ will be removed from the label set after targetrelabeling is completed.

If a relabeling step needs to store a label value only temporarily (as theinput to a subsequent relabeling step), use the __tmp label name prefix. Thisprefix is guaranteed to never be used by Prometheus itself.

  1. # The source labels select values from existing labels. Their content is concatenated
  2. # using the configured separator and matched against the configured regular expression
  3. # for the replace, keep, and drop actions.
  4. [ source_labels: '[' <labelname> [, ...] ']' ]
  6. # Separator placed between concatenated source label values.
  7. [ separator: <string> | default = ; ]
  9. # Label to which the resulting value is written in a replace action.
  10. # It is mandatory for replace actions. Regex capture groups are available.
  11. [ target_label: <labelname> ]
  13. # Regular expression against which the extracted value is matched.
  14. [ regex: <regex> | default = (.*) ]
  16. # Modulus to take of the hash of the source label values.
  17. [ modulus: <uint64> ]
  19. # Replacement value against which a regex replace is performed if the
  20. # regular expression matches. Regex capture groups are available.
  21. [ replacement: <string> | default = $1 ]
  23. # Action to perform based on regex matching.
  24. [ action: <relabel_action> | default = replace ]

<regex> is any validRE2 regular expression. It isrequired for the replace, keep, drop, labelmap,labeldrop and labelkeep actions. The regex isanchored on both ends. To un-anchor the regex, use .<regex>..

<relabel_action> determines the relabeling action to take:

  • replace: Match regex against the concatenated source_labels. Then, settarget_label to replacement, with match group references(${1}, ${2}, …) in replacement substituted by their value. If regexdoes not match, no replacement takes place.
  • keep: Drop targets for which regex does not match the concatenated source_labels.
  • drop: Drop targets for which regex matches the concatenated source_labels.
  • hashmod: Set target_label to the modulus of a hash of the concatenated source_labels.
  • labelmap: Match regex against all label names. Then copy the values of the matching labelsto label names given by replacement with match group references(${1}, ${2}, …) in replacement substituted by their value.
  • labeldrop: Match regex against all label names. Any label that matches will beremoved from the set of labels.
  • labelkeep: Match regex against all label names. Any label that does not match will beremoved from the set of labels.Care must be taken with labeldrop and labelkeep to ensure that metrics arestill uniquely labeled once the labels are removed.

<metric_relabel_configs>

Metric relabeling is applied to samples as the last step before ingestion. Ithas the same configuration format and actions as target relabeling. Metricrelabeling does not apply to automatically generated timeseries such as up.

One use for this is to blacklist time series that are too expensive to ingest.

<alert_relabel_configs>

Alert relabeling is applied to alerts before they are sent to the Alertmanager.It has the same configuration format and actions as target relabeling. Alertrelabeling is applied after external labels.

One use for this is ensuring a HA pair of Prometheus servers with differentexternal labels send identical alerts.

<alertmanager_config>

An alertmanager_config section specifies Alertmanager instances the Prometheusserver sends alerts to. It also provides parameters to configure how tocommunicate with these Alertmanagers.

Alertmanagers may be statically configured via the static_configs parameter ordynamically discovered using one of the supported service-discovery mechanisms.

Additionally, relabelconfigs allow selecting Alertmanagers from discoveredentities and provide advanced modifications to the used API path, which is exposedthrough the _alerts_path label.

  1. # Per-target Alertmanager timeout when pushing alerts.
  2. [ timeout: <duration> | default = 10s ]
  4. # The api version of Alertmanager.
  5. [ api_version: <version> | default = v1 ]
  7. # Prefix for the HTTP path alerts are pushed to.
  8. [ path_prefix: <path> | default = / ]
  10. # Configures the protocol scheme used for requests.
  11. [ scheme: <scheme> | default = http ]
  13. # Sets the `Authorization` header on every request with the
  14. # configured username and password.
  15. # password and password_file are mutually exclusive.
  16. basic_auth:
  17.   [ username: <string> ]
  18.   [ password: <string> ]
  19.   [ password_file: <string> ]
  21. # Sets the `Authorization` header on every request with
  22. # the configured bearer token. It is mutually exclusive with `bearer_token_file`.
  23. [ bearer_token: <string> ]
  25. # Sets the `Authorization` header on every request with the bearer token
  26. # read from the configured file. It is mutually exclusive with `bearer_token`.
  27. [ bearer_token_file: /path/to/bearer/token/file ]
  29. # Configures the scrape request's TLS settings.
  30. tls_config:
  31.   [ <tls_config> ]
  33. # Optional proxy URL.
  34. [ proxy_url: <string> ]
  36. # List of Azure service discovery configurations.
  37. azure_sd_configs:
  38.   [ - <azure_sd_config> ... ]
  40. # List of Consul service discovery configurations.
  41. consul_sd_configs:
  42.   [ - <consul_sd_config> ... ]
  44. # List of DNS service discovery configurations.
  45. dns_sd_configs:
  46.   [ - <dns_sd_config> ... ]
  48. # List of EC2 service discovery configurations.
  49. ec2_sd_configs:
  50.   [ - <ec2_sd_config> ... ]
  52. # List of file service discovery configurations.
  53. file_sd_configs:
  54.   [ - <file_sd_config> ... ]
  56. # List of GCE service discovery configurations.
  57. gce_sd_configs:
  58.   [ - <gce_sd_config> ... ]
  60. # List of Kubernetes service discovery configurations.
  61. kubernetes_sd_configs:
  62.   [ - <kubernetes_sd_config> ... ]
  64. # List of Marathon service discovery configurations.
  65. marathon_sd_configs:
  66.   [ - <marathon_sd_config> ... ]
  68. # List of AirBnB's Nerve service discovery configurations.
  69. nerve_sd_configs:
  70.   [ - <nerve_sd_config> ... ]
  72. # List of Zookeeper Serverset service discovery configurations.
  73. serverset_sd_configs:
  74.   [ - <serverset_sd_config> ... ]
  76. # List of Triton service discovery configurations.
  77. triton_sd_configs:
  78.   [ - <triton_sd_config> ... ]
  80. # List of labeled statically configured Alertmanagers.
  81. static_configs:
  82.   [ - <static_config> ... ]
  84. # List of Alertmanager relabel configurations.
  85. relabel_configs:
  86.   [ - <relabel_config> ... ]

<remote_write>

write_relabel_configs is relabeling applied to samples before sending themto the remote endpoint. Write relabeling is applied after external labels. Thiscould be used to limit which samples are sent.

There is a small demo of how to usethis functionality.

  1. # The URL of the endpoint to send samples to.
  2. url: <string>
  4. # Timeout for requests to the remote write endpoint.
  5. [ remote_timeout: <duration> | default = 30s ]
  7. # List of remote write relabel configurations.
  8. write_relabel_configs:
  9.   [ - <relabel_config> ... ]
  11. # Sets the `Authorization` header on every remote write request with the
  12. # configured username and password.
  13. # password and password_file are mutually exclusive.
  14. basic_auth:
  15.   [ username: <string> ]
  16.   [ password: <string> ]
  17.   [ password_file: <string> ]
  19. # Sets the `Authorization` header on every remote write request with
  20. # the configured bearer token. It is mutually exclusive with `bearer_token_file`.
  21. [ bearer_token: <string> ]
  23. # Sets the `Authorization` header on every remote write request with the bearer token
  24. # read from the configured file. It is mutually exclusive with `bearer_token`.
  25. [ bearer_token_file: /path/to/bearer/token/file ]
  27. # Configures the remote write request's TLS settings.
  28. tls_config:
  29.   [ <tls_config> ]
  31. # Optional proxy URL.
  32. [ proxy_url: <string> ]
  34. # Configures the queue used to write to remote storage.
  35. queue_config:
  36.   # Number of samples to buffer per shard before we block reading of more
  37.   # samples from the WAL. It is recommended to have enough capacity in each
  38.   # shard to buffer several requests to keep throughput up while processing
  39.   # occasional slow remote requests.
  40.   [ capacity: <int> | default = 500 ]
  41.   # Maximum number of shards, i.e. amount of concurrency.
  42.   [ max_shards: <int> | default = 1000 ]
  43.   # Minimum number of shards, i.e. amount of concurrency.
  44.   [ min_shards: <int> | default = 1 ]
  45.   # Maximum number of samples per send.
  46.   [ max_samples_per_send: <int> | default = 100]
  47.   # Maximum time a sample will wait in buffer.
  48.   [ batch_send_deadline: <duration> | default = 5s ]
  49.   # Initial retry delay. Gets doubled for every retry.
  50.   [ min_backoff: <duration> | default = 30ms ]
  51.   # Maximum retry delay.
  52.   [ max_backoff: <duration> | default = 100ms ]

There is a list ofintegrationswith this feature.

<remote_read>

  1. # The URL of the endpoint to query from.
  2. url: <string>
  4. # An optional list of equality matchers which have to be
  5. # present in a selector to query the remote read endpoint.
  6. required_matchers:
  7.   [ <labelname>: <labelvalue> ... ]
  9. # Timeout for requests to the remote read endpoint.
  10. [ remote_timeout: <duration> | default = 1m ]
  12. # Whether reads should be made for queries for time ranges that
  13. # the local storage should have complete data for.
  14. [ read_recent: <boolean> | default = false ]
  16. # Sets the `Authorization` header on every remote read request with the
  17. # configured username and password.
  18. # password and password_file are mutually exclusive.
  19. basic_auth:
  20.   [ username: <string> ]
  21.   [ password: <string> ]
  22.   [ password_file: <string> ]
  24. # Sets the `Authorization` header on every remote read request with
  25. # the configured bearer token. It is mutually exclusive with `bearer_token_file`.
  26. [ bearer_token: <string> ]
  28. # Sets the `Authorization` header on every remote read request with the bearer token
  29. # read from the configured file. It is mutually exclusive with `bearer_token`.
  30. [ bearer_token_file: /path/to/bearer/token/file ]
  32. # Configures the remote read request's TLS settings.
  33. tls_config:
  34.   [ <tls_config> ]
  36. # Optional proxy URL.
  37. [ proxy_url: <string> ]

There is a list ofintegrationswith this feature.