Kafka Exporter

grafana-agent内置了kafka_exporter，来采集kafka的metrics指标。

我们强烈推荐您使用独立的账号运行grafana-agent，并做好访问kafka实例的最小化授权，避免过度授权带来的安全隐患，更多可以参考documentation。

配置并启用kafka_exporter

  1. kafka_exporter:
  2.   enabled: true
  3.   # Address array (host:port) of Kafka server
  4.   kafka_uris: ['xxx','yyy']

采集的关键指标列表

  1. kafka_brokers: count of kafka_brokers (gauge)
  2. kafka_topic_partitions: Number of partitions for this Topic (gauge)
  3. kafka_topic_partition_current_offset: Current Offset of a Broker at Topic/Partition (gauge)
  4. kafka_consumergroup_current_offset: Current Offset of a ConsumerGroup at Topic/Partition (gauge)
  5. kafka_consumer_lag_millis: Current approximation of consumer lag for a ConsumerGroup at Topic/Partition (gauge)
  6. kafka_topic_partition_under_replicated_partition: 1 if Topic/Partition is under Replicated

完整地配置项说明

  1.   # Enables the kafka_exporter integration, allowing the Agent to automatically
  2.   # collect system metrics from the configured dnsmasq server address
  3.   [enabled: <boolean> | default = false]
  5.   # Sets an explicit value for the instance label when the integration is
  6.   # self-scraped. Overrides inferred values.
  7.   #
  8.   # The default value for this integration is inferred from the hostname
  9.   # portion of the first kafka_uri value. If there is more than one string
  10.   # in kafka_uri, the integration will fail to load and an instance value
  11.   # must be manually provided.
  12.   [instance: <string>]
  14.   # Automatically collect metrics from this integration. If disabled,
  15.   # the dnsmasq_exporter integration will be run but not scraped and thus not
  16.   # remote-written. Metrics for the integration will be exposed at
  17.   # /integrations/dnsmasq_exporter/metrics and can be scraped by an external
  18.   # process.
  19.   [scrape_integration: <boolean> | default = <integrations_config.scrape_integrations>]
  21.   # How often should the metrics be collected? Defaults to
  22.   # prometheus.global.scrape_interval.
  23.   [scrape_interval: <duration> | default = <global_config.scrape_interval>]
  25.   # The timeout before considering the scrape a failure. Defaults to
  26.   # prometheus.global.scrape_timeout.
  27.   [scrape_timeout: <duration> | default = <global_config.scrape_timeout>]
  29.   # Allows for relabeling labels on the target.
  30.   relabel_configs:
  31.   [- <relabel_config> ... ]
  33.   # Relabel metrics coming from the integration, allowing to drop series
  34.   # from the integration that you don't care about.
  35.   metric_relabel_configs:
  36.   [ - <relabel_config> ... ]
  38.   # How frequent to truncate the WAL for this integration.
  39.   [wal_truncate_frequency: <duration> | default = "60m"]
  41.   # Monitor the exporter itself and include those metrics in the results.
  42.   [include_exporter_metrics: <bool> | default = false]
  44.   # Address array (host:port) of Kafka server
  45.   [kafka_uris: <[]string>]
  47.   # Connect using SASL/PLAIN
  48.   [use_sasl: <bool>]
  50.   # Only set this to false if using a non-Kafka SASL proxy
  51.   [use_sasl_handshake: <bool> | default = true]
  53.   # SASL user name
  54.   [sasl_username: <string>]
  56.   # SASL user password
  57.   [sasl_password: <string>]
  59.   # The SASL SCRAM SHA algorithm sha256 or sha512 as mechanism
  60.   [sasl_mechanism: <string>]
  62.   # Connect using TLS
  63.   [use_tls: <bool>]
  65.   # The optional certificate authority file for TLS client authentication
  66.   [ca_file: <string>]
  68.   # The optional certificate file for TLS client authentication
  69.   [cert_file: <string>]
  71.   # The optional key file for TLS client authentication
  72.   [key_file: <string>]
  74.   # If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
  75.   [insecure_skip_verify: <bool>]
  77.   # Kafka broker version
  78.   [kafka_version: <string> | default = "2.0.0"]
  80.   # if you need to use a group from zookeeper
  81.   [use_zookeeper_lag: <bool>]
  83.   # Address array (hosts) of zookeeper server.
  84.   [zookeeper_uris: <[]string>]
  86.   # Kafka cluster name
  87.   [kafka_cluster_name: <string>]
  89.   # Metadata refresh interval
  90.   [metadata_refresh_interval: <duration> | default = "1m"]
  92.   # If true, all scrapes will trigger kafka operations otherwise, they will share results. WARN: This should be disabled on large clusters
  93.   [allow_concurrency: <bool> | default = true]
  95.   # Maximum number of offsets to store in the interpolation table for a partition
  96.   [max_offsets: <int> | default = 1000]
  98.   # How frequently should the interpolation table be pruned, in seconds
  99.   [prune_interval_seconds: <int> | default = 30]
  101.   # Regex filter for topics to be monitored
  102.   [topics_filter_regex: <string> | default = ".*"]
  104.   # Regex filter for consumer groups to be monitored
  105.   [groups_filter_regex: <string> | default = ".*"]