Selectors and Scrape Configs

The Monitoring app sets prometheus.prometheusSpec.ignoreNamespaceSelectors=false, which enables monitoring across all namespaces by default.

This ensures you can view traffic, metrics and graphs for resources deployed in a namespace with istio-injection=enabled label.

If you would like to limit Prometheus to specific namespaces, set prometheus.prometheusSpec.ignoreNamespaceSelectors=true. Once you do this, you will need to add additional configuration to continue to monitor your resources.

Limiting Monitoring to Specific Namespaces by Setting ignoreNamespaceSelectors to True

To limit monitoring to specific namespaces, you will edit the ignoreNamespaceSelectors Helm chart option. You will configure this option when installing or upgrading the Monitoring Helm chart:

  1. When installing or upgrading the Monitoring Helm chart, edit the values.yml and setprometheus.prometheusSpec.ignoreNamespaceSelectors=true.
  2. Complete the install or upgrade.

Result: Prometheus will be limited to specific namespaces which means one of the following configurations will need to be set up to continue to view data in various dashboards

Enabling Prometheus to Detect Resources in Other Namespaces

There are two different ways to enable Prometheus to detect resources in other namespaces when prometheus.prometheusSpec.ignoreNamespaceSelectors=true:

  • Monitoring specific namespaces: Add a Service Monitor or Pod Monitor in the namespace with the targets you want to scrape.
  • Monitoring across namespaces: Add an additionalScrapeConfig to your rancher-monitoring instance to scrape all targets in all namespaces.

Monitoring Specific Namespaces: Create a Service Monitor or Pod Monitor

This option allows you to define which specific services or pods you would like monitored in a specific namespace.

The usability tradeoff is that you have to create the service monitor or pod monitor per namespace since you cannot monitor across namespaces.

Selectors and Scrape Configs - 图1Prerequisite:

Define a ServiceMonitor or PodMonitor for <your namespace>. An example ServiceMonitor is provided below.

  1. Click ☰ > Cluster Management.
  2. Go to the cluster that you created and click Explore.
  3. In the top navigation bar, open the kubectl shell.
  4. If the ServiceMonitor or PodMonitor file is stored locally in your cluster, in kubectl create -f <name of service/pod monitor file>.yaml.
  5. If the ServiceMonitor or PodMonitor is not stored locally, run cat<< EOF | kubectl apply -f -, paste the file contents into the terminal, then run EOF to complete the command.
  6. Run kubectl label namespace <your namespace> istio-injection=enabled to enable the envoy sidecar injection.

Result: <your namespace> can be scraped by prometheus.

Example Service Monitor for Istio Proxies

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: envoy-stats-monitor
  5.   namespace: istio-system
  6.   labels:
  7.     monitoring: istio-proxies
  8. spec:
  9.   selector:
  10.     matchExpressions:
  11.     - {key: istio-prometheus-ignore, operator: DoesNotExist}
  12.   namespaceSelector:
  13.     any: true
  14.   jobLabel: envoy-stats
  15.   endpoints:
  16.   - path: /stats/prometheus
  17.     targetPort: 15090
  18.     interval: 15s
  19.     relabelings:
  20.     - sourceLabels: [__meta_kubernetes_pod_container_port_name]
  21.       action: keep
  22.       regex: '.*-envoy-prom'
  23.     - action: labeldrop
  24.       regex: "__meta_kubernetes_pod_label_(.+)"
  25.     - sourceLabels: [__meta_kubernetes_namespace]
  26.       action: replace
  27.       targetLabel: namespace
  28.     - sourceLabels: [__meta_kubernetes_pod_name]
  29.       action: replace
  30.       targetLabel: pod_name

Monitoring across namespaces: Set ignoreNamespaceSelectors to False

This enables monitoring across namespaces by giving Prometheus additional scrape configurations.

The usability tradeoff is that all of Prometheus’ additionalScrapeConfigs are maintained in a single Secret. This could make upgrading difficult if monitoring is already deployed with additionalScrapeConfigs before installing Istio.

  1. When installing or upgrading the Monitoring Helm chart, edit the values.yml and set the prometheus.prometheusSpec.additionalScrapeConfigs array to the Additional Scrape Config provided below.
  2. Complete the install or upgrade.

Result: All namespaces with the istio-injection=enabled label will be scraped by prometheus.

Additional Scrape Config

  1. - job_name: 'istio/envoy-stats'
  2.   scrape_interval: 15s
  3.   metrics_path: /stats/prometheus
  4.   kubernetes_sd_configs:
  5.     - role: pod
  6.   relabel_configs:
  7.     - source_labels: [__meta_kubernetes_pod_container_port_name]
  8.       action: keep
  9.       regex: '.*-envoy-prom'
  10.     - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  11.       action: replace
  12.       regex: ([^:]+)(?::\d+)?;(\d+)
  13.       replacement: $1:15090
  14.       target_label: __address__
  15.     - action: labelmap
  16.       regex: __meta_kubernetes_pod_label_(.+)
  17.     - source_labels: [__meta_kubernetes_namespace]
  18.       action: replace
  19.       target_label: namespace
  20.     - source_labels: [__meta_kubernetes_pod_name]
  21.       action: replace
  22.       target_label: pod_name