Exposing custom application metrics for autoscaling

You can export custom application metrics for the horizontal pod autoscaler.

Prometheus Adapter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

Exposing custom application metrics for horizontal pod autoscaling

You can use the prometheus-adapter resource to expose custom application metrics for the horizontal pod autoscaler.

Prerequisites

  • You have a custom Prometheus instance installed. In this example, it is presumed that Prometheus was installed in a user-defined custom-prometheus project.

    Custom Prometheus instances and the Prometheus Operator installed through Operator Lifecycle Manager (OLM) can cause issues with user-defined workload monitoring if it is enabled. Custom Prometheus instances are not supported in OKD.

  • You have deployed an application and a service in a user-defined project. In this example, it is presumed that the application and its service monitor were installed in a user-defined custom-prometheus project.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a YAML file for your configuration. In this example, the file is called deploy.yaml.

  2. Add configuration details for creating the service account, roles, and role bindings for prometheus-adapter:

    1. kind: ServiceAccount
    2. apiVersion: v1
    3. metadata:
    4.   name: custom-metrics-apiserver
    5.   namespace: custom-prometheus
    6. ---
    7. apiVersion: rbac.authorization.k8s.io/v1
    8. kind: ClusterRole
    9. metadata:
    10.   name: custom-metrics-server-resources
    11. rules:
    12. - apiGroups:
    13.   - custom.metrics.k8s.io
    14.   resources: ["*"]
    15.   verbs: ["*"]
    16. ---
    17. apiVersion: rbac.authorization.k8s.io/v1
    18. kind: ClusterRole
    19. metadata:
    20.   name: custom-metrics-resource-reader
    21. rules:
    22. - apiGroups:
    23.   - ""
    24.   resources:
    25.   - namespaces
    26.   - pods
    27.   - services
    28.   verbs:
    29.   - get
    30.   - list
    31. ---
    32. apiVersion: rbac.authorization.k8s.io/v1
    33. kind: ClusterRoleBinding
    34. metadata:
    35.   name: custom-metrics:system:auth-delegator
    36. roleRef:
    37.   apiGroup: rbac.authorization.k8s.io
    38.   kind: ClusterRole
    39.   name: system:auth-delegator
    40. subjects:
    41. - kind: ServiceAccount
    42.   name: custom-metrics-apiserver
    43.   namespace: custom-prometheus
    44. ---
    45. apiVersion: rbac.authorization.k8s.io/v1
    46. kind: RoleBinding
    47. metadata:
    48.   name: custom-metrics-auth-reader
    49.   namespace: kube-system
    50. roleRef:
    51.   apiGroup: rbac.authorization.k8s.io
    52.   kind: Role
    53.   name: extension-apiserver-authentication-reader
    54. subjects:
    55. - kind: ServiceAccount
    56.   name: custom-metrics-apiserver
    57.   namespace: custom-prometheus
    58. ---
    59. apiVersion: rbac.authorization.k8s.io/v1
    60. kind: ClusterRoleBinding
    61. metadata:
    62.   name: custom-metrics-resource-reader
    63. roleRef:
    64.   apiGroup: rbac.authorization.k8s.io
    65.   kind: ClusterRole
    66.   name: custom-metrics-resource-reader
    67. subjects:
    68. - kind: ServiceAccount
    69.   name: custom-metrics-apiserver
    70.   namespace: custom-prometheus
    71. ---
    72. apiVersion: rbac.authorization.k8s.io/v1
    73. kind: ClusterRoleBinding
    74. metadata:
    75.   name: hpa-controller-custom-metrics
    76. roleRef:
    77.   apiGroup: rbac.authorization.k8s.io
    78.   kind: ClusterRole
    79.   name: custom-metrics-server-resources
    80. subjects:
    81. - kind: ServiceAccount
    82.   name: horizontal-pod-autoscaler
    83.   namespace: kube-system
    84. ---

  3. Add configuration details for the custom metrics for prometheus-adapter:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: adapter-config
    5.   namespace: custom-prometheus
    6. data:
    7.   config.yaml: |
    8.     rules:
    9.     - seriesQuery: 'http_requests_total{namespace!="",pod!=""}' (1)
    10.       resources:
    11.         overrides:
    12.           namespace: {resource: "namespace"}
    13.           pod: {resource: "pod"}
    14.           service: {resource: "service"}
    15.       name:
    16.         matches: "^(.*)_total"
    17.         as: "${1}_per_second" (2)
    18.       metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
    19. ---
    1Specifies the chosen metric to be the number of HTTP requests.
    2Specifies the frequency for the metric.

  4. Add configuration details for registering prometheus-adapter as an API service:

    1. apiVersion: v1
    2. kind: Service
    3. metadata:
    4.   annotations:
    5.     service.alpha.openshift.io/serving-cert-secret-name: prometheus-adapter-tls
    6.   labels:
    7.     name: prometheus-adapter
    8.   name: prometheus-adapter
    9.   namespace: custom-prometheus
    10. spec:
    11.   ports:
    12.   - name: https
    13.     port: 443
    14.     targetPort: 6443
    15.   selector:
    16.     app: prometheus-adapter
    17.   type: ClusterIP
    18. ---
    19. apiVersion: apiregistration.k8s.io/v1beta1
    20. kind: APIService
    21. metadata:
    22.   name: v1beta1.custom.metrics.k8s.io
    23. spec:
    24.   service:
    25.     name: prometheus-adapter
    26.     namespace: custom-prometheus
    27.   group: custom.metrics.k8s.io
    28.   version: v1beta1
    29.   insecureSkipTLSVerify: true
    30.   groupPriorityMinimum: 100
    31.   versionPriority: 100
    32. ---

  5. List the Prometheus Adapter image:

    1. $ oc get -n openshift-monitoring deploy/prometheus-adapter -o jsonpath="{..image}"

  6. Add configuration details for deploying prometheus-adapter:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   labels:
    5.     app: prometheus-adapter
    6.   name: prometheus-adapter
    7.   namespace: custom-prometheus
    8. spec:
    9.   replicas: 1
    10.   selector:
    11.     matchLabels:
    12.       app: prometheus-adapter
    13.   template:
    14.     metadata:
    15.       labels:
    16.         app: prometheus-adapter
    17.       name: prometheus-adapter
    18.     spec:
    19.       serviceAccountName: custom-metrics-apiserver
    20.       containers:
    21.       - name: prometheus-adapter
    22.         image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a46915a206cd7d97f240687c618dd59e8848fcc3a0f51e281f3384153a12c3e0 (1)
    23.         args:
    24.         - --secure-port=6443
    25.         - --tls-cert-file=/var/run/serving-cert/tls.crt
    26.         - --tls-private-key-file=/var/run/serving-cert/tls.key
    27.         - --logtostderr=true
    28.         - --prometheus-url=http://prometheus-operated.default.svc:9090/
    29.         - --metrics-relist-interval=1m
    30.         - --v=4
    31.         - --config=/etc/adapter/config.yaml
    32.         ports:
    33.         - containerPort: 6443
    34.         volumeMounts:
    35.         - mountPath: /var/run/serving-cert
    36.           name: volume-serving-cert
    37.           readOnly: true
    38.         - mountPath: /etc/adapter/
    39.           name: config
    40.           readOnly: true
    41.         - mountPath: /tmp
    42.           name: tmp-vol
    43.       volumes:
    44.       - name: volume-serving-cert
    45.         secret:
    46.           secretName: prometheus-adapter-tls
    47.       - name: config
    48.         configMap:
    49.           name: adapter-config
    50.       - name: tmp-vol
    51.         emptyDir: {}
    1Specifies the Prometheus Adapter image found in the previous step.

  7. Apply the configuration to the cluster:

    1. $ oc apply -f deploy.yaml

    Example output

    1. serviceaccount/custom-metrics-apiserver created
    2. clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
    3. clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
    4. clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
    5. rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
    6. clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
    7. clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created
    8. configmap/adapter-config created
    9. service/prometheus-adapter created
    10. apiservice.apiregistration.k8s.io/v1.custom.metrics.k8s.io created
    11. deployment.apps/prometheus-adapter created

  8. Verify that the prometheus-adapter pod in your user-defined project is in a Running state. In this example the project is custom-prometheus:

    1. $ oc -n custom-prometheus get pods prometheus-adapter-<string>

  9. The metrics for the application are now exposed and they can be used to configure horizontal pod autoscaling.

Additional resources

Next steps