Recommendation Framework

Introduce for Recommendation Framework and How to use it

Recommendation Framework provides the ability to automatically analyze various resources in Kubernetes cluster and make recommendations for optimization。

Overview

Crane’s recommendation module periodically detects cluster resource configuration problems and provides optimization suggestions. Framework provides a variety of recommender to implement the optimization and recommendation for different resources. If you want to know how Crane makes Recommendations, or if you want to try to implement a custom Recommender, or change the recommender’s implements, this article will help you know How it does.

Use Case

The following are typical use cases for recommendation framework：

  • Create a RecommendationRule configuration. RecommendationRule Controller will conduct Recommendation tasks periodically based on the configuration and give recommendations.
  • Adjust the allocation of resources according to the optimization Recommendation.

Create RecommendationRule

This is a sample for RecommendationRule: workload-rule.yaml。

  1. apiVersion: analysis.crane.io/v1alpha1
  2. kind: RecommendationRule
  3. metadata:
  4.   name: workloads-rule
  5.  spec:
  6.   runInterval: 24h                            # Run every 24 hours
  7.   resourceSelectors:                          # configuration for resources
  8.     - kind: Deployment
  9.       apiVersion: apps/v1
  10.     - kind: StatefulSet
  11.       apiVersion: apps/v1
  12.   namespaceSelector:
  13.     any: true                                 # scan all namespace
  14.   recommenders:                               # Replicas and Resource recommenders for Workload 
  15.     - name: Replicas
  16.     - name: Resource

In this example：

  • The analysis is run every 24 hours. runInterval format is an time interval, for example, 1h or 1m. If this parameter is set to null, the analysis is run only once.
  • Resources to be analyzed are set by configuring the ‘resourceSelectors’ array. Each resourceSelector selects resources in k8s by kind, apiVersion, name. If no name is specified, it indicates all resources based on namespaceSelector.
  • namespaceSelector defines the namespace of the resource to be analyzed. any: true indicates that all namespaces are selected
  • recommenders define which recommender should be used to analyze the resources. Currently, supported types: recommenders
  • the Resource type and recommenders need to match, such as the Resource recommended default only support Deployments and StatefulSets. Please refer to the Recommender docs to know which resources it supports
  1. Create the RecommendationRule with the following command, a recommendation will start as soon as it is created.
  1. kubectl apply -f workload-rules.yaml

This example will analysis Resource and Replicas for Deployments and StatefulSets in all namespace.。

  1. Check the RecommendationRule recommendation progress. Observe the progress of recommendation tasks through Status.recommendations. Recommendation tasks are executed sequentially. If lastStartTime of all tasks is the latest time and message has value, it indicates that this recommendation is completed.
  1. kubectl get rr workloads-rule
  1. status:
  2.   lastUpdateTime: "2022-09-28T10:36:02Z"
  3.   recommendations:
  4.   - apiVersion: analysis.crane.io/v1alpha1
  5.     kind: Recommendation
  6.     lastStartTime: "2022-09-28T10:36:02Z"
  7.     message: Success
  8.     name: workloads-rule-replicas-rckvb
  9.     namespace: default
  10.     recommenderRef:
  11.       name: Replicas
  12.     targetRef:
  13.       apiVersion: apps/v1
  14.       kind: Deployment
  15.       name: php-apache
  16.       namespace: default
  17.     uid: b15cbcd7-6fe2-4ace-9ae8-11cc0a6e69c2
  18.   - apiVersion: analysis.crane.io/v1alpha1
  19.     kind: Recommendation
  20.     lastStartTime: "2022-09-28T10:36:02Z"
  21.     message: Success
  22.     name: workloads-rule-resource-pnnxn
  23.     namespace: default
  24.     recommenderRef:
  25.       name: Resource
  26.     targetRef:
  27.       apiVersion: apps/v1
  28.       kind: Deployment
  29.       name: php-apache
  30.       namespace: default
  31.     uid: 8472013a-bda2-4025-b0df-3fdc69c1c910
  1. Check Recommendation

You can filter Recommendation through these labels，like kubectl get recommend -l analysis.crane.io/recommendation-rule-name=workloads-rule

  • RecommendationRule Name：analysis.crane.io/recommendation-rule-name
  • RecommendationRule UID：analysis.crane.io/recommendation-rule-uid
  • RecommendationRule recommender：analysis.crane.io/recommendation-rule-recommender
  • Recommended resource’s kind：analysis.crane.io/recommendation-target-kind
  • Recommended resource’s version：analysis.crane.io/recommendation-target-version
  • Recommended resource’s name：analysis.crane.io/recommendation-target-name

In general, the namespace of Recommendation is equal to the namespace of the recommended resource. But Recommendation for idle nodes is excluded, which are in the root namespace of crane, and the default root namespace is Crane-system.

Adjust the allocation of resources according to the optimization Recommendation

For resource/replicas recommendation and recommendedInfo, users can PATCH status.recommendedinfo to workload to update the resource configuration, for example:

  1. patchData=`kubectl get recommend workloads-rule-replicas-rckvb -n default -o jsonpath='{.status.recommendedInfo}'`;kubectl patch Deployment php-apache -n default --patch "${patchData}"

For idle nodes, users can offline or reduce the capacity of idle nodes based on their requirements.

Recommender

Currently, Crane support these Recommenders:

  • Resource Recommendation: Use the VPA algorithm to analyze the actual usage of applications and recommend more appropriate resource configurations.
  • Replicas Recommendation: Use the HPA algorithm to analyze the actual usage of applications and recommend more appropriate replicas configurations.

Recommender Framework

Recommender framework defines a set of workflow, The workflow execution sequence according to the process, the process is divided into four stages: Filter, Prepare, Recommend, Observe. Recommender performs recommends functions by implementing these four stages.

If you want to implement or extend a Recommender, please refer to :How to develop recommender

RecommendationConfiguration

RecommendationConfiguration defines the configuration for recommender. It will deploy a ConfigMap in crane root namespace: recommendation-configuration.

The following is a sample for RecommendationConfiguration.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: recommendation-configuration
  5.   namespace: crane-system
  6. data:
  7.   config.yaml: |-
  8.     apiVersion: analysis.crane.io/v1alpha1
  9.     kind: RecommendationConfiguration
  10.     recommenders:
  11.       - name: Replicas
  12.         acceptedResources:                   # Accepted resources
  13.           - kind: Deployment
  14.             apiVersion: apps/v1
  15.           - kind: StatefulSet
  16.             apiVersion: apps/v1
  17.         config:                              # settings for recommender
  18.           workload-min-replicas: "1"         
  19.       - name: Resource
  20.         acceptedResources:                   # Accepted resources
  21.           - kind: Deployment
  22.             apiVersion: apps/v1
  23.           - kind: StatefulSet
  24.             apiVersion: apps/v1

Users can modify ConfigMap and rolling update Crane to make new configuration works.

How to make recommendations more accurate

The more historical data stored in a monitoring system, such as Prometheus, the more accurate the recommendation will be. More than two weeks’s data is recommended for production. Predictions about new apps are often inaccurate.