Metrics Monitoring

Collect runtime Metrics indicators and integrate with Prometheus and Grafana systems

Dubbo supports the collection of runtime Metrics indicators and their integration with Prometheus and Grafana systems to achieve visual monitoring of microservice clusters. Below is a specific usage example. You can view the complete source code of the example.

server/main.go - is the main definition of the service, handler and rpc server
client/main.go - is the rpc client
proto - contains the protobuf definition of the API

How to run

Run server

go run ./go-server/cmd/main.go

test server work as expected:

curl \
    --header "Content-Type: application/json" \
    --data '{"name": "Dubbo"}' \
    http://localhost:20000/greet.GreetService/Greet

Run client

go run ./go-client/cmd/main.go

deploy to local

install prometheus and open prometheus config file prometheus.yml, write the config like this

global:
  evaluation_interval: 15s
  scrape_interval: 15s
scrape_configs:
- job_name: dubbo-provider
  scrape_interval: 15s
  scrape_timeout: 5s
  metrics_path: /prometheus
  static_configs:
    - targets: ['localhost:9099']
- job_name: dubbo-consumer
  scrape_interval: 15s
  scrape_timeout: 5s
  metrics_path: /prometheus
  static_configs:
    - targets: ['localhost:9097']

install grafana and open grafana web page like localhost:3000

open: 【Home / Connections / Data sources】

click 【Add new data source】

select Prometheus

enter 【Prometheus server URL】 like http://localhost:9090 and click 【Save & test】

open 【Home / Dashboards 】click 【New】【import】and enter 19294 click Load

if your grafana can’t access internet you can open https://grafana.com/grafana/dashboards/19294-dubbo-observability/ and click 【Download JSON】

paste the JSON

click 【Import】button and you will see the Dubbo Observability dashboard, enjoy it

Deploy to Kubernetes

kube-prometheus

install prometheus in k8s kube-prometheus

Set prometheus-service.yaml type to NodePort

add dubboPodMoitor.yaml to kube-prometheus manifests dir, The content is as follows

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: podmonitor
 labels:
   app: podmonitor
 namespace: monitoring
spec:
 namespaceSelector:
   matchNames:
     - dubbo-system
 selector:
   matchLabels:
     app-type: dubbo
 podMetricsEndpoints:
   - port: metrics # ref to dubbo-app port name metrics
     path: /prometheus
---
# rbac
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 namespace: dubbo-system
 name: pod-reader
rules:
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["get", "list", "watch"]
---
# rbac
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: pod-reader-binding
 namespace: dubbo-system
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: pod-reader
subjects:
 - kind: ServiceAccount
   name: prometheus-k8s
   namespace: monitoring