Getting Per-Route Metrics

To get per-route metrics, you must first create a service profile. Once a service profile has been created, Linkerd will add labels to the Prometheus metrics that associate a specific request to a specific route.

For a tutorial that shows this functionality off, check out the books demo.

You can view per-route metrics in the CLI by running linkerd viz routes:

  1. $ linkerd viz routes svc/webapp
  2. ROUTE                       SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
  3. GET /                        webapp   100.00%   0.6rps          25ms          30ms          30ms
  4. GET /authors/{id}            webapp   100.00%   0.6rps          22ms          29ms          30ms
  5. GET /books/{id}              webapp   100.00%   1.2rps          18ms          29ms          30ms
  6. POST /authors                webapp   100.00%   0.6rps          32ms          46ms          49ms
  7. POST /authors/{id}/delete    webapp   100.00%   0.6rps          45ms          87ms          98ms
  8. POST /authors/{id}/edit      webapp     0.00%   0.0rps           0ms           0ms           0ms
  9. POST /books                  webapp    50.76%   2.2rps          26ms          38ms          40ms
  10. POST /books/{id}/delete      webapp   100.00%   0.6rps          24ms          29ms          30ms
  11. POST /books/{id}/edit        webapp    60.71%   0.9rps          75ms          98ms         100ms
  12. [DEFAULT]                    webapp     0.00%   0.0rps           0ms           0ms           0ms

The [DEFAULT] route is a catch-all, anything that does not match the regexes specified in your service profile will end up there.

It is also possible to look the metrics up by other resource types, such as:

  1. $ linkerd viz routes deploy/webapp
  2. ROUTE                          SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
  3. [DEFAULT]                   kubernetes     0.00%   0.0rps           0ms           0ms           0ms
  4. GET /                           webapp   100.00%   0.5rps          27ms          38ms          40ms
  5. GET /authors/{id}               webapp   100.00%   0.6rps          18ms          29ms          30ms
  6. GET /books/{id}                 webapp   100.00%   1.1rps          17ms          28ms          30ms
  7. POST /authors                   webapp   100.00%   0.5rps          25ms          30ms          30ms
  8. POST /authors/{id}/delete       webapp   100.00%   0.5rps          58ms          96ms          99ms
  9. POST /authors/{id}/edit         webapp     0.00%   0.0rps           0ms           0ms           0ms
  10. POST /books                     webapp    45.58%   2.5rps          33ms          82ms          97ms
  11. POST /books/{id}/delete         webapp   100.00%   0.6rps          33ms          48ms          50ms
  12. POST /books/{id}/edit           webapp    55.36%   0.9rps          79ms         160ms         192ms
  13. [DEFAULT]                       webapp     0.00%   0.0rps           0ms           0ms           0ms

Then, it is possible to filter all the way down to requests going from a specific resource to other services:

  1. $ linkerd viz routes deploy/webapp --to svc/books
  2. ROUTE                     SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
  3. DELETE /books/{id}.json     books   100.00%   0.5rps          18ms          29ms          30ms
  4. GET /books.json             books   100.00%   1.1rps           7ms          12ms          18ms
  5. GET /books/{id}.json        books   100.00%   2.5rps           6ms          10ms          10ms
  6. POST /books.json            books    52.24%   2.2rps          23ms          34ms          39ms
  7. PUT /books/{id}.json        books    41.98%   1.4rps          73ms          97ms          99ms
  8. [DEFAULT]                   books     0.00%   0.0rps           0ms           0ms           0ms

Troubleshooting

If you’re not seeing any metrics, there are two likely culprits. In both cases, linkerd viz tap can be used to understand the problem. For the resource that the service points to, run:

  1. linkerd viz tap deploy/webapp -o wide | grep req

A sample output is:

  1. req id=3:1 proxy=in  src=10.4.0.14:58562 dst=10.4.1.4:7000 tls=disabled :method=POST :authority=webapp:7000 :path=/books/24783/edit src_res=deploy/traffic src_ns=default dst_res=deploy/webapp dst_ns=default rt_route=POST /books/{id}/edit

This will select only the requests observed and show the :authority and rt_route that was used for each request.

  • Linkerd discovers the right service profile to use via :authority or Host headers. The name of your service profile must match these headers. There are many reasons why these would not match, see ingress for one reason. Another would be clients that use IPs directly such as Prometheus.
  • Getting regexes to match can be tough and the ordering is important. Pay attention to rt_route. If it is missing entirely, compare the :path to the regex you’d like for it to match, and use a tester with the Golang flavor of regex.