Metrics Quick Start

Getting Started with metrics in DC/OS

This page explains how to get started with metrics in DC/OS. A metrics pipeline is natively integrated with DC/OS and no additional setup is required.

Prerequisites:

  • You must have the DC/OS CLI installed and be logged in as a superuser via the dcos auth login command.

  1. Optional: Deploy a sample Marathon app for use in this quick start guide. If you already have tasks running on DC/OS, you can skip this setup step.

    1. Create the following Marathon app definition and save as test-metrics.json.

      1. {
      2.   "id": "/test-metrics",
      3.   "cmd": "./statsd-emitter",
      4.   "fetch": [{"uri": "https://downloads.mesosphere.com/dcos-metrics/1.11.0/statsd-emitter", "executable": true}],
      5.   "cpus": 0.01,
      6.   "instances": 1,
      7.   "mem": 128
      8. }

    2. Deploy the app with the following CLI command:

      1. dcos marathon app add test-metrics.json

  2. To get the Mesos ID of the node that is running your app, run dcos task followed by dcos node. For example:

    1. Running dcos task shows that host 10.0.0.193 is running the Marathon task test-metrics.93fffc0c-fddf-11e6-9080-f60c51db292b.

      1. dcos task
      2. NAME          HOST        USER  STATE  ID                                                  
      3. test-metrics  10.0.0.193  root    R    test-metrics.93fffc0c-fddf-11e6-9080-f60c51db292b

    2. Running dcos node shows that host 10.0.0.193 has the Mesos ID 7749eada-4974-44f3-aad9-42e2fc6aedaf-S1.

      1. dcos node
      2.  HOSTNAME       IP                         ID                    
      3. 10.0.0.193  10.0.0.193  7749eada-4974-44f3-aad9-42e2fc6aedaf-S1

  3. View metrics.

    • Container metrics for a specific task

      For an overview of the resource consumption for a specific container, execute the following command:

      1. dcos task metrics summary <task-id>

      The output should resemble:

      1. CPU           MEM              DISK
      2. 0.17 (1.35%)  0.01GiB (6.46%)  0.00GiB (0.00%)

      The metrics summary command displays a summary of raw and percentage utilization of CPU, Memory and Disk resources using the metrics documented in the metrics reference summary.

      In particular, the following metrics and formula are used to compute the displayed values:

      1. CPU usage:
      1. cpus.system_time_secs + cpus.user_time_secs (raw)
      2. (cpus.system_time_secs + cpus.user_time_secs) / cpus.throttled_time_secs (percentage)
      1. Memory usage:
      1. mem.total_bytes (raw)
      2. mem.total_byes/mem.limit_bytes (percentage)
      1. Disk usage:
      1. disk.used_bytes (raw)
      2. disk.used_bytes/disk.total_bytes (percentage)

    • All metrics for a specific task

      To get a detailed list of all metrics related to a task, execute the following command:

      1. dcos task metrics details <task-id>

      The output is a combination of container resource utilization and metrics transmitted by the workload. For example:

      1. NAME                      VALUE
      2. cpus.limit                0.20            
      3. cpus.nr_periods           1272            
      4. cpus.nr_throttled         8               
      5. cpus.system_time_secs     0.23            
      6. cpus.throttled_time_secs  0.45            
      7. cpus.user_time_secs       0.15            
      8. mem.anon_bytes            9359360         
      9. mem.cache_bytes           106496          
      10. mem.file_bytes            106496          
      11. mem.limit_bytes           44040192        
      12. mem.rss_bytes             9359360         
      13. mem.total_bytes           9465856      
      14. perf.timestamp            1556720487.68

      The CPU, disk, and memory statistics come from container data supplied by Mesos. The statsd_tester.time.uptime statistic comes from the application itself.

    • Host level metrics

      For task data, host-level metrics are available in the form of a summary or a detailed table. To view host-level metrics, execute the following command:

      1. dcos node metrics details <mesos-id>

      The output displays the statistics about available resources on the node and their utilization. For example:

      1. NAME                       VALUE      TAGS
      2. cpu.idle                   99.56%
      3. cpu.system                 0.09%
      4. cpu.total                  0.34%
      5. cpu.user                   0.25%
      6. cpu.wait                   0.01%
      7. filesystem.capacity.free   134.75GiB  path: /
      8. filesystem.capacity.total  143.02GiB  path: /
      9. filesystem.capacity.used   2.33GiB    path: /
      10. filesystem.inode.free      38425263   path: /
      11. filesystem.inode.total     38504832   path: /
      12. filesystem.inode.used      79569      path: /
      13. load.15min                 0
      14. load.1min                  0
      15. load.5min                  0
      16. memory.buffers             0.08GiB
      17. memory.cached              2.41GiB
      18. memory.free                12.63GiB
      19. memory.total               15.67GiB
      20. process.count              175
      21. swap.free                  0.00GiB
      22. swap.total                 0.00GiB
      23. swap.used                  0.00GiB
      24. system.uptime              28627

    • Programmatic use of metrics

      All dcos-cli metrics commands can be executed with the --json for use in scripts. For example:

      1. dcos node metrics summary <mesos-id> --json

      The output displays the same data, but in JSON format, for convenient parsing:

      1. [
      2.   {
      3.     "name": "cpu.total",
      4.     "timestamp": "2018-04-09T23:46:16.834008315Z",
      5.     "value": 0.32,
      6.     "unit": "percent"
      7.   },
      8.   {
      9.     "name": "memory.total",
      10.     "timestamp": "2018-04-09T23:46:16.834650407Z",
      11.     "value": 16830304256,
      12.     "unit": "bytes"
      13.   },
      14.   {
      15.     "name": "memory.free",
      16.     "timestamp": "2018-04-09T23:46:16.834650407Z",
      17.     "value": 13553008640,
      18.     "unit": "bytes"
      19.   },
      20.   {
      21.     "name": "filesystem.capacity.total",
      22.     "timestamp": "2018-04-09T23:46:16.834373702Z",
      23.     "value": 153567944704,
      24.     "tags": {
      25.       "path": "/"
      26.     },
      27.     "unit": "bytes"
      28.   },
      29.   {
      30.     "name": "filesystem.capacity.used",
      31.     "timestamp": "2018-04-09T23:46:16.834373702Z",
      32.     "value": 2498990080,
      33.     "tags": {
      34.       "path": "/"
      35.     },
      36.     "unit": "bytes"
      37.   }
      38. ]