Logging Quick Start

Getting started with DC/OS logging

Use this guide to get started with DC/OS logging. Many of the commands used here are described more fully in the CLI Command Reference.

Prerequisites:

Deploy a sample app

Deploy a sample Marathon app for use in this Quick Start Guide.

  1. Create the following Marathon app definition and save as test-log.json.

    1. {
    2.   "id": "/test-log",
    3.   "cmd": "while true;do echo stdout;echo stderr >&2;sleep 1;done",
    4.   "cpus": 0.001,
    5.   "instances": 1,
    6.   "mem": 128
    7. }

  2. Deploy the app with this CLI command:

    1. dcos marathon app add test-log.json

  3. Verify that the app has been successfully deployed and note the task ID:

    1. dcos task test-log

    The output should resemble:

    1. NAME      HOST        USER  STATE  ID
    2. test-log  10.0.1.105  root    R    test-log.e69c4b2f-c255-11e6-a451-aa711cbcaa78

View the Mesos and DC/OS logs

You can access the Mesos stderr and stdout logs natively through the DC/OS CLI dcos task log command. In this example, a task is launched and the stderr and stdout logs from Mesos are accessed.

  1. Run this command to view the stdout logs, where <task_id> is the task ID:

    1. dcos task log <task_id>

    The output should resemble:

    1. Thu Dec 15 00:49:10 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout
    2. Thu Dec 15 00:49:11 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout

  2. Run this command to follow the logs, where <task_id> is the task ID:

    1. dcos task log --follow <task_id>

    This will create a running stream of logs similar to this:

    1. Wed Dec 14 16:50:12 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131]: stdout
    2. Wed Dec 14 16:50:13 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131]: stdout

  3. Run this command to get last 5 log entries:

    1. dcos task log <task_id> --lines=5

    The output should resemble:

    1. Thu Dec 15 00:51:27 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout
    2. Thu Dec 15 00:51:28 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout
    3. Thu Dec 15 00:51:29 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout
    4. Thu Dec 15 00:51:30 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout
    5. Thu Dec 15 00:51:31 2016 ip-10-0-1-177.us-west-2.compute.internal Command Executor (Task: test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd) (Command: sh -c 'while true;d...') [7131] stdout

View the Mesos task and system logs

You can view logs from tasks or the host subsystem with the dcos node log command. find more information on these commands in the CLI Command Reference section.

  1. Run this command to view the leading Mesos master logs:

    1. dcos node log --leader --lines 3

    The output should resemble:

    1. Thu Dec 15 00:29:28 2016 ip-10-0-6-165.us-west-2.compute.internal [10530] ip-10-0-6-165.us-west-2.compute.internal nginx: 10.0.6.72 - - [15/Dec/2016:00:29:28 +0000] "GET /service/marathon/v2/groups?_timestamp=1481761768409&embed=group.groups&embed=group.apps&embed=group.pods&embed=group.apps.deployments&embed=group.apps.counts&embed=group.apps.tasks&embed=group.apps.taskStats&embed=group.apps.lastTaskFailure HTTP/1.1" 200 1941 "http://joel-logg-elasticl-m6yuis5u674t-297942863.us-west-2.elb.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
    2. Thu Dec 15 00:29:29 2016 ip-10-0-6-165.us-west-2.compute.internal nginx [2929] 2016/12/15 00:29:29 [notice] 10530#0: *1136 [lua] auth.lua:131: validate_jwt_or_exit(): UID from valid JWT: `email@email.io`, client: 10.0.6.72, server: dcos.*, request: "GET /system/v1/logs/v1/range/?skip_prev=3 HTTP/1.1", host: "joel-logg-elasticl-m6yuis5u674t-297942863.us-west-2.elb.amazonaws.com"
    3. Thu Dec 15 00:29:29 2016 ip-10-0-6-165.us-west-2.compute.internal dcos-oauth [1505] time="2016-12-15T00:29:29Z" level=info msg="HTTP request received" method=GET uri="/acs/api/v1/users/youremail@email.io"

  2. Run this command to view the Mesos agent logs, where node ID (<node_id>) is specified:

    1. dcos node log --mesos-id=<node_id> --lines 3

    Run dcos task to identify which node is running your app, followed by dcos node to get the node ID.

    The output should resemble:

    1. Thu Dec 15 00:46:18 2016 ip-10-0-1-175.us-west-2.compute.internal mesos-agent [3284] I1215 00:46:18.794333  3315 http.cpp:288] HTTP GET for /slave(1)/state from 10.0.1.175:44661 with User-Agent='Mesos-State / Host: ip-10-0-1-175, Pid: 3023'
    2. Thu Dec 15 00:46:20 2016 ip-10-0-1-175.us-west-2.compute.internal mesos-agent [3284] I1215 00:46:20.800422  3319 http.cpp:288] HTTP GET for /slave(1)/state from 10.0.1.175:44661 with User-Agent='Mesos-State / Host: ip-10-0-1-175, Pid: 3023'
    3. Thu Dec 15 00:46:22 2016 ip-10-0-1-175.us-west-2.compute.internal spartan-env [2621] 00:46:22.575 [error] Lager event handler error_logger_lager_h exited with reason {'EXIT',,{wait_for_query,{state,#Port<0.9301>,{spartan_tcp_listener,{198,51,100,3}},ranch_tcp,<0.27148.0>}},exit,tcp_closed,state_functions,[{gen_statem,loop_event_result,9,[{file,"gen_statem.erl"},{line,978}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]]},[{error_logger_lager_h,log_event,2,[{file,"/pkg/src/spartan/_build/default/lib/lager/src/error_logger_lager_h.erl"},{line,155}]},{gen_event,server_update,4,[{file,...},...]},...]}}

  3. Run these commands to view a list of components running on the leader or agent node:

    • Leader node:

      1. dcos node list-components --leader

      The output should resemble:

      1. dcos-diagnostics.service
      2. dcos-adminrouter.service
      3. dcos-cosmos.service
      4. dcos-epmd.service
      5. dcos-exhibitor.service
      6. dcos-gen-resolvconf.service
      7. dcos-gen-resolvconf.timer
      8. ...

    • Agent node, where your node ID (<mesos-id>) is specified:

      1. dcos node list-components --mesos-id=<mesos-id>

      The output should resemble:

      1. dcos-diagnostics.service
      2. dcos-diagnostics.socket
      3. dcos-adminrouter-agent.service
      4. dcos-docker-gc.service
      5. dcos-docker-gc.timer
      6. dcos-epmd.service
      7. dcos-gen-resolvconf.service
      8. ...

  4. Run this command to view the leading master component log for DC/OS components. In this example, the Marathon component logs are queried:

    1. dcos node log --leader --component dcos-marathon.service

    The output should resemble:

    1. Thu Dec 15 00:34:08 2016 ip-10-0-6-165.us-west-2.compute.internal java [2541] [2016-12-15 00:34:08,121] INFO  Received status update for task test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$28056dde:Thread-296)
    2. Thu Dec 15 00:34:08 2016 ip-10-0-6-165.us-west-2.compute.internal java [2541] [2016-12-15 00:34:08,121] INFO  Received status update for task test-log.2fc56009-c25d-11e6-81b2-9a5d88789ccd: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$28056dde:Thread-297)
    3. ...