Filtering Logs with ELK

Filtering log output for specific tasks

The file system paths of DC/OS task logs contain information such as the agent ID, framework ID, and executor ID. You can use this information to filter the log output for specific tasks, applications, or agents.

Prerequisite

• An Elasticsearch installation that aggregates DC/OS logs

Install, configure, and start Logstash

1. Install Logstash.

2. Create the following dcos pattern file in your custom patterns directory, located at $PATTERNS_DIR:

   PATHELEM [^/]+
   TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/%{PATHELEM:framework}/executors/%{PATHELEM:executor}/runs/%{PATHELEM:run}

3. Update the configuration file for your Logstash instance to include the following grok filter, where $PATTERNS_DIR is replaced with your custom patterns directory:

   filter {
       grok {
           patterns_dir => "$PATTERNS_DIR"
           match => { "file" => "%{TASKPATH}" }
       }
   }

4. Start Logstash.

   Logstash will extract the agent, framework, executor, and run fields. These fields are shown in the metadata of all Mesos task log events. Elasticsearch queries will also show results from those fields.

Usage example

In the screenshots below, we are using Kibana hosted by logz.io, but your Kibana interface will look similar.

1. Type framework:* into the Search field. This will show all of the events where the framework field is defined:

   

   Figure 1. Logstash events

2. Click the disclosure triangle next to one of these events to view the details. This will show all of the fields extracted from the task log file path:

   

   Figure 2. Event details

3. Search for all of the events that reference the framework ID of the event shown in the screenshot above, but that do not contain the chosen framework field. This will show only non-task results:

   

   Figure 3. Search results

Template examples

Here are some example query templates. Replace the template parameters $executor1, $framework2, and any others with the actual values from your cluster.

• Logs related to a specific executor $executor1, including logs for tasks run from that executor:

  "$executor1"

• Non-task logs related to a specific executor $executor1:

  "$executor1" AND NOT executor:$executor1

• Logs (including task logs) for a framework $framework1, if $executor1 and $executor2 are that framework’s executors:

  "$framework1" OR "$executor1" OR "$executor2"

• Non-task logs for a framework $framework1, if $executor1 and $executor2 are that framework’s executors:

  ("$framework1" OR "$executor1" OR "$executor2") AND NOT (framework:$framework1 OR executor:$executor1 OR executor:$executor2)

• Logs for a framework $framework1 on a specific agent host $agent_host1:

  host:$agent_host1 AND ("$framework1" OR "$executor1" OR "$executor2")

• Non-task logs for a framework $framework1 on a specific agent $agent1 with host $agent_host1:

  host:$agent_host1 AND ("$framework1" OR "$executor1" OR "$executor2") AND NOT agent:$agent