Log Management with Splunk

Managing system and application logs with a Splunk server

Overview

You can pipe system and application logs from a DC/OS cluster to your existing Splunk server. This document describes how to configure a Splunk universal forwarder to send output from each node to a Splunk installation. This document does not explain how to set up and configure a Splunk server.

These instructions are based on CoreOS and might differ substantially from other Linux distributions.

IMPORTANT: The agent node Splunk forwarder configuration expects tasks to write logs to stdout and stderr. Some DC/OS services, including Cassandra and Kafka, do not write logs to stdout and stderr. If you want to log these services, you must customize your agent node Splunk forwarder configuration.

Prerequisites

  • An existing Splunk installation that can ingest data for indexing
  • All DC/OS nodes must be able to connect to your Splunk indexer via HTTP or HTTPS
  • The ulimit of open files must be set to unlimited for your user with root access.

Step 1: All nodes

For all nodes in your DC/OS cluster:

  1. Install Splunk’s universal forwarder.
  2. Make sure the forwarder has the credentials it needs to send data to the indexer.
  3. Start the forwarder.

Step 2: Master nodes

For each master node in your DC/OS cluster:

  1. Create a script $SPLUNK_HOME/bin/scripts/journald-master.sh that will obtain the Mesos master logs from journald. This script can be used with DC/OS and DC/OS Enterprise. Log entries that do not apply are ignored.

    1. #!/bin/sh
    3. exec journalctl --since=now -f     \
    4. -u dcos-diagnostics.service        \
    5. -u dcos-diagnostics.socket         \
    6. -u dcos-adminrouter-reload.service \
    7. -u dcos-adminrouter-reload.timer   \
    8. -u dcos-adminrouter.service        \
    9. -u dcos-bouncer.service            \
    10. -u dcos-ca.service                 \
    11. -u dcos-cfn-signal.service         \
    12. -u dcos-cosmos.service             \
    13. -u dcos-download.service           \
    14. -u dcos-epmd.service               \
    15. -u dcos-exhibitor.service          \
    16. -u dcos-gen-resolvconf.service     \
    17. -u dcos-gen-resolvconf.timer       \
    18. -u dcos-history.service            \
    19. -u dcos-link-env.service           \
    20. -u dcos-logrotate-master.timer     \
    21. -u dcos-marathon.service           \
    22. -u dcos-mesos-dns.service          \
    23. -u dcos-mesos-master.service       \
    24. -u dcos-metronome.service          \
    25. -u dcos-minuteman.service          \
    26. -u dcos-navstar.service            \
    27. -u dcos-networking_api.service     \
    28. -u dcos-secrets.service            \
    29. -u dcos-setup.service              \
    30. -u dcos-signal.service             \
    31. -u dcos-signal.timer               \
    32. -u dcos-spartan-watchdog.service   \
    33. -u dcos-spartan-watchdog.timer     \
    34. -u dcos-spartan.service            \
    35. -u dcos-vault.service              \
    36. -u dcos-logrotate-master.service

  2. Make the script executable:

    1. chmod +x "$SPLUNK_HOME/bin/scripts/journald-master.sh"

  3. Add the script as an input to the forwarder:

    1. "$SPLUNK_HOME/bin/splunk" add exec \
    2.     -source "$SPLUNK_HOME/bin/scripts/journald-master.sh" \
    3.     -interval 0

Step 3: Agent nodes

For each agent node in your DC/OS cluster:

  1. Create a script $SPLUNK_HOME/bin/scripts/journald-agent.sh that will obtain the Mesos agent logs from journald. This script can be used with DC/OS and DC/OS Enterprise. Log entries that do not apply are ignored.

    1. #!/bin/sh
    3. journalctl --since="now" -f              \
    4. -u dcos-diagnostics.service              \
    5. -u dcos-logrotate-agent.timer            \
    6. -u dcos-diagnostics.socket               \
    7. -u dcos-mesos-slave.service              \
    8. -u dcos-adminrouter-agent.service        \
    9. -udcos-minuteman.service                \
    10. -udcos-adminrouter-reload.service       \
    11. -udcos-navstar.service                  \
    12. -udcos-adminrouter-reload.timer         \
    13. -udcos-rexray.service                   \
    14. -udcos-cfn-signal.service               \
    15. -udcos-setup.service                    \
    16. -udcos-download.service                 \
    17. -udcos-signal.timer                     \
    18. -udcos-epmd.service                     \
    19. -udcos-spartan-watchdog.service         \
    20. -udcos-gen-resolvconf.service           \
    21. -udcos-spartan-watchdog.timer           \
    22. -udcos-gen-resolvconf.timer             \
    23. -udcos-spartan.service                  \
    24. -udcos-link-env.service                 \
    25. -udcos-vol-discovery-priv-agent.service \
    26. -udcos-logrotate-agent.service

  2. Make the script executable:

    1. chmod +x "$SPLUNK_HOME/bin/scripts/journald-agent.sh"

  3. Add the script as an input to the forwarder:

    1. "$SPLUNK_HOME/bin/splunk" add exec \
    2.     -source "$SPLUNK_HOME/bin/scripts/journald-agent.sh" \
    3.     -interval 0

  4. Add the task logs as inputs to the forwarder:

    1.     "$SPLUNK_HOME/bin/splunk" add monitor '/var/lib/mesos/slave' \
    2.         -whitelist '/stdout$|/stderr$'

Get more details

For details on how to filter your logs with Splunk, see Filtering logs with Splunk.