Mount Disk Resources

Using Mesos to mount storage resources

With DC/OS, you can configure Mesos Mount disk resources across your cluster by mounting storage resources on agents using a well-known path.

When a DC/OS agent starts, it scans for volumes that match the pattern /dcos/volume<N>, where <N> is an integer. The agent is then automatically configured to offer these disk resources to other services.

Example using loopback device

In this example, a disk resource is added to a DC/OS agent post-install on a running cluster. These same steps can be used pre-install without having to stop services or clear the agent state.

Please note that this example handles adding resources exclusively and can not get applied the same way when removing resources.

WARNING: This will terminate any running tasks or services on the node.

  1. Connect to an agent in the cluster with SSH.

  2. Examine the current agent resource state.

    1. cat /var/lib/dcos/mesos-resources
    2. # Generated by make_disk_resources.py on 2016-05-05 17:04:29.868595
    3. #
    5. MESOS_RESOURCES='[{"ranges": {"range": [{"end": 21, "begin": 1}, {"end": 5050, "begin": 23}, {"end": 32000, "begin": 5052}]}, "type":  "RANGES", "name": "ports"}, {"role": "*", "type": "SCALAR", "name": "disk", "scalar": {"value": 47540}}]'

    Note there are no references yet for /dcos/volume0.

  3. Stop the agent.

    • On a private agent:

      1. sudo systemctl stop dcos-mesos-slave.service

    • On a public agent:

      1. sudo systemctl stop dcos-mesos-slave-public.service

  4. Clear agent state.

    1. Cache volume mount discovery resource state with this command:

      1. sudo mv -f /var/lib/dcos/mesos-resources /var/lib/dcos/mesos-resources.cache

      DC/OS will check this file later to generate a new resource state for the agent.

    2. Remove agent checkpoint state with this command:

      1. sudo rm -f /var/lib/mesos/slave/meta/slaves/latest

  5. Create a 200 MB loopback device.

    1. sudo mkdir -p /dcos/volume0
    2. sudo dd if=/dev/zero of=/root/volume0.img bs=1M count=200
    3. sudo losetup /dev/loop0 /root/volume0.img
    4. sudo mkfs -t ext4 /dev/loop0
    5. sudo losetup -d /dev/loop0

    This is suitable for testing purposes only. Mount volumes must have at least 200 MB of free space available. 100 MB on each volume is reserved by DC/OS and is not available for other services.

  6. Create fstab entry and mount.

    Ensure the volume is mounted automatically at boot time. Something similar could also be done with a systemd mount unit.

    1. echo "/root/volume0.img /dcos/volume0 auto loop 0 2" | sudo tee -a /etc/fstab
    2. sudo mount /dcos/volume0

  7. Reboot.

    1. sudo reboot

  8. SSH to the agent and review the journald logs for references to the new volume /dcos/volume0.

    1. journalctl -b | grep '/dcos/volume0'

    In particular, there should be an entry for the agent starting up and the new volume0 Mount disk resource:

    1. May 05 19:18:40 dcos-agent-public-01234567000001 systemd[1]: Mounting /dcos/volume0...
    2. May 05 19:18:42 dcos-agent-public-01234567000001 systemd[1]: Mounted /dcos/volume0.
    3. May 05 19:18:46 dcos-agent-public-01234567000001 make_disk_resources.py[888]: Found matching mounts : [('/dcos/volume0', 74)]
    4. May 05 19:18:46 dcos-agent-public-01234567000001 make_disk_resources.py[888]: Generated disk resources map: [{'name': 'disk', 'type': 'SCALAR', 'disk': {'source': {'mount': {'root': '/dcos/volume0'}, 'type': 'MOUNT'}}, 'role': '*', 'scalar': {'value': 74}}, {'name': 'disk', 'type': 'SCALAR', 'role': '*', 'scalar': {'value': 47540}}]
    5. May 05 19:18:58 dcos-agent-public-01234567000001 mesos-slave[1891]: " --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resources="[{"name": "ports", "type": "RANGES", "ranges": {"range": [{"end": 21, "begin": 1}, {"end": 5050, "begin": 23}, {"end": 32000, "begin": 5052}]}}, {"name": "disk", "type": "SCALAR", "disk": {"source": {"mount": {"root": "/dcos/volume0"}, "type": ""}}, "role": "*", "scalar": {"value": 74}}, {"name": "disk", "type": "SCALAR", "role": "*", "scalar": {"value": 47540}}]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --slave_subsystems="cpu,memory" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos/slave"

Example using a mount volume within a Marathon app

  1. {
  2.   "id": "/mount-test-svc1",
  3.   "instances": 1,
  4.   "cpus": 0.1,
  5.   "mem": 128,
  6.   "networks": [
  7.     {
  8.       "mode": "container/bridge"
  9.     }
  10.   ],
  11.   "disk": 0,
  12.   "gpus": 0,
  13.   "backoffSeconds": 1,
  14.   "backoffFactor": 1.15,
  15.   "maxLaunchDelaySeconds": 300,
  16.   "container": {
  17.     "type": "DOCKER",
  18.     "volumes": [
  19.       {
  20.         "persistent": {
  21.           "size": 25,
  22.           "type": "mount"
  23.         },
  24.         "mode": "RW",
  25.         "containerPath": "volume0"
  26.       }
  27.     ],
  28.     "docker": {
  29.       "image": "nginx",
  30.       "privileged": false,
  31.       "forcePullImage": false
  32.     },
  33.     "portMappings": [
  34.       {
  35.         "containerPort": 80,
  36.         "hostPort": 0,
  37.         "servicePort": 10101,
  38.         "protocol": "tcp",
  39.         "name": "httpport",
  40.         "labels": {
  41.           "VIP_0": "/mount-test-svc1:80"
  42.         }
  43.       }
  44.     ]
  45.   },
  46.   "healthChecks": [
  47.     {
  48.       "gracePeriodSeconds": 300,
  49.       "intervalSeconds": 60,
  50.       "timeoutSeconds": 20,
  51.       "maxConsecutiveFailures": 3,
  52.       "portIndex": 0,
  53.       "path": "/",
  54.       "protocol": "MESOS_HTTP",
  55.       "delaySeconds": 15
  56.     }
  57.   ],
  58.   "upgradeStrategy": {
  59.     "minimumHealthCapacity": 0.5,
  60.     "maximumOverCapacity": 0
  61.   },
  62.   "unreachableStrategy": "disabled",
  63.   "killSelection": "YOUNGEST_FIRST",
  64.   "requirePorts": true,
  65.   "labels": {
  66.     "HAPROXY_GROUP": "external"
  67.   }
  68. }

After running this service, navigate to the Services > Volumes tab in the web interface:

Mount disk

Figure 1. Services > Volumes tab

Cloud provider resources

Cloud provider storage services are typically used to back DC/OS mount volumes. This reference material can be useful when designing a production DC/OS deployment:

Best practices

Mount disk resources are primarily for stateful services like Kafka and Cassandra which can benefit from having dedicated storage available throughout the cluster. Any service that uses a Mount disk resource has exclusive access to the reserved resource. However, it is still important to consider the performance and reliability requirements for the service. The performance of a Mount disk resource is based on the characteristic of the underlying storage and DC/OS does not provide any data replication services. Consider the following:

  • Use Mount disk resources with stateful services that have strict storage requirements.
  • Carefully consider the filesystem type, storage media (network attached, SSD, etc.), and volume characteristics (RAID levels, sizing, etc.) based on the storage needs and requirements of the stateful service.
  • Label Mesos agents using a Mesos attribute that reflects the characteristics of the agent’s disk mounts, e.g. IOPS200, RAID1, etc.
  • Associate stateful services with storage agents using Mesos Attribute constraints.
  • Consider isolating demanding storage services to dedicated storage agents, since the filesystem page cache is a host-level shared resource.
  • Ensure that all services using Mount disk resources are designed to handle the permanent loss of one or more Mount disk resources. Services are still responsible for managing data replication and retention, graceful recovery from failed agents, and backups of critical service state.