Running Stateful Services on DC/OS

Tutorial - Running stateful services on DC/OS

IMPORTANT: Tutorials are intended to give you hands-on experience working with a limited set of DC/OS features with no implied or explicit warranty of any kind. None of the information provided—including sample scripts, commands, or applications—is officially supported by Mesosphere. You should not use this information in a production environment without independent testing and validation.

This tutorial shows you how to install and run stateful services on DC/OS. A stateful service acts on persistent data. Simple, stateless services run in an empty sandbox each time they are launched. In contrast, stateful services make use of persistent volumes that reside on agents in a cluster until explicitly destroyed.

These persistent volumes are mounted into a task’s Mesos sandbox and are therefore continuously accessible to a service. DC/OS creates persistent volumes for each task and all resources required to run the task are dynamically reserved. That way, DC/OS ensures that a service can be relaunched and can reuse its data when needed. This is useful for databases, caches, and other data-aware services.

If the service you intend to run does not replicate data on its own, you need to take care of backups or have a suitable replication strategy.

Stateful services leverage two underlying Mesos features:

Time Estimate:

Approximately 20 minutes.

Target Audience:

This tutorial is for developers who want to run stateful services on DC/OS.

NOTE: The DC/OS persistent volume feature is still in beta and is not ready for production use without a data replication strategy to guard against data loss.

Prerequisites

Install a Stateful Service (PostgreSQL)

This is the DC/OS service definition JSON to start the official PostgreSQL Docker image:

  1. {
  2.   "id": "/postgres",
  3.   "cpus": 1,
  4.   "mem": 1024,
  5.   "instances": 1,
  6.   "networks": [
  7.     { "mode": "container/bridge" }
  8.   ],
  9.   "container": {
  10.     "type": "DOCKER",
  11.     "volumes": [
  12.       {
  13.         "containerPath": "pgdata",
  14.         "mode": "RW",
  15.         "persistent": {
  16.           "size": 100
  17.         }
  18.       }
  19.     ],
  20.     "docker": {
  21.       "image": "postgres:9.5"
  22.     },
  23.     "portMappings": [
  24.       {
  25.         "containerPort": 5432,
  26.         "hostPort": 0,
  27.         "protocol": "tcp",
  28.         "labels": {
  29.           "VIP_0": "5.4.3.2:5432"
  30.         }
  31.       }
  32.     ]
  33.   },
  34.   "env": {
  35.     "POSTGRES_PASSWORD": "DC/OS_ROCKS",
  36.     "PGDATA": "/mnt/mesos/sandbox/pgdata"
  37.   },
  38.   "healthChecks": [
  39.     {
  40.       "protocol": "TCP",
  41.       "portIndex": 0,
  42.       "gracePeriodSeconds": 300,
  43.       "intervalSeconds": 60,
  44.       "timeoutSeconds": 20,
  45.       "maxConsecutiveFailures": 3,
  46.       "ignoreHttp1xx": false
  47.     }
  48.   ],
  49.   "upgradeStrategy": {
  50.     "maximumOverCapacity": 0,
  51.     "minimumHealthCapacity": 0
  52.   }
  53. }

Notice the volumes field, which declares the persistent volume for postgres to use for its data. Even if the task dies and restarts, it will get that volume back and data will not be lost.

Next, add this service to your cluster:

  1. dcos marathon app add //tutorials/stateful-services/postgres.marathon.json

Once the service has been scheduled and the Docker container has downloaded, postgres will become healthy and be ready to use. You can verify this from the DC/OS CLI:

  1. dcos marathon task list
  2. APP        HEALTHY          STARTED              HOST     ID
  3. /postgres    True   2016-04-13T17:25:08.301Z  10.0.1.223  postgres.f2419e31-018a-11e6-b721-0261677b407a

Stop the service

To stop the service:

  1. dcos marathon app stop postgres

This command scales the instances count down to 0 and kills all running tasks. If you inspect the tasks list again, you will notice that the task is still there. The list provides information about which agent it was placed on and which persistent volume it had attached, but without a startedAt value. This allows you to restart the service with the same metadata.

  1. dcos marathon task list
  2. APP        HEALTHY  STARTED     HOST     ID
  3. /postgres    True     N/A    10.0.1.223  postgres.f2419e31-018a-11e6-b721-0261677b407a

Restart

Start the stateful service again:

  1. dcos marathon app start postgres

The metadata of the previous postgres task is used to launch a new task that takes over the reservations and volumes of the previously stopped service. Inspect the running task again by repeating the command from the previous step. You will see that the running service task is using the same data as the previous one.

Cleanup

To restore the state of your cluster as it was before installing the stateful service, delete the service:

  1. dcos marathon app remove postgres

Appendix

For further information on stateful services in DC/OS, visit the Storage section of the documentation.