Cluster Deployments in Mesos, DC/OS

ArangoDB has a sophisticated and yet easy way to use cluster mode. To leverage thefull cluster feature set (monitoring, scaling, automatic failover and automaticreplacement of failed nodes) you might want to run ArangoDB on some kind of clustermanagement system, e.g. Apache Mesos. Mesos is a cluster operating system whichpowers some of the worlds biggest datacenters running several thousands of nodes.

DC/OS

DC/OS eases much of the process to install a Mesos cluster. You can deploy it veryquickly on a variety of cloud hosters or setup your own DC/OS locally. DC/OS isa set of tools built on top of Apache Mesos. Apache Mesos is a so called “DistributedCluster Operation System” and the core of DC/OS. Apache Mesos has the concept ofso called persistent volumeswhich make it perfectly suitable for a database.

Installing

First prepare a DC/OS cluster by going to https://dcos.io and following the instructions there.

DC/OS comes with its own package management. Packages can be installed from theso called “Universe”. As an official DC/OS partner, ArangoDB can be installed fromthere straight away.

  • Installing via DC/OS UI

    • Open your browser and go to the DC/OS admin interface
    • Open the “Universe” tab
    • Locate arangodb and hit “Install Package”
    • Press “Install Package”

  • Installing via the DC/OS command line

    • Install the dcos cli
    • Open a terminal and issue dcos install arangodbBoth options are essentially doing the same in the background. Both are startingArangoDB with its default options set.

To review the default options using the web interface simply click “Advanced Installation”in the web interface. There you will find a list of options including some explanation.

To review the default options using the CLI first type dcos package describe —config arangodb.This will give you a flat list of default settings.

To get an explanation of the various command line options please check the latestoptions here (choose the most recent number and have a look at config.json):

https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/A/arangodb

After installation DC/OS will start deploying the ArangoDB cluster on the DC/OScluster. You can watch ArangoDB starting on the “Services” tab in the web interface.Once it is listed as healthy click the link next to it and you should see the ArangoDBweb interface.

ArangoDB Mesos framework

As soon as ArangoDB was deployed Mesos will keep your cluster running. The web interfacehas many monitoring facilities so be sure to make yourself familiar with the DC/OSweb interface. As a fault tolerant system Mesos will take care of most failurescenarios automatically. Mesos does that by running ArangoDB as a so called “framework”.This framework has been specifically built to keep ArangoDB running in a healthycondition on the Mesos cluster. From time to time a task might fail. The ArangoDBframework will then take care of rescheduling the failed task. As it knows aboutthe very specifics of each cluster task and its role it will automatically takecare of most failure scenarios.

To inspect what the framework is doing go to http://web-interface-url/mesos inyour browser. Locate the task “arangodb” and inspect stderr in the “Sandbox”. Thiscan be of interest for example when a slave got lost and the framework is reschedulingthe task.

Using ArangoDB

To use ArangoDB as a datastore in your DC/OS cluster you can facilitate the servicediscovery of DC/OS. Assuming you deployed a standard ArangoDB cluster themesos dns will know about arangodb.mesos.By doing a SRV DNS request (check the documentation of mesos dns) you can find outthe port where the internal HAProxy of ArangoDB is running. This will offer a roundrobin load balancer to access all ArangoDB coordinators.

Scaling ArangoDB

To change the settings of your ArangoDB Cluster access the ArangoDB UI and hit “Nodes”.On the scale tab you will have the ability to scale your cluster up and down.

After changing the settings the ArangoDB framework will take care of the rest.Scaling your cluster up is generally a straightforward operation as Mesos will simplylaunch another task and be done with it. Scaling down is a bit more complicated asthe data first has to be moved to some other place so that will naturally take somewhatlonger.

Please note that scaling operations might not always work. For example if the underlyingMesos cluster is completely saturated with its running tasks scaling up will notbe possible. Scaling down might also fail due to the cluster not being able to moveall shards of a DBServer to a new destination because of size limitations. Be sureto check the output of the ArangoDB framework.

Deinstallation

Deinstalling ArangoDB is a bit more difficult as there is much state being kept inthe Mesos cluster which is not automatically cleaned up. To deinstall from the commandline use the following one liner:

dcos arangodb uninstall ; dcos package uninstall arangodb

This will first cleanup the state in the cluster and then uninstall arangodb.

arangodb-cleanup-framework

Should you forget to cleanup the state you can do so later by usingthe arangodb-cleanup-frameworkcontainer. Otherwise you might not be able to deploy a new arangodb installation.

The cleanup framework will announce itself as a normal ArangoDB. Mesos will recognizethis and offer all persistent volumes it still has for ArangoDB to this framework.The cleanup framework will then properly free the persistent volumes. Finally itwill clean up any state left in zookeeper (the central configuration manager ina Mesos cluster).

To deploy the cleanup framework, follow the instructions in the github repository.After deployment watch the output in the sandbox of the Mesos web interface. Aftera while there shouldn’t be any persistent resource offers anymore as everythingwas cleaned up. After that you can delete the cleanup framework again via Marathon.

Apache Mesos and Marathon

You can also install ArangoDB on a bare Apache Mesos cluster provided that Marathonis running on it.

Doing so has the following downsides:

  • Manual Mesos cluster setup
  • You need to implement your own service discovery
  • You are missing the dcos cli
  • Installation and deinstallation are tedious
  • You need to setup some kind of proxy tunnel to access ArangoDB from the outside
  • Sparse monitoring capabilitiesHowever these are things which do not influence ArangoDB itself and operating yourcluster like this is fully supported.

Installing via Marathon

To install ArangoDB via marathon you need a proper config file:

  1. {
  2.   "id": "arangodb",
  3.   "cpus": 0.25,
  4.   "mem": 256.0,
  5.   "ports": [0, 0, 0],
  6.   "instances": 1,
  7.   "args": [
  8.     "framework",
  9.     "--framework_name=arangodb",
  10.     "--master=zk://172.17.0.2:2181/mesos",
  11.     "--zk=zk://172.17.0.2:2181/arangodb",
  12.     "--user=",
  13.     "--principal=pri",
  14.     "--role=arangodb",
  15.     "--mode=cluster",
  16.     "--async_replication=true",
  17.     "--minimal_resources_agent=mem(*):512;cpus(*):0.25;disk(*):512",
  18.     "--minimal_resources_dbserver=mem(*):512;cpus(*):0.25;disk(*):1024",
  19.     "--minimal_resources_secondary=mem(*):512;cpus(*):0.25;disk(*):1024",
  20.     "--minimal_resources_coordinator=mem(*):512;cpus(*):0.25;disk(*):1024",
  21.     "--nr_agents=1",
  22.     "--nr_dbservers=2",
  23.     "--nr_coordinators=2",
  24.     "--failover_timeout=86400",
  25.     "--arangodb_image=arangodb/arangodb-mesos:3.1",
  26.     "--secondaries_with_dbservers=false",
  27.     "--coordinators_with_dbservers=false"
  28.   ],
  29.   "container": {
  30.     "type": "DOCKER",
  31.     "docker": {
  32.       "image": "arangodb/arangodb-mesos-framework:3.1",
  33.       "network": "HOST"
  34.     }
  35.   },
  36.   "healthChecks": [
  37.     {
  38.       "protocol": "HTTP",
  39.       "path": "/framework/v1/health.json",
  40.       "gracePeriodSeconds": 3,
  41.       "intervalSeconds": 10,
  42.       "portIndex": 0,
  43.       "timeoutSeconds": 10,
  44.       "maxConsecutiveFailures": 0
  45.     }
  46.   ]
  47. }

Carefully review the settings (especially the IPs and the resources). Then you candeploy to Marathon:

  1. curl -X POST -H "Content-Type: application/json" http://url-of-marathon/v2/apps -d @arangodb3.json

Alternatively use the web interface of Marathon to deploy ArangoDB. It has a JSONmode and you can use the above configuration file.

Deinstallation via Marathon

As with DC/OS you first need to properly cleanup any state leftovers.

The easiest is to simply delete ArangoDB and then deploy the cleanup-framework(see section arangodb-cleanup-framework).

Configuration options

The Arangodb Mesos framework has a ton of different options which are listed anddescribed here: https://github.com/arangodb/arangodb-mesos-framework/tree/3.2