Creating an M3 Cluster with Binaries

This guide shows you the steps involved in creating an M3 cluster using M3 binaries, typically you would automate this with infrastructure as code tools such as Terraform or Kubernetes.

This guide assumes you have read the quickstart, and builds upon the concepts in that guide.

M3 Architecture

Here’s a typical M3 deployment:

Typical Deployment

An M3 deployment typically has two main node types:

  • Coordinator node: m3coordinator nodes coordinate reads and writes across all nodes in the cluster. It’s a lightweight process, and does not store any data. This role typically runs alongside a Prometheus instance, or is part of a collector agent such as statsD.
  • Storage node: The m3dbnode processes are the workhorses of M3, they store data and serve reads and writes.

A m3coordinator node exposes two ports:

  • 7201 to manage the cluster topology, you make most API calls to this endpoint
  • 7203 for Prometheus to scrape the metrics produced by M3DB and M3Coordinator

Prerequisites

M3 uses etcd as a distributed key-value storage for the following functions:

  • Update cluster configuration in realtime
  • Manage placements for distributed and sharded clusters

Download and Install a Binary

You can download the latest release as pre-compiled binaries from the M3 GitHub page. Inside the expanded archive are binaries for m3dbnode, which combines a coordinator and storage node, and a binary for m3coordinator, which is a standalone coordinator node.

Build from Source

Prerequisites

Build

Clone the codebase and run make m3dbnode to generate a binary for a combination coordinator and storage node, or make m3coordinator to generate a binary for a standalone coordinator node.

Provision a Host

Enough background, let’s create a real cluster!

M3 in production can run on local or cloud-based VMs, or bare-metal servers. M3 supports all popular Linux distributions (Ubuntu, RHEL, CentOS), and let us know if you have any issues with your preferred distribution.

Network

If you use AWS or GCP, we recommend you use static IPs so that if you need to replace a host, you don’t have to update configuration files on all the hosts, but decommission the old seed node and provision a new seed node with the same host ID and static IP that the old seed node had. If you’re using AWS you can use an Elastic Network Interface on a Virtual Private Cloud (VPC) and for GCP you can use an internal static IP address.

This example creates three static IP addresses for three storage nodes, using the embedded coordinator.

This guide assumes you have host names configured, i.e., running hostname on a host in the cluster returns the host ID you use when creating the cluster placement.

When using GCP the name of your instance is the host name. When you create an instance, click Management, disks, networking, SSH keys, under Networking, click the default interface, click the Primary internal IP drop down, select Reserve a static internal IP address, give it an appropriate name and description, and use Assign automatically.

When using AWS, you can use the host name supplied for the provisioned VM as your host ID, or use the environment host ID resolver and pass the host ID when launching the database process with an environment variable.

For example, if you used M3DB_HOST_ID for the environment variable name, use the following in your configuration:

  1. hostID:
  2.   resolver: config
  3.   value: ${M3DB_HOST_ID:""}

Then start the m3dbnode process with:

  1. M3DB_HOST_ID=m3db001 m3dbnode -f <config-file.yml>

Depending on your operating system setup, you might need to prefix the command with sudo.

Docker pull and run

Kernel Configuration

Depending on the default limits of your bare-metal machine or VM, M3 may need some Kernel tweaks to run as efficiently as possible, and we recommend you review those before running M3 in production.

Configuration files

You configure each M3 component by passing the location of a YAML file with the -f argument.

The file groups configuration into coordinator or db sections that represent the M3Coordinator and M3DB instances of single-node cluster.

You can find more information on configuring M3DB in the operational guides section.

The steps in this guide have the following 3 seed nodes, you need to change your configuration to suit the details of yours, including the details of an etcd cluster in the m3dbCluster > endpoints section of the M3 configuration file.

  • m3db001 (Region=us-east1, Zone=us-east1-a, Static IP=10.142.0.1)
  • m3db002 (Region=us-east1, Zone=us-east1-b, Static IP=10.142.0.2)
  • m3db003 (Region=us-east1, Zone=us-east1-c, Static IP=10.142.0.3)

M3DB node

Start with the M3DB configuration template and change it to suit your cluster.

The example below connects to an etcd instance in a zone called eu-1 This example updates the service and seedNodes sections to match the node details above:

  1. config:
  2.   discovery: 
  3.     type: m3db_cluster 
  4.     m3dbCluster: 
  5.       env: default_env
  6.       endpoints:
  7.         - 10.142.0.1:2379
  8.         - 10.142.0.2:2379
  9.         - 10.142.0.3:2379

Start the storage nodes

Start each seed node in the cluster using the same configuration file, and adjusting the M3DB_HOST_ID value to match the host name.

  1. M3DB_HOST_ID=m3db001 m3dbnode -f <config-file.yml>

You can daemon-ize the node startup process using your favorite utility such as systemd, init.d, or supervisor.

Create Namespace and Initialize Placement

This guide uses the http://localhost:7201/api/v1/database/create endpoint that creates a namespace, and the placement if it doesn’t already exist based on the type argument.

You can create placements and namespaces separately if you need more control over their settings.

In the example below, the configuration for each host matches the details outlined above for the three nodes used. isolationGroup specifies how the cluster places shards to avoid more than one replica of a shard appearing in the same replica group. You should use at least as many isolation groups as your replication factor. This example uses the availability zones us-east1-a, us-east1-b, us-east1-c as the isolation groups which matches our replication factor of 3. Read more details in this guide.

  1. curl -X POST http://localhost:7201/api/v1/database/create -d '{
  2.   "type": "cluster",
  3.   "namespaceName": "default",
  4.   "retentionTime": "48h",
  5.   "numShards": "1024",
  6.   "replicationFactor": "3",
  7.   "hosts": [
  8.         {
  9.             "id": "m3db001",
  10.             "isolationGroup": "us-east1-a",
  11.             "zone": "embedded",
  12.             "weight": 100,
  13.             "address": "10.142.0.1",
  14.             "port": 9000
  15.         },
  16.         {
  17.             "id": "m3db002",
  18.             "isolationGroup": "us-east1-b",
  19.             "zone": "embedded",
  20.             "weight": 100,
  21.             "address": "10.142.0.2",
  22.             "port": 9000
  23.         },
  24.         {
  25.             "id": "m3db003",
  26.             "isolationGroup": "us-east1-c",
  27.             "zone": "embedded",
  28.             "weight": 100,
  29.             "address": "10.142.0.3",
  30.             "port": 9000
  31.         }
  32.     ]
  33. }'
  1. 20:10:12.911218[I] updating database namespaces [{adds [default]} {updates []} {removals []}]
  2. 20:10:13.462798[I] node tchannelthrift: listening on 0.0.0.0:9000
  3. 20:10:13.463107[I] cluster tchannelthrift: listening on 0.0.0.0:9001
  4. 20:10:13.747173[I] node httpjson: listening on 0.0.0.0:9002
  5. 20:10:13.747506[I] cluster httpjson: listening on 0.0.0.0:9003
  6. 20:10:13.747763[I] bootstrapping shards for range starting ...
  7. ...
  8. 20:10:13.757834[I] bootstrap finished [{namespace metrics} {duration 10.1261ms}]
  9. 20:10:13.758001[I] bootstrapped
  10. 20:10:14.764771[I] successfully updated topology to 3 hosts

If you need to setup multiple namespaces, you can run the command above multiple times with different namespace configurations.

Ready a Namespace

Once a namespace has finished bootstrapping, you must mark it as ready before receiving traffic by using the http://localhost:7201/api/v1/namespace/ready.

  1. #!/bin/bash
  2. curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
  3.   "name": "default"
  4. }' | jq .
  1. {
  2.   "ready": true
  3. }

Replication factor

We recommend a replication factor of 3, with each replica spread across failure domains such as a physical server rack, data center or availability zone. Read our replication factor recommendations for more details.

Shards

Read the placement configuration guide to determine the appropriate number of shards to specify.

Writing and Querying Metrics

Writing Metrics

M3 supports ingesting statsd and Prometheus formatted metrics.

This quickstart focuses on Prometheus metrics which consist of a value, a timestamp, and tags to bring context and meaning to the metric.

You can write metrics using one of two endpoints:

For this quickstart, use the http://localhost:7201/api/v1/json/write endpoint to write a tagged metric to M3 with the following data in the request body, all fields are required:

  • tags: An object of at least one name/value pairs
  • timestamp: The UNIX timestamp for the data
  • value: The value for the data, can be of any type

The examples below use __name__ as the name for one of the tags, which is a Prometheus reserved tag that allows you to query metrics using the value of the tag to filter results.

Label names may contain ASCII letters, numbers, underscores, and Unicode characters. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label names beginning with __ are reserved for internal use. Read more in the Prometheus documentation.

  1. #!/bin/bash
  2. curl -X POST http://localhost:7201/api/v1/json/write -d '{
  3.   "tags": 
  4.     {
  5.       "__name__": "third_avenue",
  6.       "city": "new_york",
  7.       "checkout": "1"
  8.     },
  9.     "timestamp": '\"$(date "+%s")\"',
  10.     "value": 3347.26
  11. }'
  1. #!/bin/bash
  2. curl -X POST http://localhost:7201/api/v1/json/write -d '{
  3.   "tags": 
  4.     {
  5.       "__name__": "third_avenue",
  6.       "city": "new_york",
  7.       "checkout": "1"
  8.     },
  9.     "timestamp": '\"$(date "+%s")\"',
  10.     "value": 5347.26
  11. }'
  1. #!/bin/bash
  2. curl -X POST http://localhost:7201/api/v1/json/write -d '{
  3.   "tags": 
  4.     {
  5.       "__name__": "third_avenue",
  6.       "city": "new_york",
  7.       "checkout": "1"
  8.     },
  9.     "timestamp": '\"$(date "+%s")\"',
  10.     "value": 7347.26
  11. }'

Querying metrics

M3 supports three query engines: Prometheus (default), Graphite, and the M3 Query Engine.

This quickstart uses Prometheus as the query engine, and you have access to all the features of PromQL queries.

To query metrics, use the http://localhost:7201/api/v1/query\_range endpoint with the following data in the request body, all fields are required:

  • query: A PromQL query
  • start: Timestamp in RFC3339Nano of start range for results
  • end: Timestamp in RFC3339Nano of end range for results
  • step: A duration or float of the query resolution, the interval between results in the timespan between start and end.

Below are some examples using the metrics written above.

Return results in past 45 seconds

  1. curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  2.   -d "query=third_avenue" \
  3.   -d "start=$(date "+%s" -d "45 seconds ago")" \
  4.   -d "end=$( date +%s )" \
  5.   -d "step=5s" | jq .
  1. curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  2.   -d "query=third_avenue" \
  3.   -d "start=$( date -v -45S +%s )" \
  4.   -d "end=$( date +%s )" \
  5.   -d "step=5s" | jq .
  1. {
  2.   "status": "success",
  3.   "data": {
  4.     "resultType": "matrix",
  5.     "result": [
  6.       {
  7.         "metric": {
  8.           "__name__": "third_avenue",
  9.           "checkout": "1",
  10.           "city": "new_york"
  11.         },
  12.         "values": [
  13.           [
  14.             1610746220,
  15.             "3347.26"
  16.           ],
  17.           [
  18.             1610746220,
  19.             "5347.26"
  20.           ],
  21.           [
  22.             1610746220,
  23.             "7347.26"
  24.           ]
  25.         ]
  26.       }
  27.     ]
  28.   }
  29. }

Values above a certain number

  1. curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  2.   -d "query=third_avenue > 6000" \
  3.   -d "start=$(date "+%s" -d "45 seconds ago")" \
  4.   -d "end=$( date +%s )" \
  5.   -d "step=5s" | jq .
  1. curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
  2.   -d "query=third_avenue > 6000" \
  3.   -d "start=$(date -v -45S "+%s")" \
  4.   -d "end=$( date +%s )" \
  5.   -d "step=5s" | jq .
  1. {
  2.   "status": "success",
  3.   "data": {
  4.     "resultType": "matrix",
  5.     "result": [
  6.       {
  7.         "metric": {
  8.           "__name__": "third_avenue",
  9.           "checkout": "1",
  10.           "city": "new_york"
  11.         },
  12.         "values": [
  13.           [
  14.             1610746220,
  15.             "7347.26"
  16.           ]
  17.         ]
  18.       }
  19.     ]
  20.   }
  21. }