Setting up M3 Query

M3 Query is used to query data that is stored in M3DB. For instance, if you are using the Prometheus remote write endpoint with m3coordinator, you can use m3query instead of the Prometheus remote read endpoint. By doing so, you get all of the benefits of m3query’s engine such as block processing. Furthermore, since m3query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana.

Configuration

Before setting up m3query, make sure that you have at least one M3DB node running. In order to start m3query, you need to configure a yaml file, that will be used to connect to M3DB. Here is a link to a sample config file that is used for an embedded etcd cluster within M3DB.

Running

You can run m3query by either building and running the binary yourself:

  1. make m3query
  2. ./bin/m3query -f ./src/query/config/m3query-local-etcd.yml

Or you can run it with Docker using the Docker file located at $GOPATH/src/github.com/m3db/m3/docker/m3query/Dockerfile.

Namespaces

All namespaces that you wish to query from must be configured when setting up M3DB. If you wish to add or change an existing namespace, please follow the namespace operational guide here.

etcd

The configuration file linked above uses an embedded etcd cluster, which is fine for development purposes. However, if you wish to use this in production, you will want an external etcd cluster.

Aggregation

You will notice that in the setup linked above, M3DB has just one unaggregated namespace configured. If you want aggregated metrics, you will need to set up an aggregated namespace. It is important to note that all writes go to all namespaces marked as ready. Aggregation is done strictly by the query service. As an example, to configure an aggregated namespace named metrics_10s_48h, you can execute the following API call:

  1. curl -X POST <M3_COORDINATOR_IP_ADDRESS>:<CONFIGURED_PORT(default 7201)>/api/v1/services/m3db/namespace -d '{
  2.   "name": "metrics_10s_48h",
  3.   "options": {
  4.     "bootstrapEnabled": true,
  5.     "flushEnabled": true,
  6.     "writesToCommitLog": true,
  7.     "cleanupEnabled": true,
  8.     "snapshotEnabled": true,
  9.     "repairEnabled": false,
  10.     "retentionOptions": {
  11.       "retentionPeriodDuration": "48h",
  12.       "blockSizeDuration": "2h",
  13.       "bufferFutureDuration": "10m",
  14.       "bufferPastDuration": "10m",
  15.       "blockDataExpiry": true,
  16.       "blockDataExpiryAfterNotAccessedPeriodDuration": "5m"
  17.     },
  18.     "indexOptions": {
  19.       "enabled": true,
  20.       "blockSizeDuration": "2h"
  21.     },
  22.     "aggregationOptions": {
  23.       "aggregations": [
  24.          {
  25.           "aggregated": true,
  26.           "attributes": { "resolutionDuration": "10s" }
  27.         }
  28.       ]
  29.     }
  30.   }
  31. }'

Disabling automatic aggregation

If you run Statsite, m3agg, or some other aggregation tier, you will want to set the all flag under downsample to false. Otherwise, you will be aggregating metrics that have already been aggregated. Using the example above, aggregationOptions would be configured as follows

  1.     ...
  2.     "aggregationOptions": {
  3.       "aggregations": [
  4.         {
  5.           "aggregated": true,
  6.           "attributes": {
  7.             "resolutionDuration": "10s",
  8.             "downsampleOptions": { "all": false }
  9.           }
  10.         }
  11.       ]
  12.     }
  13.     ...

ID generation

As of 1.0, the default generation scheme for IDs is quoted. If you are using the deprecated legacy scheme, please contact the M3 project contributors on the open source Slack channel.

The quoted generation scheme yields the most human-readable IDs, whereas prepend_meta is better for more compact IDs, or if tags are expected to contain non-ASCII characters. To set the ID generation scheme, add the following to your m3coordinator configuration yaml file:

  1. tagOptions:
  2.   idScheme: <name>

As an example of how these schemes generate IDs, consider a series with the following 4 tags, [{"t1":v1}, {t2:"v2"}, {t3:v3}, {t4:v4}]. The following is an example of how different schemes will generate IDs.

  1. legacy: "t1"=v1,t2="v2",t3=v3,t4=v4,
  2. prepend_meta: 4,2,2,4,2,2,2,2!"t1"v1t2"v2"t3v3t4v4
  3. quoted: {\"t1\"="v1",t2="\"v2\"",t3="v3",t4="v4"}

If there is a chance that your metric tags will contain “control” characters, specifically , and =, it is highly recommended that one of either the quoted or prepend_meta schemes are specified, as the legacy scheme may cause ID collisions. As a general guideline, we suggest quoted, as it mirrors the more familiar Prometheus style IDs.

We technically have a fourth ID generation scheme that is used for Graphite IDs, but it is exclusive to the Graphite ingestion path and is not selectable as a general scheme.

WARNING: Once a scheme is selected, be very careful about changing it. If changed, all incoming metrics will resolve to a new ID, effectively doubling the metric cardinality until all of the older-style metric IDs fall out of retention.

Migration

We recently updated our ID generation scheme in m3coordinator to avoid the collision issues discussed above. To ease migration, we’re temporarily enforcing that an ID generation scheme be explicitly provided in the m3coordinator configuration files.

If you have been running m3query or m3coordinator already, you may want to counterintuitively select the collision-prone legacy scheme, as all the IDs for all of your current metrics would have already been generated with this scheme, and choosing another will effectively double your index size. If the twofold increase in cardinality is an acceptable increase (and unfortunately, this is likely to mean doubled cardinality until your longest retention cluster rotates out), it’s suggested to choose a collision-resistant scheme instead.

An example of a configuration file for a standalone m3query instance with the ID generation scheme can be found here. If you’re running m3query or m3coordinator embedded, these configuration options should be nested under the coordinator: heading, as seen here.

If none of these options work for you, or you would like further clarification, please stop by our Slack and we’ll be happy to help you.

Grafana

You can also set up m3query as a datasource in Grafana. To do this, add a new datasource with a type of Prometheus. The URL should point to the host/port running m3query. By default, m3query runs on port 7201.