DistinctCount Aggregator

To use this Apache Druid extension, make sure to include the druid-distinctcount extension.

Additionally, follow these steps:

  1. First, use a single dimension hash-based partition spec to partition data by a single dimension. For example visitor_id. This to make sure all rows with a particular value for that dimension will go into the same segment, or this might over count.
  2. Second, use distinctCount to calculate the distinct count, make sure queryGranularity is divided exactly by segmentGranularity or else the result will be wrong.

There are some limitations, when used with groupBy, the groupBy keys’ numbers should not exceed maxIntermediateRows in every segment. If exceeded the result will be wrong. When used with topN, numValuesPerPass should not be too big. If too big the distinctCount will use a lot of memory and might cause the JVM to go our of memory.

Example:

Timeseries query

  1. {
  2.   "queryType": "timeseries",
  3.   "dataSource": "sample_datasource",
  4.   "granularity": "day",
  5.   "aggregations": [
  6.     {
  7.       "type": "distinctCount",
  8.       "name": "uv",
  9.       "fieldName": "visitor_id"
  10.     }
  11.   ],
  12.   "intervals": [
  13.     "2016-03-01T00:00:00.000/2013-03-20T00:00:00.000"
  14.   ]
  15. }

TopN query

  1. {
  2.   "queryType": "topN",
  3.   "dataSource": "sample_datasource",
  4.   "dimension": "sample_dim",
  5.   "threshold": 5,
  6.   "metric": "uv",
  7.   "granularity": "all",
  8.   "aggregations": [
  9.     {
  10.       "type": "distinctCount",
  11.       "name": "uv",
  12.       "fieldName": "visitor_id"
  13.     }
  14.   ],
  15.   "intervals": [
  16.     "2016-03-06T00:00:00/2016-03-06T23:59:59"
  17.   ]
  18. }

GroupBy query

  1. {
  2.   "queryType": "groupBy",
  3.   "dataSource": "sample_datasource",
  4.   "dimensions": "[sample_dim]",
  5.   "granularity": "all",
  6.   "aggregations": [
  7.     {
  8.       "type": "distinctCount",
  9.       "name": "uv",
  10.       "fieldName": "visitor_id"
  11.     }
  12.   ],
  13.   "intervals": [
  14.     "2016-03-06T00:00:00/2016-03-06T23:59:59"
  15.   ]
  16. }