Request logging

All Apache Druid services that can serve queries can also log the query requests they process. Request logs contain information on query metrics, including execution time and memory usage. You can use information in the request logs to monitor query performance, determine bottlenecks, and analyze and improve slow queries.

Request logging is disabled by default. This topic describes how to configure Druid to generate request logs to track query metrics.

Configure request logging

To enable request logging, determine the type of request logger to use, then set the configurations specific to the request logger type.

The following request logger types are available:

  • noop: Disables request logging, the default behavior.
  • file: Stores logs to disk.
  • emitter: Logs request to an external location, which is configured through an emitter.
  • slf4j: Logs queries via the SLF4J Java logging API.
  • filtered: Filters requests by query type or execution time before logging the filtered queries by the delegated request logger.
  • composing: Logs all requests to multiple request loggers.
  • switching: Logs native queries and SQL queries to separate request loggers.

Define the type of request logger in druid.request.logging.type. See the Request logging configuration for properties to set for each type of request logger. Specify these properties in the common.runtime.properties file. You must restart Druid for the changes to take effect.

Druid stores the results in the Broker logs, unless the request logging type is emitter. If you use emitter request logging, you must also configure metrics emission.

Configure metrics emission

Druid includes various emitters to send metrics and alerts. To emit query metrics, set druid.request.logging.feed=emitter, and define the emitter type in the druid.emitter property.

You can use any of the following emitters in Druid:

  • noop: Disables metric emission, the default behavior.
  • logging: Emits metrics to Log4j 2. See Logging to configure Log4j 2 for use with Druid.
  • http: Sends HTTP POST requests containing the metrics in JSON format to a user-defined endpoint.
  • parametrized: Operates like the http emitter but fine-tunes the recipient URL based on the event feed.
  • composing: Emits metrics to multiple emitter types.
  • graphite: Emits metrics to a Graphite Carbon service.

Specify these properties in the common.runtime.properties file. See the Metrics emitters configuration for properties to set for each type of metrics emitter. You must restart Druid for the changes to take effect.

Example

The following configuration shows how to enable request logging and post query metrics to the endpoint http://example.com:8080/path.

  1. # Enable request logging and configure the emitter request logger
  2. druid.request.logging.type=emitter
  3. druid.request.logging.feed=myRequestLogFeed
  5. # Enable metrics emission and tell Druid where to emit messages
  6. druid.emitter=http
  7. druid.emitter.http.recipientBaseUrl=http://example.com:8080/path
  9. # Authenticate to the base URL, if needed
  10. druid.emitter.http.basicAuthentication=username:password

The following shows an example log emitter output:

  1. [
  2.     {
  3.         "feed": "metrics",
  4.         "timestamp": "2022-01-06T20:32:06.628Z",
  5.         "service": "druid/broker",
  6.         "host": "localhost:8082",
  7.         "version": "2022.01.0-iap-SNAPSHOT",
  8.         "metric": "sqlQuery/bytes",
  9.         "value": 9351,
  10.         "dataSource": "[wikipedia]",
  11.         "id": "56e8317b-31cc-443d-b109-47f51b21d4c3",
  12.         "nativeQueryIds": "[2b9cbced-11fc-4d78-a58c-c42863dff3c8]",
  13.         "remoteAddress": "127.0.0.1",
  14.         "success": "true"
  15.     },
  16.     {
  17.         "feed": "myRequestLogFeed",
  18.         "timestamp": "2022-01-06T20:32:06.585Z",
  19.         "remoteAddr": "127.0.0.1",
  20.         "service": "druid/broker",
  21.         "sqlQueryContext":
  22.         {
  23.             "useApproximateCountDistinct": false,
  24.             "sqlQueryId": "56e8317b-31cc-443d-b109-47f51b21d4c3",
  25.             "useApproximateTopN": false,
  26.             "useCache": false,
  27.             "sqlOuterLimit": 101,
  28.             "populateCache": false,
  29.             "nativeQueryIds": "[2b9cbced-11fc-4d78-a58c-c42863dff3c8]"
  30.         },
  31.         "queryStats":
  32.         {
  33.             "sqlQuery/time": 43,
  34.             "sqlQuery/bytes": 9351,
  35.             "success": true,
  36.             "context":
  37.             {
  38.                 "useApproximateCountDistinct": false,
  39.                 "sqlQueryId": "56e8317b-31cc-443d-b109-47f51b21d4c3",
  40.                 "useApproximateTopN": false,
  41.                 "useCache": false,
  42.                 "sqlOuterLimit": 101,
  43.                 "populateCache": false,
  44.                 "nativeQueryIds": "[2b9cbced-11fc-4d78-a58c-c42863dff3c8]"
  45.             },
  46.             "identity": "allowAll"
  47.         },
  48.         "query": null,
  49.         "host": "localhost:8082",
  50.         "sql": "SELECT * FROM wikipedia WHERE cityName = 'Buenos Aires'"
  51.     },
  52.     {
  53.         "feed": "myRequestLogFeed",
  54.         "timestamp": "2022-01-06T20:32:07.652Z",
  55.         "remoteAddr": "",
  56.         "service": "druid/broker",
  57.         "sqlQueryContext":
  58.         {},
  59.         "queryStats":
  60.         {
  61.             "query/time": 16,
  62.             "query/bytes": -1,
  63.             "success": true,
  64.             "identity": "allowAll"
  65.         },
  66.         "query":
  67.         {
  68.             "queryType": "scan",
  69.             "dataSource":
  70.             {
  71.                 "type": "table",
  72.                 "name": "wikipedia"
  73.             },
  74.             "intervals":
  75.             {
  76.                 "type": "intervals",
  77.                 "intervals":
  78.                 [
  79.                     "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
  80.                 ]
  81.             },
  82.             "virtualColumns":
  83.             [
  84.                 {
  85.                     "type": "expression",
  86.                     "name": "v0",
  87.                     "expression": "'Buenos Aires'",
  88.                     "outputType": "STRING"
  89.                 }
  90.             ],
  91.             "resultFormat": "compactedList",
  92.             "batchSize": 20480,
  93.             "limit": 101,
  94.             "filter":
  95.             {
  96.                 "type": "selector",
  97.                 "dimension": "cityName",
  98.                 "value": "Buenos Aires",
  99.                 "extractionFn": null
  100.             },
  101.             "columns":
  102.             [
  103.                 "__time",
  104.                 "added",
  105.                 "channel",
  106.                 "comment",
  107.                 "commentLength",
  108.                 "countryIsoCode",
  109.                 "countryName",
  110.                 "deleted",
  111.                 "delta",
  112.                 "deltaBucket",
  113.                 "diffUrl",
  114.                 "flags",
  115.                 "isAnonymous",
  116.                 "isMinor",
  117.                 "isNew",
  118.                 "isRobot",
  119.                 "isUnpatrolled",
  120.                 "metroCode",
  121.                 "namespace",
  122.                 "page",
  123.                 "regionIsoCode",
  124.                 "regionName",
  125.                 "user",
  126.                 "v0"
  127.             ],
  128.             "legacy": false,
  129.             "context":
  130.             {
  131.                 "populateCache": false,
  132.                 "queryId": "62e3d373-6e50-41b4-873b-1e56347c2950",
  133.                 "sqlOuterLimit": 101,
  134.                 "sqlQueryId": "cbb3d519-aee9-4566-8920-dbbeab6269f5",
  135.                 "useApproximateCountDistinct": false,
  136.                 "useApproximateTopN": false,
  137.                 "useCache": false
  138.             },
  139.             "descending": false,
  140.             "granularity":
  141.             {
  142.                 "type": "all"
  143.             }
  144.         },
  145.         "host": "localhost:8082",
  146.         "sql": null
  147.     },
  148.     ...
  149. ]

Learn more

See the following topics for more information.