Lookups API

This document describes the API endpoints to configure, update, retrieve, and manage lookups for Apache Druid.

Configure lookups

Bulk update

Lookups can be updated in bulk by posting a JSON object to /druid/coordinator/v1/lookups/config. The format of the json object is as follows:

  1. {
  2.     "<tierName>": {
  3.         "<lookupName>": {
  4.           "version": "<version>",
  5.           "lookupExtractorFactory": {
  6.             "type": "<someExtractorFactoryType>",
  7.             "<someExtractorField>": "<someExtractorValue>"
  8.           }
  9.         }
  10.     }
  11. }

Note that “version” is an arbitrary string assigned by the user, when making updates to existing lookup then user would need to specify a lexicographically higher version.

For example, a config might look something like:

  1. {
  2.   "__default": {
  3.     "country_code": {
  4.       "version": "v0",
  5.       "lookupExtractorFactory": {
  6.         "type": "map",
  7.         "map": {
  8.           "77483": "United States"
  9.         }
  10.       }
  11.     },
  12.     "site_id": {
  13.       "version": "v0",
  14.       "lookupExtractorFactory": {
  15.         "type": "cachedNamespace",
  16.         "extractionNamespace": {
  17.           "type": "jdbc",
  18.           "connectorConfig": {
  19.             "createTables": true,
  20.             "connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
  21.             "user": "druid",
  22.             "password": "diurd"
  23.           },
  24.           "table": "lookupTable",
  25.           "keyColumn": "country_id",
  26.           "valueColumn": "country_name",
  27.           "tsColumn": "timeColumn"
  28.         },
  29.         "firstCacheTimeout": 120000,
  30.         "injective": true
  31.       }
  32.     },
  33.     "site_id_customer1": {
  34.       "version": "v0",
  35.       "lookupExtractorFactory": {
  36.         "type": "map",
  37.         "map": {
  38.           "847632": "Internal Use Only"
  39.         }
  40.       }
  41.     },
  42.     "site_id_customer2": {
  43.       "version": "v0",
  44.       "lookupExtractorFactory": {
  45.         "type": "map",
  46.         "map": {
  47.           "AHF77": "Home"
  48.         }
  49.       }
  50.     }
  51.   },
  52.   "realtime_customer1": {
  53.     "country_code": {
  54.       "version": "v0",
  55.       "lookupExtractorFactory": {
  56.         "type": "map",
  57.         "map": {
  58.           "77483": "United States"
  59.         }
  60.       }
  61.     },
  62.     "site_id_customer1": {
  63.       "version": "v0",
  64.       "lookupExtractorFactory": {
  65.         "type": "map",
  66.         "map": {
  67.           "847632": "Internal Use Only"
  68.         }
  69.       }
  70.     }
  71.   },
  72.   "realtime_customer2": {
  73.     "country_code": {
  74.       "version": "v0",
  75.       "lookupExtractorFactory": {
  76.         "type": "map",
  77.         "map": {
  78.           "77483": "United States"
  79.         }
  80.       }
  81.     },
  82.     "site_id_customer2": {
  83.       "version": "v0",
  84.       "lookupExtractorFactory": {
  85.         "type": "map",
  86.         "map": {
  87.           "AHF77": "Home"
  88.         }
  89.       }
  90.     }
  91.   }
  92. }

All entries in the map will UPDATE existing entries. No entries will be deleted.

Update lookup

A POST to a particular lookup extractor factory via /druid/coordinator/v1/lookups/config/{tier}/{id} creates or updates that specific extractor factory.

For example, a post to /druid/coordinator/v1/lookups/config/realtime_customer1/site_id_customer1 might contain the following:

  1. {
  2.   "version": "v1",
  3.   "lookupExtractorFactory": {
  4.     "type": "map",
  5.     "map": {
  6.       "847632": "Internal Use Only"
  7.     }
  8.   }
  9. }

This will replace the site_id_customer1 lookup in the realtime_customer1 with the definition above.

Assign a unique version identifier each time you update a lookup extractor factory. Otherwise the call will fail.

Get all lookups

A GET to /druid/coordinator/v1/lookups/config/all will return all known lookup specs for all tiers.

Get lookup

A GET to a particular lookup extractor factory is accomplished via /druid/coordinator/v1/lookups/config/{tier}/{id}

Using the prior example, a GET to /druid/coordinator/v1/lookups/config/realtime_customer2/site_id_customer2 should return

  1. {
  2.   "version": "v1",
  3.   "lookupExtractorFactory": {
  4.     "type": "map",
  5.     "map": {
  6.       "AHF77": "Home"
  7.     }
  8.   }
  9. }

Delete lookup

A DELETE to /druid/coordinator/v1/lookups/config/{tier}/{id} will remove that lookup from the cluster. If it was last lookup in the tier, then tier is deleted as well.

Delete tier

A DELETE to /druid/coordinator/v1/lookups/config/{tier} will remove that tier from the cluster.

List tier names

A GET to /druid/coordinator/v1/lookups/config will return a list of known tier names in the dynamic configuration. To discover a list of tiers currently active in the cluster in addition to ones known in the dynamic configuration, the parameter discover=true can be added as per /druid/coordinator/v1/lookups/config?discover=true.

List lookup names

A GET to /druid/coordinator/v1/lookups/config/{tier} will return a list of known lookup names for that tier.

These end points can be used to get the propagation status of configured lookups to processes using lookups such as Historicals.

Lookup status

List load status of all lookups

GET /druid/coordinator/v1/lookups/status with optional query parameter detailed.

List load status of lookups in a tier

GET /druid/coordinator/v1/lookups/status/{tier} with optional query parameter detailed.

List load status of single lookup

GET /druid/coordinator/v1/lookups/status/{tier}/{lookup} with optional query parameter detailed.

List lookup state of all processes

GET /druid/coordinator/v1/lookups/nodeStatus with optional query parameter discover to discover tiers advertised by other Druid nodes, or by default, returning all configured lookup tiers. The default response will also include the lookups which are loaded, being loaded, or being dropped on each node, for each tier, including the complete lookup spec. Add the optional query parameter detailed=false to only include the ‘version’ of the lookup instead of the complete spec.

List lookup state of processes in a tier

GET /druid/coordinator/v1/lookups/nodeStatus/{tier}

List lookup state of single process

GET /druid/coordinator/v1/lookups/nodeStatus/{tier}/{host:port}

Internal API

The Peon, Router, Broker, and Historical processes all have the ability to consume lookup configuration. There is an internal API these processes use to list/load/drop their lookups starting at /druid/listen/v1/lookups. These follow the same convention for return values as the cluster wide dynamic configuration. Following endpoints can be used for debugging purposes but not otherwise.

Get lookups

A GET to the process at /druid/listen/v1/lookups will return a json map of all the lookups currently active on the process. The return value will be a json map of the lookups to their extractor factories.

  1. {
  2.   "site_id_customer2": {
  3.     "version": "v1",
  4.     "lookupExtractorFactory": {
  5.       "type": "map",
  6.       "map": {
  7.         "AHF77": "Home"
  8.       }
  9.     }
  10.   }
  11. }

Get lookup

A GET to the process at /druid/listen/v1/lookups/some_lookup_name will return the LookupExtractorFactory for the lookup identified by some_lookup_name. The return value will be the json representation of the factory.

  1. {
  2.   "version": "v1",
  3.   "lookupExtractorFactory": {
  4.     "type": "map",
  5.     "map": {
  6.       "AHF77": "Home"
  7.     }
  8.   }
  9. }