To use this Apache Druid extension, include postgresql-metadata-storage in the extensions load list.

Setting up PostgreSQL

  1. Install PostgreSQL

Use your favorite package manager to install PostgreSQL, e.g.:

  • on Ubuntu/Debian using apt apt-get install postgresql
  • on OS X, using Homebrew brew install postgresql
  1. Create a druid database and user

On the machine where PostgreSQL is installed, using an account with proper postgresql permissions:

Create a druid user, enter diurd when prompted for the password.

  1. createuser druid -P

Create a druid database owned by the user we just created

  1. createdb druid -O druid

Note: On Ubuntu / Debian you may have to prefix the createuser and createdb commands with sudo -u postgres in order to gain proper permissions.

  1. Configure your Druid metadata storage extension:

Add the following parameters to your Druid configuration, replacing <host> with the location (host name and port) of the database.

  1. druid.extensions.loadList=["postgresql-metadata-storage"]
  2. druid.metadata.storage.type=postgresql
  3. druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid
  4. druid.metadata.storage.connector.user=druid
  5. druid.metadata.storage.connector.password=diurd

Configuration

In most cases, the configuration options map directly to the postgres JDBC connection options.

PropertyDescriptionDefaultRequired
druid.metadata.postgres.ssl.useSSLEnables SSLfalseno
druid.metadata.postgres.ssl.sslPasswordThe Password Provider or String password for the client’s key.noneno
druid.metadata.postgres.ssl.sslFactoryThe class name to use as the SSLSocketFactorynoneno
druid.metadata.postgres.ssl.sslFactoryArgAn optional argument passed to the sslFactory’s constructornoneno
druid.metadata.postgres.ssl.sslModeThe sslMode. Possible values are “disable”, “require”, “verify-ca”, “verify-full”, “allow” and “prefer”noneno
druid.metadata.postgres.ssl.sslCertThe full path to the certificate file.noneno
druid.metadata.postgres.ssl.sslKeyThe full path to the key file.noneno
druid.metadata.postgres.ssl.sslRootCertThe full path to the root certificate.noneno
druid.metadata.postgres.ssl.sslHostNameVerifierThe classname of the hostname verifier.noneno
druid.metadata.postgres.ssl.sslPasswordCallbackThe classname of the SSL password provider.noneno
druid.metadata.postgres.dbTableSchemadruid meta table schemapublicno

PostgreSQL Firehose

The PostgreSQL extension provides an implementation of an SqlFirehose which can be used to ingest data into Druid from a PostgreSQL database.

  1. {
  2.   "type": "index_parallel",
  3.   "spec": {
  4.     "dataSchema": {
  5.       "dataSource": "some_datasource",
  6.       "parser": {
  7.         "parseSpec": {
  8.           "format": "timeAndDims",
  9.           "dimensionsSpec": {
  10.             "dimensionExclusions": [],
  11.             "dimensions": [
  12.               "dim1",
  13.               "dim2",
  14.               "dim3"
  15.             ]
  16.           },
  17.           "timestampSpec": {
  18.             "format": "auto",
  19.             "column": "ts"
  20.           }
  21.         }
  22.       },
  23.       "metricsSpec": [],
  24.       "granularitySpec": {
  25.         "type": "uniform",
  26.         "segmentGranularity": "DAY",
  27.         "queryGranularity": {
  28.           "type": "none"
  29.         },
  30.         "rollup": false,
  31.         "intervals": null
  32.       },
  33.       "transformSpec": {
  34.         "filter": null,
  35.         "transforms": []
  36.       }
  37.     },
  38.     "ioConfig": {
  39.       "type": "index_parallel",
  40.       "firehose": {
  41.         "type": "sql",
  42.         "database": {
  43.           "type": "postgresql",
  44.           "connectorConfig": {
  45.             "connectURI": "jdbc:postgresql://some-rds-host.us-west-1.rds.amazonaws.com:5432/druid",
  46.             "user": "admin",
  47.             "password": "secret"
  48.           }
  49.         },
  50.         "sqls": [
  51.           "SELECT * FROM some_table"
  52.         ]
  53.       }
  54.     },
  55.     "tuningConfig": {
  56.       "type": "index_parallel"
  57.     }
  58.   }
  59. }