6.18. Redis Connector

The Redis connector allows querying of live data stored in Redis. This can beused to join data between different systems like Redis and Hive.

Each Redis key/value pair is presented as a single row in Presto. Rows can bebroken down into cells by using table definition files.

Only Redis string and hash value types are supported; sets and zsets cannot bequeried from Presto.

The connector requires Redis 2.8.0 or later.

Configuration

To configure the Redis connector, create a catalog properties fileetc/catalog/redis.properties with the following content,replacing the properties as appropriate:

  1. connector.name=redis
  2. redis.table-names=schema1.table1,schema1.table2
  3. redis.nodes=host:port

Multiple Redis Servers

You can have as many catalogs as you need, so if you have additionalRedis servers, simply add another properties file to etc/catalogwith a different name (making sure it ends in .properties).

Configuration Properties

The following configuration properties are available:

Property NameDescription
redis.table-namesList of all tables provided by the catalog
redis.default-schemaDefault schema name for tables
redis.nodesLocation of the Redis server
redis.scan-countRedis parameter for scanning of the keys
redis.key-prefix-schema-tableRedis keys have schema-name:table-name: prefix
redis.key-delimiterDelimiter separating schema_name and table_name if redis.key-prefix-schema-table is used
redis.table-description-dirDirectory containing table description files
redis.hide-internal-columnsControls whether internal columns are part of the table schema or not
redis.database-indexRedis database index
redis.passwordRedis server password

redis.table-names

Comma-separated list of all tables provided by this catalog. A table namecan be unqualified (simple name) and will be put into the default schema(see below) or qualified with a schema name (<schema-name>.<table-name>).

For each table defined here, a table description file (see below) mayexist. If no table description file exists, thetable will only contain internal columns (see below).

This property is required; there is no default and at least one table must bedefined.

redis.default-schema

Defines the schema which will contain all tables that were defined withouta qualifying schema name.

This property is optional; the default is default.

redis.nodes

The hostname:port pair for the Redis server.

This property is required; there is no default.

Redis clusters are not supported.

redis.scan-count

The internal COUNT parameter for Redis SCAN command when connector is usingSCAN to find keys for the data. This parameter can be used to tune performanceof the Redis connector.

This property is optional; the default is 100.

redis.key-prefix-schema-table

If true, only keys prefixed with the schema-name:table-name: are scannedfor a table, and all other keys will be filtered out. If false, all keys arescanned. Note that if the schema-name is default schema, then the prefix tobe scanned is just table-name:.

This property is optional; the default is false.

redis.key-delimiter

The character used for separating schema-name and table-name whenredis.key-prefix-schema-table is true

This property is optional; the default is :.

redis.table-description-dir

References a folder within Presto deployment that holds one or more JSONfiles (must end with .json) which contain table description files.

This property is optional; the default is etc/redis.

redis.hide-internal-columns

In addition to the data columns defined in a table description file, theconnector maintains a number of additional columns for each table. Ifthese columns are hidden, they can still be used in queries but do notshow up in DESCRIBE <table-name> or SELECT *.

This property is optional; the default is true.

redis.database-index

The Redis database to query.

This property is optional; the default is 0.

redis.password

The password for password-protected Redis server.

This property is optional; the default is null.

Internal Columns

For each defined table, the connector maintains the following columns:

Column nameTypeDescription
_keyVARCHARRedis key.
_valueVARCHARRedis value corresponding to the key.
_key_lengthBIGINTNumber of bytes in the key.
_value_lengthBIGINTNumber of bytes in the value.
_key_corruptBOOLEANTrue if the decoder could not decode the key for this row. When true, data columns mapped from the key should be treated as invalid.
_value_corruptBOOLEANTrue if the decoder could not decode the message for this row. When true, data columns mapped from the value should be treated as invalid.

For tables without a table definition file, the _key_corrupt and_value_corrupt columns will always be false.

Table Definition Files

With the Redis connector it’s possible to further reduce Redis key/value pairs intogranular cells provided the key/value sting follow a particular format. This processwill define new columns that can be further queried from Presto.

A table definition file consists of a JSON definition for a table. Thename of the file can be arbitrary but must end in .json.

  1. {
  2. "tableName": ...,
  3. "schemaName": ...,
  4. "key": {
  5. "dataFormat": ...,
  6. "fields": [
  7. ...
  8. ]
  9. },
  10. "value": {
  11. "dataFormat": ...,
  12. "fields": [
  13. ...
  14. ]
  15. }
  16. }
FieldRequiredTypeDescription
tableNamerequiredstringPresto table name defined by this file.
schemaNameoptionalstringSchema which will contain the table. If omitted, the default schema name is used.
keyoptionalJSON objectField definitions for data columns mapped to the value key.
valueoptionalJSON objectField definitions for data columns mapped to the value itself.

Please refer to the Kafka connector page for the description of the dataFormat as well as various available decoders.

In addition to the above Kafka types, the Redis connector supports hash type for the value field which represent data stored in the Redis hash.

  1. {
  2. "tableName": ...,
  3. "schemaName": ...,
  4. "value": {
  5. "dataFormat": "hash",
  6. "fields": [
  7. ...
  8. ]
  9. }
  10. }