Memory Connector

The Memory connector stores all data and metadata in RAM on workers and both are discarded when openLooKeng restarts.

Configuration

To configure the Memory connector, create a catalog properties file etc/catalog/memory.properties with the following contents:

  1. connector.name=memory
  2. memory.max-data-per-node=128MB

memory.max-data-per-node defines memory limit for pages stored in this connector per each node (default value is 128MB).

Examples

Create a table using the Memory connector:

  1. CREATE TABLE memory.default.nation AS
  2. SELECT * from tpch.tiny.nation;

Insert data into a table in the Memory connector:

  1. INSERT INTO memory.default.nation
  2. SELECT * FROM tpch.tiny.nation;

Select from the Memory connector:

  1. SELECT * FROM memory.default.nation;

Drop table:

  1. DROP TABLE memory.default.nation;

Memory Connector Limitations

  • After DROP TABLE memory is not released immediately. It is released after next write access to memory connector.
  • When one worker fails/restarts all data that were stored in its memory will be lost forever. To prevent silent data loss this connector will throw an error on any read access to such corrupted table.
  • When query fails for any reason during writing to memory table, table will be in undefined state. Such table should be dropped and recreated manually. Reading attempt from such table may fail or may return partial data.
  • When coordinator fails/restarts all metadata about tables will be lost, but tables’ data will be still present on the workers however they will be inaccessible.
  • This connector will not work properly with multiple coordinators, since each coordinator will have a different metadata.