5.18. Thrift Connector

The Thrift connector makes it possible to integrate with external storage systemswithout a custom Presto connector implementation.

In order to use the Thrift connector with an external system, you need to implementthe PrestoThriftService interface, found below. Next, you configure the Thrift connectorto point to a set of machines, called Thrift servers, that implement the interface.As part of the interface implementation, the Thrift servers will provide metadata,splits and data. The Thrift server instances are assumed to be stateless and independentfrom each other.

Configuration

To configure the Thrift connector, create a catalog properties fileetc/catalog/thrift.properties with the following content,replacing the properties as appropriate:

connector.name=presto-thrift
presto.thrift.client.addresses=host:port,host:port

Multiple Thrift Systems

You can have as many catalogs as you need, so if you have additionalThrift systems to connect to, simply add another properties file to etc/catalogwith a different name (making sure it ends in .properties).

Configuration Properties

The following configuration properties are available:

presto.thrift.client.addresses

Comma-separated list of thrift servers in the form of host:port. For example:

presto.thrift.client.addresses=192.0.2.3:7777,192.0.2.4:7779

This property is required; there is no default.

presto-thrift.max-response-size

Maximum size of a data response that the connector accepts. This value is sentby the connector to the Thrift server when requesting data, allowing it to sizethe response appropriately.

This property is optional; the default is 16MB.

presto-thrift.metadata-refresh-threads

Number of refresh threads for metadata cache.

This property is optional; the default is 1.

Thrift IDL File

The following IDL describes the PrestoThriftService that must be implemented:

enumPrestoThriftBound{
BELOW=1;
EXACTLY=2;
ABOVE=3;
}
 
exceptionPrestoThriftServiceException{
1:stringmessage;
2:boolretryable;
}
 
structPrestoThriftNullableSchemaName{
1:optionalstringschemaName;
}
 
structPrestoThriftSchemaTableName{
1:stringschemaName;
2:stringtableName;
}
 
structPrestoThriftTableMetadata{
1:PrestoThriftSchemaTableNameschemaTableName;
2:list<PrestoThriftColumnMetadata>columns;
3:optionalstringcomment;
 
/**
   * Returns a list of key sets which can be used for index lookups.
   * The list is expected to have only unique key sets.
   * {@code set<set<string>>} is not used here because some languages (like php) don't support it.
   */
4:optionallist<set<string>>indexableKeys;
}
 
structPrestoThriftColumnMetadata{
1:stringname;
2:stringtype;
3:optionalstringcomment;
4:boolhidden;
}
 
structPrestoThriftNullableColumnSet{
1:optionalset<string>columns;
}
 
structPrestoThriftTupleDomain{
/**
   * Return a map of column names to constraints.
   */
1:optionalmap<string,PrestoThriftDomain>domains;
}
 
/**
 * Set that either includes all values, or excludes all values.
 */
structPrestoThriftAllOrNoneValueSet{
1:boolall;
}
 
/**
 * A set containing values that are uniquely identifiable.
 * Assumes an infinite number of possible values. The values may be collectively included (aka whitelist)
 * or collectively excluded (aka !whitelist).
 * This structure is used with comparable, but not orderable types like "json", "map".
 */
structPrestoThriftEquatableValueSet{
1:boolwhiteList;
2:list<PrestoThriftBlock>values;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code ints} array are values for each row. If row is null then value is ignored.
 */
structPrestoThriftInteger{
1:optionallist<bool>nulls;
2:optionallist<i32>ints;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code longs} array are values for each row. If row is null then value is ignored.
 */
structPrestoThriftBigint{
1:optionallist<bool>nulls;
2:optionallist<i64>longs;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code doubles} array are values for each row. If row is null then value is ignored.
 */
structPrestoThriftDouble{
1:optionallist<bool>nulls;
2:optionallist<double>doubles;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
 * If row is null then the corresponding element in {@code sizes} is ignored.
 * {@code bytes} array contains UTF-8 encoded byte values.
 * Values for all rows are written to {@code bytes} array one after another.
 * The total number of bytes must be equal to the sum of all sizes.
 */
structPrestoThriftVarchar{
1:optionallist<bool>nulls;
2:optionallist<i32>sizes;
3:optionalbinarybytes;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code booleans} array are values for each row. If row is null then value is ignored.
 */
structPrestoThriftBoolean{
1:optionallist<bool>nulls;
2:optionallist<bool>booleans;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code dates} array are date values for each row represented as the number
 * of days passed since 1970-01-01.
 * If row is null then value is ignored.
 */
structPrestoThriftDate{
1:optionallist<bool>nulls;
2:optionallist<i32>dates;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Elements of {@code timestamps} array are values for each row represented as the number
 * of milliseconds passed since 1970-01-01T00:00:00 UTC.
 * If row is null then value is ignored.
 */
structPrestoThriftTimestamp{
1:optionallist<bool>nulls;
2:optionallist<i64>timestamps;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
 * If row is null then the corresponding element in {@code sizes} is ignored.
 * {@code bytes} array contains UTF-8 encoded byte values for string representation of json.
 * Values for all rows are written to {@code bytes} array one after another.
 * The total number of bytes must be equal to the sum of all sizes.
 */
structPrestoThriftJson{
1:optionallist<bool>nulls;
2:optionallist<i32>sizes;
3:optionalbinarybytes;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
 * If row is null then the corresponding element in {@code sizes} is ignored.
 * {@code bytes} array contains encoded byte values for HyperLogLog representation as defined in
 * Airlift specification: href="https://github.com/airlift/airlift/blob/master/stats/docs/hll.md
 * Values for all rows are written to {@code bytes} array one after another.
 * The total number of bytes must be equal to the sum of all sizes.
 */
structPrestoThriftHyperLogLog{
1:optionallist<bool>nulls;
2:optionallist<i32>sizes;
3:optionalbinarybytes;
}
 
/**
 * Elements of {@code nulls} array determine if a value for a corresponding row is null.
 * Each elements of {@code sizes} array contains the number of elements in the corresponding values array.
 * If row is null then the corresponding element in {@code sizes} is ignored.
 * {@code values} is a bigint block containing array elements one after another for all rows.
 * The total number of elements in bigint block must be equal to the sum of all sizes.
 */
structPrestoThriftBigintArray{
1:optionallist<bool>nulls;
2:optionallist<i32>sizes;
3:optionalPrestoThriftBigintvalues;
}
 
/**
 * A set containing zero or more Ranges of the same type over a continuous space of possible values.
 * Ranges are coalesced into the most compact representation of non-overlapping Ranges.
 * This structure is used with comparable and orderable types like bigint, integer, double, varchar, etc.
 */
structPrestoThriftRangeValueSet{
1:list<PrestoThriftRange>ranges;
}
 
structPrestoThriftId{
1:binaryid;
}
 
structPrestoThriftSplitBatch{
1:list<PrestoThriftSplit>splits;
2:optionalPrestoThriftIdnextToken;
}
 
structPrestoThriftSplit{
1:PrestoThriftIdsplitId;
2:list<PrestoThriftHostAddress>hosts;
}
 
structPrestoThriftHostAddress{
1:stringhost;
2:i32port;
}
 
structPrestoThriftPageResult{
/**
   * Returns data in a columnar format.
   * Columns in this list must be in the order they were requested by the engine.
   */
1:list<PrestoThriftBlock>columnBlocks;
 
2:i32rowCount;
3:optionalPrestoThriftIdnextToken;
}
 
structPrestoThriftNullableTableMetadata{
1:optionalPrestoThriftTableMetadatatableMetadata;
}
 
structPrestoThriftValueSet{
1:optionalPrestoThriftAllOrNoneValueSetallOrNoneValueSet;
2:optionalPrestoThriftEquatableValueSetequatableValueSet;
3:optionalPrestoThriftRangeValueSetrangeValueSet;
}
 
structPrestoThriftBlock{
1:optionalPrestoThriftIntegerintegerData;
2:optionalPrestoThriftBigintbigintData;
3:optionalPrestoThriftDoubledoubleData;
4:optionalPrestoThriftVarcharvarcharData;
5:optionalPrestoThriftBooleanbooleanData;
6:optionalPrestoThriftDatedateData;
7:optionalPrestoThriftTimestamptimestampData;
8:optionalPrestoThriftJsonjsonData;
9:optionalPrestoThriftHyperLogLoghyperLogLogData;
10:optionalPrestoThriftBigintArraybigintArrayData;
}
 
/**
 * LOWER UNBOUNDED is specified with an empty value and an ABOVE bound
 * UPPER UNBOUNDED is specified with an empty value and a BELOW bound
 */
structPrestoThriftMarker{
1:optionalPrestoThriftBlockvalue;
2:PrestoThriftBoundbound;
}
 
structPrestoThriftNullableToken{
1:optionalPrestoThriftIdtoken;
}
 
structPrestoThriftDomain{
1:PrestoThriftValueSetvalueSet;
2:boolnullAllowed;
}
 
structPrestoThriftRange{
1:PrestoThriftMarkerlow;
2:PrestoThriftMarkerhigh;
}
 
/**
 * Presto Thrift service definition.
 * This thrift service needs to be implemented in order to be used with Thrift Connector.
 */
servicePrestoThriftService{
/**
   * Returns available schema names.
   */
list<string>prestoListSchemaNames()
throws(1:PrestoThriftServiceExceptionex1);
 
/**
   * Returns tables for the given schema name.
   *
   * @param schemaNameOrNull a structure containing schema name or {@literal null}
   * @return a list of table names with corresponding schemas. If schema name is null then returns
   * a list of tables for all schemas. Returns an empty list if a schema does not exist
   */
list<PrestoThriftSchemaTableName>prestoListTables(
1:PrestoThriftNullableSchemaNameschemaNameOrNull)
throws(1:PrestoThriftServiceExceptionex1);
 
/**
   * Returns metadata for a given table.
   *
   * @param schemaTableName schema and table name
   * @return metadata for a given table, or a {@literal null} value inside if it does not exist
   */
PrestoThriftNullableTableMetadataprestoGetTableMetadata(
1:PrestoThriftSchemaTableNameschemaTableName)
throws(1:PrestoThriftServiceExceptionex1);
 
/**
   * Returns a batch of splits.
   *
   * @param schemaTableName schema and table name
   * @param desiredColumns a superset of columns to return; empty set means "no columns", {@literal null} set means "all columns"
   * @param outputConstraint constraint on the returned data
   * @param maxSplitCount maximum number of splits to return
   * @param nextToken token from a previous split batch or {@literal null} if it is the first call
   * @return a batch of splits
   */
PrestoThriftSplitBatchprestoGetSplits(
1:PrestoThriftSchemaTableNameschemaTableName,
2:PrestoThriftNullableColumnSetdesiredColumns,
3:PrestoThriftTupleDomainoutputConstraint,
4:i32maxSplitCount,
5:PrestoThriftNullableTokennextToken)
throws(1:PrestoThriftServiceExceptionex1);
 
/**
   * Returns a batch of index splits for the given batch of keys.
   * This method is called if index join strategy is chosen for a query.
   *
   * @param schemaTableName schema and table name
   * @param indexColumnNames specifies columns and their order for keys
   * @param outputColumnNames a list of column names to return
   * @param keys keys for which records need to be returned
   * @param outputConstraint constraint on the returned data
   * @param maxSplitCount maximum number of splits to return
   * @param nextToken token from a previous split batch or {@literal null} if it is the first call
   * @return a batch of splits
   */
PrestoThriftSplitBatchprestoGetIndexSplits(
1:PrestoThriftSchemaTableNameschemaTableName,
2:list<string>indexColumnNames,
3:list<string>outputColumnNames,
4:PrestoThriftPageResultkeys,
5:PrestoThriftTupleDomainoutputConstraint,
6:i32maxSplitCount,
7:PrestoThriftNullableTokennextToken)
throws(1:PrestoThriftServiceExceptionex1);
 
/**
   * Returns a batch of rows for the given split.
   *
   * @param splitId split id as returned in split batch
   * @param columns a list of column names to return
   * @param maxBytes maximum size of returned data in bytes
   * @param nextToken token from a previous batch or {@literal null} if it is the first call
   * @return a batch of table data
   */
PrestoThriftPageResultprestoGetRows(
1:PrestoThriftIdsplitId,
2:list<string>columns,
3:i64maxBytes,
4:PrestoThriftNullableTokennextToken)
throws(1:PrestoThriftServiceExceptionex1);
}

原文: https://prestodb.io/docs/current/connector/thrift.html