Paimon

SinceVersion dev

Instructions for use

  1. When data in hdfs,need to put core-site.xml, hdfs-site.xml and hive-site.xml in the conf directory of FE and BE. First read the hadoop configuration file in the conf directory, and then read the related to the environment variable HADOOP_CONF_DIR configuration file.
  2. The currently adapted version of the payment is 0.6.0

Create Catalog

Paimon Catalog Currently supports two types of Metastore creation catalogs:

  • filesystem(default),Store both metadata and data in the file system.
  • hive metastore,It also stores metadata in Hive metastore. Users can access these tables directly from Hive.

Creating a Catalog Based on FileSystem

For versions 2.0.1 and earlier, please use the following Create Catalog based on Hive Metastore.

HDFS

  1. CREATE CATALOG `paimon_hdfs` PROPERTIES (
  2.     "type" = "paimon",
  3.     "warehouse" = "hdfs://HDFS8000871/user/paimon",
  4.     "dfs.nameservices" = "HDFS8000871",
  5.     "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
  6.     "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
  7.     "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
  8.     "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  9.     "hadoop.username" = "hadoop"
  10. );
  12. CREATE CATALOG `paimon_kerberos` PROPERTIES (
  13.     'type'='paimon',
  14.     "warehouse" = "hdfs://HDFS8000871/user/paimon",
  15.     "dfs.nameservices" = "HDFS8000871",
  16.     "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
  17.     "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
  18.     "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
  19.     "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  20.     'hadoop.security.authentication' = 'kerberos',
  21.     'hadoop.kerberos.keytab' = '/doris/hdfs.keytab',   
  22.     'hadoop.kerberos.principal' = 'hdfs@HADOOP.COM'
  23. );

MINIO

  1. CREATE CATALOG `paimon_s3` PROPERTIES (
  2.     "type" = "paimon",
  3.     "warehouse" = "s3://bucket_name/paimons3",
  4.     "s3.endpoint" = "http://<ip>:<port>",
  5.     "s3.access_key" = "ak",
  6.     "s3.secret_key" = "sk"
  7. );

OBS

  1. CREATE CATALOG `paimon_obs` PROPERTIES (
  2.     "type" = "paimon",
  3.     "warehouse" = "obs://bucket_name/paimon",
  4.     "obs.endpoint"="obs.cn-north-4.myhuaweicloud.com",
  5.     "obs.access_key"="ak",
  6.     "obs.secret_key"="sk"
  7. );

COS

  1. CREATE CATALOG `paimon_cos` PROPERTIES (
  2.     "type" = "paimon",
  3.     "warehouse" = "cosn://paimon-1308700295/paimoncos",
  4.     "cos.endpoint" = "cos.ap-beijing.myqcloud.com",
  5.     "cos.access_key" = "ak",
  6.     "cos.secret_key" = "sk"
  7. );

OSS

  1. CREATE CATALOG `paimon_oss` PROPERTIES (
  2.     "type" = "paimon",
  3.     "warehouse" = "oss://paimon-zd/paimonoss",
  4.     "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
  5.     "oss.access_key" = "ak",
  6.     "oss.secret_key" = "sk"
  7. );

Creating a Catalog Based on Hive Metastore

  1. CREATE CATALOG `paimon_hms` PROPERTIES (
  2.     "type" = "paimon",
  3.     "paimon.catalog.type" = "hms",
  4.     "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
  5.     "hive.metastore.uris" = "thrift://172.21.0.44:7004",
  6.     "dfs.nameservices" = "HDFS8000871",
  7.     "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
  8.     "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
  9.     "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
  10.     "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  11.     "hadoop.username" = "hadoop"
  12. );
  14. CREATE CATALOG `paimon_kerberos` PROPERTIES (
  15.     "type" = "paimon",
  16.     "paimon.catalog.type" = "hms",
  17.     "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
  18.     "hive.metastore.uris" = "thrift://172.21.0.44:7004",
  19.     "hive.metastore.sasl.enabled" = "true",
  20.     "hive.metastore.kerberos.principal" = "hive/xxx@HADOOP.COM",
  21.     "dfs.nameservices" = "HDFS8000871",
  22.     "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
  23.     "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
  24.     "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
  25.     "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  26.     "hadoop.security.authentication" = "kerberos",
  27.     "hadoop.kerberos.principal" = "hdfs@HADOOP.COM",
  28.     "hadoop.kerberos.keytab" = "/doris/hdfs.keytab"
  29. );

Column Type Mapping

Paimon Data TypeDoris Data TypeComment
BooleanTypeBoolean
TinyIntTypeTinyInt
SmallIntTypeSmallInt
IntTypeInt
FloatTypeFloat
BigIntTypeBigInt
DoubleTypeDouble
VarCharTypeVarChar
CharTypeChar
DecimalType(precision, scale)Decimal(precision, scale)
TimestampType,LocalZonedTimestampTypeDateTime
DateTypeDate
MapTypeMapSupport Map nesting
ArrayTypeArraySupport Array nesting
VarBinaryType, BinaryTypeBinary

FAQ

  1. Kerberos

    • Make sure principal and keytab are correct.
    • You need to start a scheduled task (such as crontab) on the BE node, and execute the kinit -kt your_principal your_keytab command every certain time (such as 12 hours).

  2. Unknown type value: UNSUPPORTED

    This is a compatible issue exist in 2.0.2 with Paimon 0.5, you need to upgrade to 2.0.3 or higher to solve this problem. Or patch yourself.

  3. When accessing object storage (OSS, S3, etc.), encounter “file system does not support”.

    In versions before 2.0.5 (inclusive), users need to manually download the following jar package and place it in the ${DORIS_HOME}/be/lib/java_extensions/preload-extensions directory, and restart BE.