Paimon

SinceVersion dev

Instructions for use

  1. When data in hdfs,need to put core-site.xml, hdfs-site.xml and hive-site.xml in the conf directory of FE and BE. First read the hadoop configuration file in the conf directory, and then read the related to the environment variable HADOOP_CONF_DIR configuration file.
  2. The currently adapted version of the payment is 0.6.0

Create Catalog

Paimon Catalog Currently supports two types of Metastore creation catalogs:

  • filesystem(default),Store both metadata and data in the file system.
  • hive metastore,It also stores metadata in Hive metastore. Users can access these tables directly from Hive.

Creating a Catalog Based on FileSystem

For versions 2.0.1 and earlier, please use the following Create Catalog based on Hive Metastore.

HDFS

1. CREATE CATALOG `paimon_hdfs` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "hdfs://HDFS8000871/user/paimon",
4. "dfs.nameservices" = "HDFS8000871",
5. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
6. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
7. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
8. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
9. "hadoop.username" = "hadoop"
10. );

12. CREATE CATALOG `paimon_kerberos` PROPERTIES (
13. 'type'='paimon',
14. "warehouse" = "hdfs://HDFS8000871/user/paimon",
15. "dfs.nameservices" = "HDFS8000871",
16. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
17. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
18. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
19. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
20. 'hadoop.security.authentication' = 'kerberos',
21. 'hadoop.kerberos.keytab' = '/doris/hdfs.keytab',
22. 'hadoop.kerberos.principal' = 'hdfs@HADOOP.COM'
23. );

MINIO

1. CREATE CATALOG `paimon_s3` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "s3://bucket_name/paimons3",
4. "s3.endpoint" = "http://<ip>:<port>",
5. "s3.access_key" = "ak",
6. "s3.secret_key" = "sk"
7. );

OBS

1. CREATE CATALOG `paimon_obs` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "obs://bucket_name/paimon",
4. "obs.endpoint"="obs.cn-north-4.myhuaweicloud.com",
5. "obs.access_key"="ak",
6. "obs.secret_key"="sk"
7. );

COS

1. CREATE CATALOG `paimon_cos` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "cosn://paimon-1308700295/paimoncos",
4. "cos.endpoint" = "cos.ap-beijing.myqcloud.com",
5. "cos.access_key" = "ak",
6. "cos.secret_key" = "sk"
7. );

OSS

1. CREATE CATALOG `paimon_oss` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "oss://paimon-zd/paimonoss",
4. "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
5. "oss.access_key" = "ak",
6. "oss.secret_key" = "sk"
7. );

Creating a Catalog Based on Hive Metastore

1. CREATE CATALOG `paimon_hms` PROPERTIES (
2. "type" = "paimon",
3. "paimon.catalog.type" = "hms",
4. "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
5. "hive.metastore.uris" = "thrift://172.21.0.44:7004",
6. "dfs.nameservices" = "HDFS8000871",
7. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
8. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
9. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
10. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
11. "hadoop.username" = "hadoop"
12. );

14. CREATE CATALOG `paimon_kerberos` PROPERTIES (
15. "type" = "paimon",
16. "paimon.catalog.type" = "hms",
17. "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
18. "hive.metastore.uris" = "thrift://172.21.0.44:7004",
19. "hive.metastore.sasl.enabled" = "true",
20. "hive.metastore.kerberos.principal" = "hive/xxx@HADOOP.COM",
21. "dfs.nameservices" = "HDFS8000871",
22. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
23. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
24. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
25. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
26. "hadoop.security.authentication" = "kerberos",
27. "hadoop.kerberos.principal" = "hdfs@HADOOP.COM",
28. "hadoop.kerberos.keytab" = "/doris/hdfs.keytab"
29. );

Column Type Mapping

Paimon Data Type Doris Data Type Comment
BooleanType Boolean
TinyIntType TinyInt
SmallIntType SmallInt
IntType Int
FloatType Float
BigIntType BigInt
DoubleType Double
VarCharType VarChar
CharType Char
DecimalType(precision, scale) Decimal(precision, scale)
TimestampType,LocalZonedTimestampType DateTime
DateType Date
MapType Map Support Map nesting
ArrayType Array Support Array nesting
VarBinaryType, BinaryType Binary

FAQ

  1. Kerberos

  2. Make sure principal and keytab are correct.

  3. You need to start a scheduled task (such as crontab) on the BE node, and execute the kinit -kt your_principal your_keytab command every certain time (such as 12 hours).
  4. Unknown type value: UNSUPPORTED

This is a compatible issue exist in 2.0.2 with Paimon 0.5, you need to upgrade to 2.0.3 or higher to solve this problem. Or patch yourself. 3. When accessing object storage (OSS, S3, etc.), encounter “file system does not support”.

In versions before 2.0.5 (inclusive), users need to manually download the following jar package and place it in the ${DORIS_HOME}/be/lib/java_extensions/preload-extensions directory, and restart BE.