Paimon

SinceVersion dev

使用须知

  1. 数据放在hdfs时，需要将 core-site.xml，hdfs-site.xml 和 hive-site.xml 放到 FE 和 BE 的 conf 目录下。优先读取 conf 目录下的 hadoop 配置文件，再读取环境变量 HADOOP_CONF_DIR 的相关配置文件。
  2. 当前适配的paimon版本为0.6.0

创建 Catalog

Paimon Catalog 当前支持两种类型的Metastore创建Catalog:

  • filesystem（默认），同时存储元数据和数据在filesystem。
  • hive metastore，它还将元数据存储在Hive metastore中。用户可以直接从Hive访问这些表。

基于FileSystem创建Catalog

2.0.1 及之前版本，请使用后面的 基于Hive Metastore创建Catalog。

HDFS

1. CREATE CATALOG `paimon_hdfs` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "hdfs://HDFS8000871/user/paimon",
4. "dfs.nameservices" = "HDFS8000871",
5. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
6. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
7. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
8. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
9. "hadoop.username" = "hadoop"
10. );

12. CREATE CATALOG `paimon_kerberos` PROPERTIES (
13. 'type'='paimon',
14. "warehouse" = "hdfs://HDFS8000871/user/paimon",
15. "dfs.nameservices" = "HDFS8000871",
16. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
17. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
18. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
19. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
20. 'hadoop.security.authentication' = 'kerberos',
21. 'hadoop.kerberos.keytab' = '/doris/hdfs.keytab',
22. 'hadoop.kerberos.principal' = 'hdfs@HADOOP.COM'
23. );

MINIO

1. CREATE CATALOG `paimon_s3` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "s3://bucket_name/paimons3",
4. "s3.endpoint" = "http://<ip>:<port>",
5. "s3.access_key" = "ak",
6. "s3.secret_key" = "sk"
7. );

OBS

1. CREATE CATALOG `paimon_obs` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "obs://bucket_name/paimon",
4. "obs.endpoint"="obs.cn-north-4.myhuaweicloud.com",
5. "obs.access_key"="ak",
6. "obs.secret_key"="sk"
7. );

COS

1. CREATE CATALOG `paimon_s3` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "cosn://paimon-1308700295/paimoncos",
4. "cos.endpoint" = "cos.ap-beijing.myqcloud.com",
5. "cos.access_key" = "ak",
6. "cos.secret_key" = "sk"
7. );

OSS

1. CREATE CATALOG `paimon_oss` PROPERTIES (
2. "type" = "paimon",
3. "warehouse" = "oss://paimon-zd/paimonoss",
4. "oss.endpoint" = "oss-cn-beijing.aliyuncs.com",
5. "oss.access_key" = "ak",
6. "oss.secret_key" = "sk"
7. );

基于Hive Metastore创建Catalog

1. CREATE CATALOG `paimon_hms` PROPERTIES (
2. "type" = "paimon",
3. "paimon.catalog.type" = "hms",
4. "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
5. "hive.metastore.uris" = "thrift://172.21.0.44:7004",
6. "dfs.nameservices" = "HDFS8000871",
7. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
8. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
9. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
10. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
11. "hadoop.username" = "hadoop"
12. );

14. CREATE CATALOG `paimon_kerberos` PROPERTIES (
15. "type" = "paimon",
16. "paimon.catalog.type" = "hms",
17. "warehouse" = "hdfs://HDFS8000871/user/zhangdong/paimon2",
18. "hive.metastore.uris" = "thrift://172.21.0.44:7004",
19. "hive.metastore.sasl.enabled" = "true",
20. "hive.metastore.kerberos.principal" = "hive/xxx@HADOOP.COM",
21. "dfs.nameservices" = "HDFS8000871",
22. "dfs.ha.namenodes.HDFS8000871" = "nn1,nn2",
23. "dfs.namenode.rpc-address.HDFS8000871.nn1" = "172.21.0.1:4007",
24. "dfs.namenode.rpc-address.HDFS8000871.nn2" = "172.21.0.2:4007",
25. "dfs.client.failover.proxy.provider.HDFS8000871" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
26. "hadoop.security.authentication" = "kerberos",
27. "hadoop.kerberos.principal" = "hdfs@HADOOP.COM",
28. "hadoop.kerberos.keytab" = "/doris/hdfs.keytab"
29. );

列类型映射

Paimon Data Type Doris Data Type Comment
BooleanType Boolean
TinyIntType TinyInt
SmallIntType SmallInt
IntType Int
FloatType Float
BigIntType BigInt
DoubleType Double
VarCharType VarChar
CharType Char
DecimalType(precision, scale) Decimal(precision, scale)
TimestampType,LocalZonedTimestampType DateTime
DateType Date
MapType Map 支持Map嵌套
ArrayType Array 支持Array嵌套
VarBinaryType, BinaryType Binary

常见问题

  1. Kerberos 问题

  2. 确保 principal 和 keytab 配置正确。

  3. 需在 BE 节点启动定时任务（如 crontab），每隔一定时间（如 12小时），执行一次 kinit -kt your_principal your_keytab 命令。
  4. Unknown type value: UNSUPPORTED

这是 Doris 2.0.2 版本和 Paimon 0.5 版本的一个兼容性问题，需要升级到 2.0.3 或更高版本解决，或自行 patch 3. 访问对象存储（OSS、S3 等）报错文件系统不支持

在 2.0.5（含）之前的版本，用户需手动下载以下 jar 包并放置在 ${DORIS_HOME}/be/lib/java_extensions/preload-extensions 目录下，重启 BE。