Iceberg Connector

Overview

The Iceberg connector allows querying data stored in Iceberg tables.

Metastores

Iceberg tables store most of the metadata in the metadata files, along with the data on the filesystem, but it still requires a central place to find the current location of the current metadata pointer for a table. This central place is called the Iceberg Catalog. The Presto Iceberg connector supports different types of Iceberg Catalogs : HIVE, NESSIE, REST, and HADOOP.

To configure the Iceberg connector, create a catalog properties file etc/catalog/iceberg.properties. To define the catalog type, iceberg.catalog.type property is required along with the following contents, with the property values replaced as follows:

Hive Metastore catalog

The Iceberg connector supports the same configuration for HMS as a Hive connector.

1. connector.name=iceberg
2. hive.metastore.uri=hostname:port
3. iceberg.catalog.type=hive

File-Based Metastore

For testing or development purposes, this connector can be configured to use a local filesystem directory as a Hive Metastore. See File-Based Metastore.

Glue catalog

The Iceberg connector supports the same configuration for Glue as a Hive connector.

1. connector.name=iceberg
2. hive.metastore=glue
3. iceberg.catalog.type=hive

There are additional configurations available when using the Iceberg connector configured with Hive or Glue catalogs.

Property Name Description Default
hive.metastore.uri The URI(s) of the Hive metastore to connect to using the Thrift protocol. If multiple URIs are provided, the first URI is used by default, and the rest of the URIs are fallback metastores. Example: thrift://192.0.2.3:9083 or thrift://192.0.2.3:9083,thrift://192.0.2.4:9083. This property is required if the iceberg.catalog.type is hive and hive.metastore is thrift.
hive.metastore.catalog.name Specifies the catalog name to be passed to the metastore.
iceberg.hive-statistics-merge-strategy Comma separated list of statistics to use from the Hive Metastore to override Iceberg table statistics. The available values are NUMBER_OF_DISTINCT_VALUES and TOTAL_SIZE_IN_BYTES. Note: Only valid when the Iceberg connector is configured with Hive.
iceberg.hive.table-refresh.backoff-min-sleep-time The minimum amount of time to sleep between retries when refreshing table metadata. 100ms
iceberg.hive.table-refresh.backoff-max-sleep-time The maximum amount of time to sleep between retries when refreshing table metadata. 5s
iceberg.hive.table-refresh.max-retry-time The maximum amount of time to take across all retries before failing a table metadata refresh operation. 1min
iceberg.hive.table-refresh.retries The number of times to retry after errors when refreshing table metadata using the Hive metastore. 20
iceberg.hive.table-refresh.backoff-scale-factor The multiple used to scale subsequent wait time between retries. 4.0
iceberg.engine.hive.lock-enabled Whether to use locks to ensure atomicity of commits. This will turn off locks but is overridden at a table level with the table configuration engine.hive.lock-enabled. true

Nessie catalog

To use a Nessie catalog, configure the catalog type as iceberg.catalog.type=nessie.

1. connector.name=iceberg
2. iceberg.catalog.type=nessie
3. iceberg.catalog.warehouse=/tmp
4. iceberg.nessie.uri=https://localhost:19120/api/v1

Additional supported properties for the Nessie catalog:

Property Name Description
iceberg.nessie.ref The branch/tag to use for Nessie, defaults to main.
iceberg.nessie.uri Nessie API endpoint URI (required). Example: https://localhost:19120/api/v1
iceberg.nessie.auth.type The authentication type to use. Available values are BASIC or BEARER. Example: BEARER Note: Nessie BASIC authentication type is deprecated, this will be removed in upcoming release
iceberg.nessie.auth.basic.username The username to use with BASIC authentication. Example: test_user
iceberg.nessie.auth.basic.password The password to use with BASIC authentication. Example: my$ecretPass
iceberg.nessie.auth.bearer.token The token to use with BEARER authentication. Example: SXVLUXUhIExFQ0tFUiEK
iceberg.nessie.read-timeout-ms The read timeout in milliseconds for requests to the Nessie server. Example: 5000
iceberg.nessie.connect-timeout-ms The connection timeout in milliseconds for the connection requests to the Nessie server. Example: 10000
iceberg.nessie.compression-enabled Configuration of whether compression should be enabled or not for requests to the Nessie server, defaults to true.
iceberg.nessie.client-builder-impl Configuration of the custom ClientBuilder implementation class to be used.

Setting Up Nessie With Docker

To set up a Nessie instance locally using the Docker image, see Setting up Nessie. Once the Docker instance is up and running, you should see logs similar to the following example:

1. 2023-09-05 13:11:37,905 INFO  [io.quarkus] (main) nessie-quarkus 0.69.0 on JVM (powered by Quarkus 3.2.4.Final) started in 1.921s. Listening on: http://0.0.0.0:19120
2. 2023-09-05 13:11:37,906 INFO  [io.quarkus] (main) Profile prod activated.
3. 2023-09-05 13:11:37,906 INFO  [io.quarkus] (main) Installed features: [agroal, amazon-dynamodb, cassandra-client, cdi, google-cloud-bigtable, hibernate-validator, jdbc-postgresql, logging-sentry, micrometer, mongodb-client, narayana-jta, oidc, opentelemetry, reactive-routes, resteasy, resteasy-jackson, security, security-properties-file, smallrye-context-propagation, smallrye-health, smallrye-openapi, swagger-ui, vertx]

If log messages related to Nessie’s OpenTelemetry collector appear similar to the following example, you can disable OpenTelemetry using the configuration option quarkus.otel.sdk.disabled=true.

1. 2023-08-27 11:10:02,492 INFO  [io.qua.htt.access-log] (executor-thread-1) 172.17.0.1 - - [27/Aug/2023:11:10:02 +0000] "GET /api/v1/config HTTP/1.1" 200 62
2. 2023-08-27 11:10:05,007 SEVERE [io.ope.exp.int.grp.OkHttpGrpcExporter] (OkHttp http://localhost:4317/...) Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317

For example, start the Docker image using the following command: docker run -p 19120:19120 -e QUARKUS_OTEL_SDK_DISABLED=true ghcr.io/projectnessie/nessie

For more information about this configuration option and other related options, see the OpenTelemetry Configuration Reference.

For more information about troubleshooting OpenTelemetry traces, see Troubleshooting traces.

If an error similar to the following example is displayed, this is probably because you are interacting with an http server, and not an https server. You need to set iceberg.nessie.uri to http://localhost:19120/api/v1.

1. Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
2. at sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:448)
3. at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:174)
4. at sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
5. at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1320)
6. at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1233)
7. at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:417)
8. at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:389)
9. at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:558)
10. at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
11. at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167)
12. at org.projectnessie.client.http.impl.jdk8.UrlConnectionRequest.executeRequest(UrlConnectionRequest.java:71)
13. ... 42 more

REST catalog

To use a REST catalog, configure the catalog type as iceberg.catalog.type=rest. A minimal configuration includes:

1. connector.name=iceberg
2. iceberg.catalog.type=rest
3. iceberg.rest.uri=https://localhost:8181

Additional supported properties for the REST catalog:

Property Name Description
iceberg.rest.uri REST API endpoint URI (required). Example: https://localhost:8181
iceberg.rest.auth.type The authentication type to use. Available values are NONE or OAUTH2 (default: NONE). OAUTH2 requires either a credential or token.
iceberg.rest.auth.oauth2.uri OAUTH2 server endpoint URI. Example: https://localhost:9191
iceberg.rest.auth.oauth2.credential The credential to use for OAUTH2 authentication. Example: key:secret
iceberg.rest.auth.oauth2.token The Bearer token to use for OAUTH2 authentication. Example: SXVLUXUhIExFQ0tFUiEK
iceberg.rest.auth.oauth2.scope The scope to use for OAUTH2 authentication. This property is only applicable when using iceberg.rest.auth.oauth2.credential. Example: PRINCIPAL_ROLE:ALL
iceberg.rest.nested.namespace.enabled In REST Catalogs, tables are grouped into namespaces, that can be nested. But if a large number of recursive namespaces result in lower performance, querying nested namespaces can be disabled. Defaults to true.
iceberg.rest.session.type The session type to use when communicating with the REST catalog. Available values are NONE or USER (default: NONE).
iceberg.catalog.warehouse A catalog warehouse root path for Iceberg tables (optional). Example: s3://warehouse/

Hadoop catalog

To use a Hadoop catalog, configure the catalog type as iceberg.catalog.type=hadoop. A minimal configuration includes:

1. connector.name=iceberg
2. iceberg.catalog.type=hadoop
3. iceberg.catalog.warehouse=hdfs://hostname:port

Hadoop catalog configuration properties:

Property Name Description Default
iceberg.catalog.warehouse The catalog warehouse root path for Iceberg tables. The Hadoop catalog requires a file system that supports an atomic rename operation, such as HDFS, to maintain metadata files in order to implement an atomic transaction commit. Example: hdfs://nn:8020/warehouse/path Do not set iceberg.catalog.warehouse to a path in object stores or local file systems in the production environment. This property is required if the iceberg.catalog.type is hadoop. Otherwise, it will be ignored.
iceberg.catalog.hadoop.warehouse.datadir The catalog warehouse root data path for Iceberg tables. It is only supported with the Hadoop catalog. Example: s3://iceberg_bucket/warehouse. This optional property can be set to a path in object stores or HDFS. If set, all tables in this Hadoop catalog default to saving their data and delete files in the specified root data directory.
iceberg.catalog.cached-catalog-num The number of Iceberg catalogs to cache. This property is required if the iceberg.catalog.type is hadoop. Otherwise, it will be ignored. 10

Configure the Amazon S3 properties to specify a S3 location as the warehouse data directory for the Hadoop catalog. This way, the data and delete files of Iceberg tables are stored in S3. An example configuration includes:

1. connector.name=iceberg
2. iceberg.catalog.type=hadoop
3. iceberg.catalog.warehouse=hdfs://nn:8020/warehouse/path
4. iceberg.catalog.hadoop.warehouse.datadir=s3://iceberg_bucket/warehouse

6. hive.s3.use-instance-credentials=false
7. hive.s3.aws-access-key=accesskey
8. hive.s3.aws-secret-key=secretkey
9. hive.s3.endpoint=http://192.168.0.103:9878
10. hive.s3.path-style-access=true

Presto C++ Support

HIVE, NESSIE, REST, and HADOOP Iceberg catalogs are supported in Presto C++.

Configuration Properties

Note

The Iceberg connector supports configuration options for Amazon S3 as a Hive connector.

The following configuration properties are available for all catalog types:

Property Name Description Default Presto Java Support Presto C++ Support
iceberg.catalog.type The catalog type for Iceberg tables. The available values are HIVE, HADOOP, and NESSIE and REST. HIVE Yes Yes, only needed on coordinator
iceberg.hadoop.config.resources The path(s) for Hadoop configuration resources. Example: /etc/hadoop/conf/core-site.xml. This property is required if the iceberg.catalog.type is hadoop. Otherwise, it will be ignored. Yes Yes, only needed on coordinator
iceberg.file-format The storage file format for Iceberg tables. The available values are PARQUET and ORC. PARQUET Yes No, write is not supported yet
iceberg.compression-codec The compression codec to use when writing files. The available values are NONE, SNAPPY, GZIP, LZ4, and ZSTD. GZIP Yes No, write is not supported yet
iceberg.max-partitions-per-writer The maximum number of partitions handled per writer. 100 Yes No, write is not supported yet
iceberg.minimum-assigned-split-weight A decimal value in the range (0, 1] is used as a minimum for weights assigned to each split. A low value may improve performance on tables with small files. A higher value may improve performance for queries with highly skewed aggregations or joins. 0.05 Yes Yes
iceberg.enable-merge-on-read-mode Enable reading base tables that use merge-on-read for updates. true Yes Yes, only needed on coordinator
iceberg.delete-as-join-rewrite-enabled When enabled, equality delete row filtering is applied as a join with the data of the equality delete files. Deprecated: This property is deprecated and will be removed in a future release. Use the iceberg.delete-as-join-rewrite-max-delete-columns configuration property instead. true Yes No, Equality delete read is not supported
iceberg.delete-as-join-rewrite-max-delete-columns When set to a number greater than 0, this property enables equality delete row filtering as a join with the data of the equality delete files. The value of this property is the maximum number of columns that can be used in the equality delete files. If the number of columns in the equality delete files exceeds this value, then the optimization is not applied and the equality delete files are applied directly to each row in the data files. This property is only applicable when iceberg.delete-as-join-rewrite-enabled is set to true. 400 Yes No, Equality delete read is not supported
iceberg.enable-parquet-dereference-pushdown Enable parquet dereference pushdown. true Yes No
iceberg.statistic-snapshot-record-difference-weight The amount that the difference in total record count matters when calculating the closest snapshot when picking statistics. A value of 1 means a single record is equivalent to 1 millisecond of time difference. Yes Yes, only needed on coordinator
iceberg.pushdown-filter-enabled Experimental: Enable filter pushdown for Iceberg. This is only supported with Presto C++. false No Yes
iceberg.rows-for-metadata-optimization-threshold The maximum number of partitions in an Iceberg table to allow optimizing queries of that table using metadata. If an Iceberg table has more partitions than this threshold, metadata optimization is skipped. Set to 0 to disable metadata optimization. 1000 Yes Yes
iceberg.split-manager-threads Number of threads to use for generating Iceberg splits. Number of available processors Yes Yes, only needed on coordinator
iceberg.metadata-previous-versions-max The maximum number of old metadata files to keep in current metadata log. 100 Yes No, write is not supported yet
iceberg.metadata-delete-after-commit Set to true to delete the oldest metadata files after each commit. false Yes No, write is not supported yet
iceberg.metrics-max-inferred-column The maximum number of columns for which metrics are collected. 100 Yes No, write is not supported yet
iceberg.max-statistics-file-cache-size Maximum size in bytes that should be consumed by the statistics file cache. 256MB Yes Yes, only needed on coordinator

Table Properties

Table properties set metadata for the underlying tables. This is key for CREATE TABLE/CREATE TABLE AS statements. Table properties are passed to the connector using a WITH clause:

1. CREATE TABLE tablename
2. WITH (
3. property_name = property_value,
4. ...
5. )

The following table properties are available, which are specific to the Presto Iceberg connector:

Property Name Description Default Presto Java Support Presto C++ Support
commit.retry.num-retries Determines the number of attempts for committing the metadata in case of concurrent upsert requests, before failing. 4 Yes No, write is not supported yet
format-version Optionally specifies the format version of the Iceberg specification to use for new tables, either 1 or 2. 2 Yes No, write is not supported yet
location Optionally specifies the file system location URI for the table. Yes Yes
partitioning Optionally specifies table partitioning. If a table is partitioned by columns c1 and c2, the partitioning property is partitioning = ARRAY[‘c1’, ‘c2’]. Yes Yes
read.split.target-size The target size for an individual split when generating splits for a table scan. Generated splits may still be larger or smaller than this value. Must be specified in bytes. 134217728 (128MB) Yes Yes
write.data.path Optionally specifies the file system location URI for storing the data and delete files of the table. This only applies to files written after this property is set. Files previously written aren’t relocated to reflect this parameter. Yes No, write is not supported yet
write.delete.mode Optionally specifies the write delete mode of the Iceberg specification to use for new tables, either copy-on-write or merge-on-read. merge-on-read Yes No, write is not supported yet
write.format.default Optionally specifies the format of table data files, either PARQUET or ORC. PARQUET Yes No, write is not supported yet
write.metadata.previous-versions-max Optionally specifies the max number of old metadata files to keep in current metadata log. 100 Yes No, write is not supported yet
write.metadata.delete-after-commit.enabled Set to true to delete the oldest metadata file after each commit. false Yes No, write is not supported yet
write.metadata.metrics.max-inferred-column-defaults Optionally specifies the maximum number of columns for which metrics are collected. 100 Yes No, write is not supported yet
write.update.mode Optionally specifies the write delete mode of the Iceberg specification to use for new tables, either copy-on-write or merge-on-read. merge-on-read Yes No, write is not supported yet

The table definition below specifies format ORC, partitioning by columns c1 and c2, and a file system location of s3://test_bucket/test_schema/test_table:

1. CREATE TABLE test_table (
2. c1 bigint,
3. c2 varchar,
4. c3 double
5. )
6. WITH (
7. format = 'ORC',
8. partitioning = ARRAY['c1', 'c2'],
9. location = 's3://test_bucket/test_schema/test_table')
10. )

Deprecated Table Properties

Some table properties have been deprecated or removed. The following table lists the deprecated properties and their replacements. Update queries to use the new property names as soon as possible. They will be removed in a future version.

Deprecated Property Name New Property Name
format write.format.default
format_version format-version
commit_retries commit.retry.num-retries
delete_mode write.delete.mode
metadata_previous_versions_max write.metadata.previous-versions-max
metadata_delete_after_commit write.metadata.delete-after-commit.enabled
metrics_max_inferred_column write.metadata.metrics.max-inferred-column-defaults

Session Properties

Session properties set behavior changes for queries executed within the given session.

Property Name Description Presto Java Support Presto C++ Support
iceberg.delete_as_join_rewrite_enabled Overrides the behavior of the connector property iceberg.delete-as-join-rewrite-enabled in the current session. Deprecated: This property is deprecated and will be removed. Use iceberg.delete_as_join_rewrite_max_delete_columns instead. Yes No, Equality delete read is not supported
iceberg.delete_as_join_rewrite_max_delete_columns Overrides the behavior of the connector property iceberg.delete-as-join-rewrite-max-delete-columns in the current session. Yes No, Equality delete read is not supported
iceberg.hive_statistics_merge_strategy Overrides the behavior of the connector property iceberg.hive-statistics-merge-strategy in the current session. Yes Yes
iceberg.rows_for_metadata_optimization_threshold Overrides the behavior of the connector property iceberg.rows-for-metadata-optimization-threshold in the current session. Yes Yes
iceberg.target_split_size_bytes Overrides the target split size for all tables in a query in bytes. Set to 0 to use the value in each Iceberg table’s read.split.target-size property. Yes Yes
iceberg.affinity_scheduling_file_section_size When the node_selection_strategy or hive.node-selection-strategy property is set to SOFT_AFFINITY, this configuration property will change the size of a file chunk that is hashed to a particular node when determining the which worker to assign a split to. Splits which read data from the same file within the same chunk will hash to the same node. A smaller chunk size will result in a higher probability splits being distributed evenly across the cluster, but reduce locality. See Node Selection Strategy. Yes Yes
iceberg.parquet_dereference_pushdown_enabled Overrides the behavior of the connector property iceberg.enable-parquet-dereference-pushdown in the current session. Yes No

Caching Support

Statistics File Caching

Support for Puffin-based statistics caching. It is enabled by default.

JMX query to get the metrics and verify the cache usage:

1. SELECT * FROM jmx.current."com.facebook.presto.iceberg.statistics:name=iceberg,type=statisticsfilecache";

Manifest File Caching

As of Iceberg version 1.1.0, Apache Iceberg provides a mechanism to cache the contents of Iceberg manifest files in memory. This feature helps to reduce repeated reads of small Iceberg manifest files from remote storage.

The following configuration properties are available:

Property Name Description Default
iceberg.io.manifest.cache-enabled Enable or disable the manifest caching feature. true
iceberg.io-impl Custom FileIO implementation to use in a catalog. It must be set to enable manifest caching. This is only needed for Hadoop, Nessie and REST catalogs. org.apache.iceberg.hadoop.HadoopFileIO
iceberg.io.manifest.cache.max-total-bytes Maximum size of cache size in bytes. 104857600
iceberg.io.manifest.cache.expiration-interval-ms Maximum time duration in milliseconds for which an entry stays in the manifest cache. Set to 0 to disable entry expiration. 60000
iceberg.io.manifest.cache.max-content-length Maximum length of a manifest file to be considered for caching in bytes. Manifest files with a length exceeding this size will not be cached. 8388608

JMX query to get the metrics and verify the cache usage:

1. SELECT * FROM jmx.current."com.facebook.presto.iceberg:name=iceberg,type=manifestfilecache";

Note

Manifest file cache statistics are only available through the JMX connector when the Iceberg connector is configured with a HIVE catalog type.

Presto C++ Support

Manifest file caching is supported in Presto C++.

Alluxio Data Cache

A Presto worker caches remote storage data in its original form (compressed and possibly encrypted) on local SSD upon read.

The following configuration properties are required to set in the Iceberg catalog file (catalog/iceberg.properties):

1. cache.enabled=true
2. cache.base-directory=file:///mnt/flash/data
3. cache.type=ALLUXIO
4. cache.alluxio.max-cache-size=1600GB
5. hive.node-selection-strategy=SOFT_AFFINITY

JMX queries to get the metrics and verify the cache usage:

1. SELECT * FROM jmx.current."com.facebook.alluxio:name=client.cachehitrate,type=gauges";

3. SELECT * FROM jmx.current."com.facebook.alluxio:name=client.cachebytesreadcache,type=meters";

5. SHOW TABLES FROM jmx.current like '%alluxio%';

Presto C++ Support

Alluxio data caching is applicable for Presto Java. Async data cache is supported in Presto C++. See Async Data Cache and Prefetching.

Caches open file descriptors and stripe or file footer information in leaf worker memory. These pieces of data are mostly frequently accessed when reading files.

The following configuration properties are required to set in the Iceberg catalog file (catalog/iceberg.properties):

1. # scheduling
2. hive.node-selection-strategy=SOFT_AFFINITY

4. # orc
5. iceberg.orc.file-tail-cache-enabled=true
6. iceberg.orc.file-tail-cache-size=100MB
7. iceberg.orc.file-tail-cache-ttl-since-last-access=6h
8. iceberg.orc.stripe-metadata-cache-enabled=true
9. iceberg.orc.stripe-footer-cache-size=100MB
10. iceberg.orc.stripe-footer-cache-ttl-since-last-access=6h
11. iceberg.orc.stripe-stream-cache-size=300MB
12. iceberg.orc.stripe-stream-cache-ttl-since-last-access=6h

14. # parquet
15. iceberg.parquet.metadata-cache-enabled=true
16. iceberg.parquet.metadata-cache-size=100MB
17. iceberg.parquet.metadata-cache-ttl-since-last-access=6h

JMX queries to get the metrics and verify the cache usage:

1. SELECT * FROM jmx.current."com.facebook.presto.hive:name=iceberg_parquetmetadata,type=cachestatsmbean";

Presto C++ Support

File and stripe footer cache is not applicable for Presto C++.

Metastore Cache

Iceberg Connector does not support Metastore Caching.

Extra Hidden Metadata Columns

The Iceberg connector exposes extra hidden metadata columns. You can query these as part of a SQL query by including them in your SELECT statement.

$path column

The full file system path name of the file for this row.

1. SELECT "$path", regionkey FROM "ctas_nation";
1. $path                    |  regionkey
2. ---------------------------------+-----------
3. /full/path/to/file/file.parquet | 2

$data_sequence_number column

The Iceberg data sequence number in which this row was added.

1. SELECT "$data_sequence_number", regionkey FROM "ctas_nation";
1. $data_sequence_number     |  regionkey
2. ----------------------------------+------------
3. 2                    | 3

$deleted column

Whether this row is a deleted row. When this column is used, deleted rows from delete files will be marked as true instead of being filtered out of the results.

1. DELETE FROM "ctas_nation" WHERE regionkey = 0;

3. SELECT "$deleted", regionkey FROM "ctas_nation";
1. $deleted | regionkey
2. ----------+-----------
3. true     |         0
4. false    |         1

$delete_file_path column

The path of the delete file corresponding to a deleted row, or NULL if the row was not deleted. When this column is used, deleted rows will not be filtered out of the results.

1. DELETE FROM "ctas_nation" WHERE regionkey = 0;

3. SELECT "$delete_file_path", regionkey FROM "ctas_nation";
1. $delete_file_path                                 | regionkey
2. -----------------------------------------------------------------------------------+-----------
3. file:/path/to/table/data/delete_file_d8510b3e-510a-4fc2-b2b2-e59ead7fd386.parquet |         0
4. NULL                                                                              |         1

Presto C++ Support

All above metadata columns are supported in Presto C++.

Extra Hidden Metadata Tables

The Iceberg connector exposes extra hidden metadata tables. You can query these as a part of a SQL query by appending name to the table.

$properties Table

General properties of the given table.

1. SELECT * FROM "ctas_nation$properties";
1. key           |  value
2. ----------------------+---------
3. write.format.default | PARQUET

$history Table

History of table state changes.

1. SELECT * FROM "ctas_nation$history";
1. made_current_at            |     snapshot_id     | parent_id | is_current_ancestor
2. --------------------------------------+---------------------+-----------+---------------------
3. 2022-11-25 20:56:31.784 Asia/Kolkata  | 7606232158543069775 | NULL      | true

$snapshots Table

Details about the table snapshots. For more information see Snapshots in the Iceberg Table Spec.

1. SELECT * FROM "ctas_nation$snapshots";
1. committed_at             |     snapshot_id     | parent_id | operation |                                                  manifest_list                                           |                                                                                 summary
2. --------------------------------------+---------------------+-----------+-----------+----------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3. 2022-11-25 20:56:31.784 Asia/Kolkata  | 7606232158543069775 | NULL      | append    | s3://my-bucket/ctas_nation/metadata/snap-7606232158543069775-1-395a2cad-b244-409b-b030-cc44949e5a4e.avro | {changed-partition-count=1, added-data-files=1, total-equality-deletes=0, added-records=25, total-position-deletes=0, added-files-size=1648, total-delete-files=0, total-files-size=1648, total-records=25, total-data-files=1}

$manifests Table

Details about the manifests of different table snapshots. For more information see Manifests in the Iceberg Table Spec.

1. SELECT * FROM "ctas_nation$manifests";
1. path                                  | length | partition_spec_id |  added_snapshot_id  | added_data_files_count | existing_data_files_count | deleted_data_files_count | partitions
2. ---------------------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+-----------
3. s3://my-bucket/ctas_nation/metadata/395a2cad-b244-409b-b030-cc44949e5a4e-m0.avro |   5957 |                 0 | 7606232158543069775 |                      1 |                         0 |                    0     |    []

$partitions Table

Detailed partition information for the table.

1. SELECT * FROM "ctas_nation$partitions";
1. row_count | file_count | total_size |           nationkey           |                   name                   |          regionkey           |                          comment
2. -----------+------------+------------+-------------------------------+------------------------------------------+------------------------------+------------------------------------------------------------
3. 25     |          1 |       1648 | {min=0, max=24, null_count=0} | {min=ALGERIA, max=VIETNAM, null_count=0} | {min=0, max=4, null_count=0} | {min= haggle. careful, max=y final packaget, null_count=0}

$files Table

Overview of data files in the current snapshot of the table.

1. SELECT * FROM "ctas_nation$files";
1. content |                                      file_path                               | file_format | record_count | file_size_in_bytes |        column_sizes         |       value_counts       |  null_value_counts   | nan_value_counts |          lower_bounds                     |             upper_bounds                   | key_metadata | split_offsets | equality_ids
2. ---------+------------------------------------------------------------------------------+-------------+--------------+--------------------+-----------------------------+--------------------------+----------------------+------------------+-------------------------------------------+--------------------------------------------+--------------+---------------+-------------
3. 0     | s3://my-bucket/ctas_nation/data/9f889274-6f74-4d28-8164-275eef99f660.parquet | PARQUET     |           25 |               1648 | {1=52, 2=222, 3=105, 4=757} | {1=25, 2=25, 3=25, 4=25} | {1=0, 2=0, 3=0, 4=0} |  NULL            | {1=0, 2=ALGERIA, 3=0, 4= haggle. careful} | {1=24, 2=VIETNAM, 3=4, 4=y final packaget} | NULL         | NULL          | NULL

$changelog Table

This table lets you view which row-level changes have occurred to the table in a particular order over time. The $changelog table presents the history of changes to the table and makes the data available to process through a query.

The result of a changelog query always returns a static schema with four columns:

  1. operation: (VARCHAR) indicating whether the row was inserted, updated, or deleted.
  2. ordinal: (int) A number indicating a relative order that a particular change needs to be applied to the table relative to all other changes.
  3. snapshotid: (bigint) Represents the snapshot a row-level change was made in.
  4. rowdata: (row(T)) which includes the data for the particular row. The inner values of this type match the schema of the parent table.

The changelog table can be queried with the following name format:

1. ... FROM "<table>[@<begin snapshot ID>]$changelog[@<end snapshot ID>]"
  • <table> is the name of the table.
  • <begin snapshot ID> is the snapshot of the table you want to begin viewing changes from. This parameter is optional. If absent, the oldest available snapshot is used.
  • <end snapshot ID> is the last snapshot for which you want to view changes. This parameter is optional. If absent, the most current snapshot of the table is used.

One use for the $changelog table would be to find when a record was inserted or removed from the table. To accomplish this, the $changelog table can be used in conjunction with the $snapshots table. First, choose a snapshot ID from the $snapshots table to choose the starting point.

1. SELECT * FROM "orders$snapshots";
1. committed_at                 |     snapshot_id     |      parent_id      | operation |                                                                                       manifest_list                                                                                        |                                                                                                              summary
2. ---------------------------------------------+---------------------+---------------------+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3. 2023-09-26 08:45:20.930 America/Los_Angeles | 2423571386296047175 | NULL                | append    | file:/var/folders/g_/6_hxl7r16qdddw7956j_r88h0000gn/T/PrestoTest8140889264166671718/catalog/tpch/ctas_orders/metadata/snap-2423571386296047175-1-3f288b1c-95a9-406b-9e17-9cfe31a11b48.avro | {changed-partition-count=1, added-data-files=4, total-equality-deletes=0, added-records=100, total-position-deletes=0, added-files-size=9580, total-delete-files=0, total-files-size=9580, total-records=100, total-data-files=4}
4. 2023-09-26 08:45:36.942 America/Los_Angeles | 8702997868627997320 | 2423571386296047175 | append    | file:/var/folders/g_/6_hxl7r16qdddw7956j_r88h0000gn/T/PrestoTest8140889264166671718/catalog/tpch/ctas_orders/metadata/snap-8702997868627997320-1-a2e1c714-7eed-4e2c-b144-dae4147ebaa4.avro | {changed-partition-count=1, added-data-files=1, total-equality-deletes=0, added-records=1, total-position-deletes=0, added-files-size=1687, total-delete-files=0, total-files-size=11267, total-records=101, total-data-files=5}
5. 2023-09-26 08:45:39.866 America/Los_Angeles | 7615903782581283889 | 8702997868627997320 | append    | file:/var/folders/g_/6_hxl7r16qdddw7956j_r88h0000gn/T/PrestoTest8140889264166671718/catalog/tpch/ctas_orders/metadata/snap-7615903782581283889-1-d94c2114-fd22-4de2-9ab5-c0b5bf67282f.avro | {changed-partition-count=1, added-data-files=3, total-equality-deletes=0, added-records=3, total-position-deletes=0, added-files-size=4845, total-delete-files=0, total-files-size=16112, total-records=104, total-data-files=8}
6. 2023-09-26 08:45:48.404 America/Los_Angeles |  677209275408372885 | 7615903782581283889 | append    | file:/var/folders/g_/6_hxl7r16qdddw7956j_r88h0000gn/T/PrestoTest8140889264166671718/catalog/tpch/ctas_orders/metadata/snap-677209275408372885-1-ad69e208-1440-459b-93e8-48e61f961758.avro  | {changed-partition-count=1, added-data-files=3, total-equality-deletes=0, added-records=5, total-position-deletes=0, added-files-size=4669, total-delete-files=0, total-files-size=20781, total-records=109, total-data-files=11}

Now that we know the snapshots available to query in the changelog, we can see what changes were made to the table since it was created. Specifically, this example uses the earliest snapshot ID: 2423571386296047175

1. SELECT * FROM "ctas_orders@2423571386296047175$changelog" ORDER BY ordinal;
1. operation | ordinal |     snapshotid      |                                                                                                                   rowdata
2. -----------+---------+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3. INSERT    |       0 | 8702997868627997320 | {orderkey=37504, custkey=1291, orderstatus=O, totalprice=165509.83, orderdate=1996-03-04, orderpriority=5-LOW, clerk=Clerk#000000871, shippriority=0, comment=c theodolites alongside of the fluffily bold requests haggle quickly against }
4. INSERT    |       1 | 7615903782581283889 | {orderkey=12001, custkey=739, orderstatus=F, totalprice=138635.75, orderdate=1994-07-07, orderpriority=2-HIGH, clerk=Clerk#000000863, shippriority=0, comment=old, even theodolites. regular, special theodolites use furio}
5. INSERT    |       1 | 7615903782581283889 | {orderkey=17989, custkey=364, orderstatus=F, totalprice=133669.05, orderdate=1994-01-17, orderpriority=4-NOT SPECIFIED, clerk=Clerk#000000547, shippriority=0, comment=ously express excuses. even theodolit}
6. INSERT    |       1 | 7615903782581283889 | {orderkey=37504, custkey=1291, orderstatus=O, totalprice=165509.83, orderdate=1996-03-04, orderpriority=5-LOW, clerk=Clerk#000000871, shippriority=0, comment=c theodolites alongside of the fluffily bold requests haggle quickly against }
7. INSERT    |       2 |  677209275408372885 | {orderkey=17991, custkey=92, orderstatus=O, totalprice=20732.51, orderdate=1998-07-09, orderpriority=4-NOT SPECIFIED, clerk=Clerk#000000636, shippriority=0, comment= the quickly express accounts. iron}
8. INSERT    |       2 |  677209275408372885 | {orderkey=17989, custkey=364, orderstatus=F, totalprice=133669.05, orderdate=1994-01-17, orderpriority=4-NOT SPECIFIED, clerk=Clerk#000000547, shippriority=0, comment=ously express excuses. even theodolit}
9. INSERT    |       2 |  677209275408372885 | {orderkey=17990, custkey=458, orderstatus=O, totalprice=218031.58, orderdate=1998-03-18, orderpriority=3-MEDIUM, clerk=Clerk#000000340, shippriority=0, comment=ounts wake final foxe}
10. INSERT    |       2 |  677209275408372885 | {orderkey=18016, custkey=403, orderstatus=O, totalprice=174070.99, orderdate=1996-03-19, orderpriority=1-URGENT, clerk=Clerk#000000629, shippriority=0, comment=ly. quickly ironic excuses are furiously. carefully ironic pack}
11. INSERT    |       2 |  677209275408372885 | {orderkey=18017, custkey=958, orderstatus=F, totalprice=203091.02, orderdate=1993-03-26, orderpriority=1-URGENT, clerk=Clerk#000000830, shippriority=0, comment=sleep quickly bold requests. slyly pending pinto beans haggle in pla}

$refs Table

Details about Iceberg references including branches and tags. For more information see Branching and Tagging.

1. SELECT * FROM "ctas_nation$refs";
1. name     |  type  |     snapshot_id     | max_reference_age_in_ms | min_snapshots_to_keep | max_snapshot_age_in_ms
2. ------------+--------+---------------------+-------------------------+-----------------------+------------------------
3. main       | BRANCH | 3074797416068623476 | NULL                    | NULL                  | NULL
4. testBranch | BRANCH | 3374797416068698476 | NULL                    | NULL                  | NULL
5. testTag    | TAG    | 4686954189838128572 | 10                      | NULL                  | NULL

Presto C++ Support

All above metadata tables, except $changelog, are supported in Presto C++.

Procedures

Use the CALL statement to perform data manipulation or administrative tasks. Procedures are available in the system schema of the catalog.

Register Table

Iceberg tables for which table data and metadata already exist in the file system can be registered with the catalog. Use the register_table procedure on the catalog’s system schema to register a table which already exists but does not known by the catalog.

The following arguments are available:

Argument Name Required Type Description
schema Yes string Schema of the table to register
table_name Yes string Name of the table to register
metadata_location Yes string The location of the table metadata which is to be registered
metadata_file string An optionally specified metadata file which is to be registered

Examples:

  • Register a table through supplying the target schema, desired table name, and the location of the table metadata:

``` 1. CALL iceberg.system.register_table('schema_name', 'table_name', 'hdfs://localhost:9000/path/to/iceberg/table/metadata/dir')

  1. CALL iceberg.system.register_table(table_name => 'table_name', schema => 'schema_name', metadata_location => 'hdfs://localhost:9000/path/to/iceberg/table/metadata/dir') ```

Note

If multiple metadata files of the same version exist at the specified location, the most recently modified one is used. - Register a table through additionally supplying a specific metadata file:

``` 1. CALL iceberg.system.register_table('schema_name', 'table_name', 'hdfs://localhost:9000/path/to/iceberg/table/metadata/dir', '00000-35a08aed-f4b0-4010-95d2-9d73ef4be01c.metadata.json')

  1. CALL iceberg.system.register_table(table_name => 'table_na

(本文超出站点单文档体积上限,已截断)