Cloud

Query Iceberg-enabled Topics

With Iceberg-enabled topics, you can retain streaming data in your lakehouse beyond your Redpanda topic retention window and continue to query it with SQL. A single query returns both the live records still in the topic and the Iceberg-committed history that has aged out under topic retention, in one result.

For the streaming-only query path (live records only, without Iceberg history), see Query Streaming Topics.

After reading this page, you will be able to:

Describe the storage and catalog objects that Redpanda Cloud configures for Iceberg-enabled topic queries
Query both the live records and the Iceberg-committed history of a topic in a single result
Handle schema differences between a Redpanda topic and its Iceberg table

Prerequisites

A Redpanda BYOC cluster on AWS or GCP with Redpanda SQL enabled. See Enable Redpanda SQL.
The cluster’s iceberg_catalog_type property is set to rest. The object_storage catalog type does not support Iceberg queries from Redpanda SQL.
The cluster’s Iceberg REST catalog is configured. See Integrate with REST Catalogs for the supported REST catalog options (AWS Glue, GCP Lakehouse, Snowflake, Databricks Unity, Polaris, and others) and their configuration steps.
An Iceberg-enabled Redpanda topic with a schema (Protobuf, Avro, or JSON) registered in Schema Registry. To enable Iceberg on a topic, see About Iceberg Topics.
Connect to Redpanda SQL with psql or another PostgreSQL client. See Connect to Redpanda SQL.

How the query catalogs are set up

When you enable Redpanda SQL on a BYOC cluster with the Iceberg REST catalog configured, Redpanda Cloud runs the following statements under the hood so you don’t have to. The result is a default_redpanda_catalog already linked to an Iceberg catalog, ready for CREATE TABLE against an Iceberg-enabled topic. The storage and Iceberg-catalog options match the cluster’s REST catalog configuration (endpoint, credentials, region).

A storage connection holds the object-storage credentials your cluster’s REST catalog uses:

CREATE STORAGE iceberg_storage TYPE = S3 WITH (
    endpoint = '<rest-catalog-storage-endpoint>',
    access_key_id = '<access-key-id>',
    secret_access_key = '<secret-access-key>',
    region = '<region>',
    path_style = 'true'
);

This example shows the storage connection for an AWS cluster. On GCP, Redpanda Cloud creates a GCS storage connection (TYPE = GCS) with the corresponding GCS credentials. See CREATE STORAGE for the options Redpanda Cloud sets for each provider.

An Iceberg catalog points at the cluster’s REST catalog endpoint:

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage WITH (
    uri = '<rest-catalog-uri>',
    warehouse = '<warehouse>',
    auth_type = '<auth-type>'
);

The default_redpanda_catalog is linked to the Iceberg catalog. The pandaproxy_url option is required when a Redpanda catalog is linked:
```
ALTER REDPANDA CATALOG default_redpanda_catalog
USING CATALOG lakehouse_catalog WITH (
    pandaproxy_url = '<pandaproxy-url>'
);
```

To inspect the link on your cluster, run DESCRIBE REDPANDA CATALOG default_redpanda_catalog. For full option lists, see CREATE STORAGE, CREATE ICEBERG CATALOG, and CREATE REDPANDA CATALOG.

Map a topic as a SQL table

Define a SQL table against the Iceberg-enabled topic in default_redpanda_catalog. Replace orders and orders-value with your topic name and Schema Registry value subject:

CREATE TABLE default_redpanda_catalog=>orders WITH (
  topic = 'orders',
  schema_subject = 'orders-value'
);

Redpanda SQL reads the registered schema from Schema Registry to map fields to SQL columns.

When you query a table mapped from a Redpanda topic, Redpanda SQL also exposes two reserved metadata columns alongside your schema’s columns:

redpanda (a struct with topic-level metadata such as partition, offset, timestamp, key, and headers)
redpanda_raw (raw key and value bytes)

These column names are reserved. A topic schema with a top-level redpanda or redpanda_raw field conflicts with the metadata columns.

Query live and historical records together

Query the table using standard SELECT syntax. The query returns records from both the live Redpanda topic and the Iceberg-committed history in one result, with no overlap between them.

After any shape change to the Iceberg table or its corresponding Redpanda topic schema, run REFRESH <catalog>⇒<table> to update the schema view Redpanda SQL uses for query planning. Without a refresh, the next query against the table fails at planning time with Schema not found for Iceberg table '<table>'. REFRESH updates only the schema view, not data; you don’t run it between writes.

SELECT * FROM default_redpanda_catalog=>orders LIMIT 10;

Redpanda SQL plans the union internally, so you don’t write a UNION ALL.

Iceberg-committed data persists independently of Redpanda topic retention. Queries continue to return records past the Redpanda topic’s retention window, provided they were committed to Iceberg first.

Query decoded keys and headers

By default, the redpanda metadata column exposes the record key and header values as raw bytea. To query them as decoded values, set the key and header options when you map the table. For the full list, see CREATE TABLE options.

For an Iceberg-enabled topic, the table’s key mode must match how the topic encodes keys and headers into its Iceberg table, which is set by the topic’s redpanda.iceberg.mode property. See Configure key, value, and header translation. Set the CREATE TABLE options to match:

Topic redpanda.iceberg.mode CREATE TABLE options

Topic `redpanda.iceberg.mode`	`CREATE TABLE` options
`value_schema_id_prefix`	Omit the key options. The key stays `bytea`.
`key:mode=schema_id_prefix;value:mode=schema_id_prefix`	`key_decode_mode = 'schema_id_prefix'`
`key:mode=string;value:mode=schema_id_prefix`	`key_decode_mode = 'string'`
`headers:value_type=string;value:mode=schema_id_prefix`	`header_value_type = 'string'`

value_schema_id_prefix

Omit the key options. The key stays bytea.

key:mode=schema_id_prefix;value:mode=schema_id_prefix

key_decode_mode = 'schema_id_prefix'

key:mode=string;value:mode=schema_id_prefix

key_decode_mode = 'string'

headers:value_type=string;value:mode=schema_id_prefix

header_value_type = 'string'

If the table’s key mode doesn’t match the topic’s Iceberg encoding, the query fails at planning time with a type mismatch instead of returning incorrect data.

Redpanda SQL exposes decoded value fields as top-level columns, so a topic whose redpanda.iceberg.mode nests value fields under a value struct (value:layout=nested) is not supported. The Iceberg table’s value column has no counterpart in the flattened topic schema, so the query fails at planning time with the Kafka schema must be a name-superset of Iceberg schema error described in Handle schema differences.

Map the table with the matching options, then access the decoded key and header values in your query:

CREATE TABLE default_redpanda_catalog=>orders WITH (
  topic = 'orders',
  schema_subject = 'orders-value',
  key_decode_mode = 'schema_id_prefix',
  header_value_type = 'string'
);

SELECT ((redpanda).key).customer_id, (redpanda).headers
FROM default_redpanda_catalog=>orders;

When you decode the key with a schema, (redpanda).key is a struct whose fields you access with ((redpanda).key).field_name. With key_decode_mode = 'string', (redpanda).key is text. With header_value_type = 'string', each header value in (redpanda).headers is text. Run DESCRIBE TABLE default_redpanda_catalog⇒orders to see the decoded key type.

key_decode_mode and header_value_type are fixed for the lifetime of the table. ALTER rejects a change to either. To change how keys or header values are decoded, drop and recreate the table.

Handle schema differences

A topic schema can evolve over time. You might add or remove fields in your Schema Registry value subject as your application changes. Redpanda writes new records to the Iceberg table forward-only: it adds columns to the Iceberg table when the topic schema widens, but it does not drop columns from the Iceberg table when the topic schema narrows. As a result, the Iceberg table can carry columns the current topic schema doesn’t have.

Redpanda SQL handles this divergence by treating the topic’s Schema Registry subject as canonical and filling in missing fields on the Iceberg side. The rules that apply to a query against a linked Redpanda catalog:

The result’s column order, count, and types come from the topic schema.
If the Iceberg table is missing a column that the topic schema has, Redpanda SQL returns NULL for that column on Iceberg-side rows. This is common after adding a new column to the topic schema, since earlier records already in Iceberg don’t have it.
If the Iceberg table has columns the topic schema does not, the query fails at planning time with the error Kafka schema must be a name-superset of Iceberg schema. This happens when a column is removed from the topic’s Schema Registry subject. To resolve, re-add the column to the topic’s Schema Registry subject so the topic schema covers every column present in the Iceberg table.

Iceberg type limitations

When Redpanda SQL maps an Iceberg-enabled topic as a SQL table, it reads the Iceberg schema and converts each column type to a SQL type. The following column types are not supported:

Columns containing a map at any nesting level, whether at the top level, inside a struct, or as a list element. CREATE TABLE or REFRESH fails.
decimal columns:
- Precision greater than 18 fails at CREATE TABLE or REFRESH, because Redpanda SQL stores decimals as 64-bit integers with a maximum precision of 18, while Iceberg allows precision up to 38.
- Precision 18 or lower can be mapped, but Redpanda SQL does not support querying the values.
A list whose element type is also a list. CREATE TABLE or REFRESH fails. Lists of primitives work as expected, as do lists of structs whose fields don’t contain an unsupported type.

These schema-mapping constraints apply only to columns in the Iceberg schema. Columns sourced from live topic records are not subject to these type restrictions.

Query UUID columns

Redpanda SQL reads a UUID column as the uuid type. Text and string functions don’t operate on uuid values directly, so cast a UUID to text explicitly, as in id::text. For casts, UUID generation, and the sources a uuid column can come from, see UUID.

UUID columns previously read as text (and returned garbled values from the Iceberg table). If a query that expected a text column behaves differently, cast the column to text (id::text).