Skip to content

Commit 3f3ef83

Browse files
kbatuigasFeediver1
andauthored
SQL: rpk byoc apply and data type limitations (#603)
* Rerun rpk byoc apply command, add replicas field to API call * Data type limitations in Iceberg * Minor edit * Minor edit * Apply suggestions from review --------- Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
1 parent e50f7dd commit 3f3ef83

3 files changed

Lines changed: 27 additions & 6 deletions

File tree

modules/sql/pages/get-started/deploy-sql-cluster.adoc

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,6 @@ Cloud API::
4747
+
4848
--
4949
. Authenticate to the link:/api/doc/cloud-controlplane/topic/topic-cloud-api-overview[Cloud API]. For details, see link:/api/doc/cloud-controlplane/authentication[Authenticate to the Cloud API].
50-
// TODO: Is selecting capacity (currently node count, expected to change to RPUs)
51-
// available with this endpoint?
5250
. Make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster[`POST /v1/clusters`] request with `rpsql.enabled` set to `true` in the cluster spec:
5351
+
5452
[,bash]
@@ -66,19 +64,22 @@ curl -X POST "https://api.redpanda.com/v1/clusters" \
6664
"throughput_tier": "<tier>",
6765
"resource_group_id": "<resource-group-id>",
6866
"rpsql": {
69-
"enabled": true
67+
"enabled": true,
68+
"replicas": <n>
7069
}
7170
}
7271
}'
7372
----
7473
+
75-
For the full request body and field reference, see the link:/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster[Create Cluster API].
74+
For the full request body and field reference, see the link:/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster[Create Cluster API]. Use the `replicas` field to set the initial number of SQL compute nodes (minimum 1).
7675
. The request returns the ID of a long-running operation. Poll the link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/{operation.id}`] endpoint until the operation completes.
7776
--
7877
=====
7978

8079
=== On an existing cluster
8180

81+
NOTE: If your BYOC cluster on AWS was created before May 28, 2026, rerun `rpk cloud byoc aws apply --redpanda-id=<cluster-id>` to update the agent's IAM permissions before enabling Redpanda SQL.
82+
8283
[tabs]
8384
=====
8485
Cloud Console::
@@ -102,7 +103,7 @@ Cloud API::
102103
curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}" \
103104
-H "Authorization: Bearer $AUTH_TOKEN" \
104105
-H "Content-Type: application/json" \
105-
-d '{"rpsql":{"enabled":true}}'
106+
-d '{"rpsql":{"enabled":true,"replicas":<n>}}'
106107
----
107108
+
108109
The request returns the ID of a long-running operation. Poll the link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/{operation.id}`] endpoint until the operation completes:

modules/sql/pages/get-started/redpanda-sql-vs-postgresql.adoc

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,16 @@ SELECT ABS(-1.0);
121121
|working as expected
122122
|===
123123

124+
== Data type limitations
125+
126+
The following Iceberg column types have no valid Redpanda SQL type mapping and cause `CREATE TABLE` or `REFRESH` to fail:
127+
128+
* `map`
129+
* `decimal` with precision greater than 18
130+
* nested `list` types
131+
132+
For details, see xref:sql:query-data/query-iceberg-topics.adoc#iceberg-type-limitations[Iceberg type limitations].
133+
124134
== Suggested reading
125135

126136
* xref:sql:get-started/overview.adoc[Redpanda SQL Overview]

modules/sql/pages/query-data/query-iceberg-topics.adoc

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
:description: Query the history of a Redpanda topic in Iceberg, and run a single SQL query that spans live records and historical Iceberg-committed records.
33
:page-topic-type: how-to
44
:personas: app_developer, data_engineer
5-
:learning-objective-1: Identify the storage and catalog objects Cloud sets up for querying Iceberg-enabled topics
5+
:learning-objective-1: Describe the storage and catalog objects that Redpanda Cloud configures for Iceberg-enabled topic queries
66
:learning-objective-2: Query both the live records and the Iceberg-committed history of a topic in a single result
77
:learning-objective-3: Handle schema differences between a Redpanda topic and its Iceberg table
88

@@ -110,6 +110,16 @@ Redpanda SQL handles this divergence by treating the topic's Schema Registry sub
110110
* If the Iceberg table is missing a column that the topic schema has, Redpanda SQL returns `NULL` for that column on Iceberg-side rows. This is common after adding a new column to the topic schema, since earlier records already in Iceberg don't have it.
111111
* If the Iceberg table has columns the topic schema does not, the query fails at planning time with the error `Kafka schema must be a name-superset of Iceberg schema`. This happens when a column is removed from the topic's Schema Registry subject. To resolve, re-add the column to the topic's Schema Registry subject so the topic schema covers every column present in the Iceberg table.
112112

113+
== Iceberg type limitations
114+
115+
When Redpanda SQL maps an Iceberg-enabled topic as a SQL table, it reads the Iceberg schema and converts each column type to a SQL type. The following column types currently have no valid Redpanda SQL type mapping and cause `CREATE TABLE` or `REFRESH` to fail with an error.
116+
117+
* Columns of type `map` are not supported, whether at the top level or nested inside a struct.
118+
* `decimal` columns with precision greater than 18 cannot be mapped. Iceberg allows decimal precision up to 38, but Redpanda SQL stores decimals as 64-bit integers with a maximum precision of 18. Decimals with precision 18 or lower work as expected.
119+
* A `list` whose element type is also a `list` is not supported. Simple lists (list of primitives or list of structs) work as expected.
120+
121+
These schema-mapping constraints apply only to columns in the Iceberg schema. Columns sourced from live topic records are not subject to these type restrictions.
122+
113123
== Next steps
114124

115125
* xref:sql:query-data/query-streaming-topics.adoc[Query streaming topics]: the streaming-only query path, without the Iceberg history layer.

0 commit comments

Comments
 (0)