Add directory for antalya docs, skills, and designs#1673
Add directory for antalya docs, skills, and designs#1673hodgesrm wants to merge 1 commit intoAltinity:antalya-26.1from
Conversation
Signed-off-by: Robert Hodges <rhodges@altinity.com>
hodgesrm
force-pushed
the
01-alter-table-export-design
branch
from
06d5b84 to
227f7b1
Compare
arthurpassos
left a comment
arthurpassos
left a comment
There was a problem hiding this comment.
I see some mistakes done by Claude here in regards to plain object storage vs iceberg exports. I assume it is because it has consumed the antalya-26.1 branch docs that do not mention Iceberg at all. Those docs are in my 1618 PR.
Half way through, still more to come
| be easy to monitor. This design covers two new ClickHouse commands to | ||
| export data. | ||
|
|
||
| * `ALTER TABLE EXPORT PART` -- Exports a single part to Iceberg |
There was a problem hiding this comment.
Not only Iceberg, but plain object storage as well
|
|
||
| * `ALTER TABLE EXPORT PART` -- Exports a single part to Iceberg | ||
|
|
||
| * `ALTER TABLE EXPORT PARTITION` -- Exports one or more partitions to Iceberg |
There was a problem hiding this comment.
As of now, it exports one partition - not more than that. That partition can have multiple parts, yes.
There was a problem hiding this comment.
Ack. However, I believe the design should properly cover the EXPORT PARTITION ALL case, even if it's not yet covered.
|
|
||
| These commands replace `INSERT INTO ... SELECT FROM` pipelines that select | ||
| rows and write them out to one or more Parquet files. This approach | ||
| costs an extra decode/sort pass per export, does not coordinate across |
There was a problem hiding this comment.
an extra decode/sort
INSERT ... SELECT does penalize on sorting, but not sure about extra decode. INSERT ... SELECT also does extra work on partitioning
|
|
||
| 2. **Efficient, order-preserving writes.** Write a specified `MergeTree` | ||
| part (or every part of a specified partition) to an object-storage | ||
| destination in Parquet, preserving the source part's sort order, |
There was a problem hiding this comment.
in Parquet
In theory, we support other formats as well - tho I have never tested it. I would keep it as parquet
There was a problem hiding this comment.
Noted. File formats other than Parquet are listed as non-requirements.
| 2. **Efficient, order-preserving writes.** Write a specified `MergeTree` | ||
| part (or every part of a specified partition) to an object-storage | ||
| destination in Parquet, preserving the source part's sort order, | ||
| without using a `SELECT` pass and also minimizing the RAM required |
There was a problem hiding this comment.
also minimizing the RAM required to hold data during transfer
I would remove this portion. We just rely on the ClickHouse internal pipeline, which is probably very similar to what INSERT ... SELECT uses.
There was a problem hiding this comment.
My point was that we want to use the same or less memory than doing a SELECT. Design updated to reflect this.
| -- Split large part across multiple Parquet files | ||
| ALTER TABLE big EXPORT PART '2025_0_32_3' TO TABLE big_dest | ||
| SETTINGS allow_experimental_export_merge_tree_part = 1, | ||
| export_merge_tree_part_max_bytes_per_file = 10000000, |
There was a problem hiding this comment.
export_merge_tree_part_max_bytes_per_file
We have a funny situation with this. When I added this setting, ClickHouse did not have the ability to split out parquet files. Plus, we did not support exporting to Iceberg.
The iceberg engine now has its own settings. Which one should we respect?
There was a problem hiding this comment.
Good point. Seems like we would go with the Iceberg engine for consistency and to simplify EXPORT commands.
| ALTER TABLE rmt_table EXPORT PARTITION ID '2020' TO TABLE s3_table; | ||
|
|
||
| -- Cancel by filter | ||
| KILL EXPORT PARTITION |
There was a problem hiding this comment.
Worth noting it is the same filter you would apply to read from system.replicated_partition_exports;
|
|
||
| Create a `ReplicatedMergeTree` source, seed two partitions from | ||
| `system.numbers`, create a hive-partitioned S3 destination (the on-disk | ||
| shape an external Iceberg catalog such as Glue / REST / Nessie / |
There was a problem hiding this comment.
Well.. this is a plain object storage end to end case, not iceberg.
|
|
||
| The following notes expand on expected behavior of commands. | ||
|
|
||
| 1. `ALTER TABLE t EXPORT PART 'p' TO TABLE s3_t` writes |
| `<dir>/commit_<part>_<checksum>`, readable end-to-end via | ||
| `SELECT * FROM s3(...)` in tests `03572_*` and `03608_*`. | ||
|
|
||
| 2. `ALTER TABLE rmt EXPORT PARTITION ID 'p' TO TABLE s3_t` exports |
There was a problem hiding this comment.
Based on the list of parts that the replica that received the command sees.
2 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added a skill for Antalya design creation and review. Also added a first cut design spec for ALTER TABLE EXPORT based on current documentation and tests.
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: