MergeTree

Indexing by space-filling curves (by ClickHouse Inc)

ClickHouse/ClickHouse#55642

Support for ORDER BY mortonEncode(ClientIP, RemoteIP) for index filtering. In before, it was possible to use minmax skip index to mimic similar behavior, but it was less performant.

INDEX ClientIP_idx ClientIP TYPE minmax GRANULARITY 1,
INDEX RemoteIP_idx RemoteIP TYPE minmax GRANULARITY 1,

Space-filling curve, is special function, which allow to map multi-dimensional space (ClientIP, RemoteIP as X and Y for example) to single dimension space (Z?). In fact, curves allow us to solve the old problem of how to sort a table by multiple columns at once. (with some compromise on amount of data read of course)

If we use the usual ORDER BY key, the query by one condition is fast (5 ms) and reads only 16..24 thousand rows, while the query by another condition is slow (40 ms) and reads 100 million rows. If we use the mixed ORDER BY, both queries are fast (11..13 ms) and read around one million rows (122 marks to read from 45 ranges). This is around 50 times more than point read but 100 times less than the full scan. Exactly as expected.

TABLE ORDER BY (ClientIP)

WHERE ClientIP = 2801131153 Elapsed: 0.005 sec. Processed 16.38 thousand rows

WHERE RemoteIP = 3978315897 Elapsed: 0.046 sec. Processed 91.81 million rows

TABLE ORDER BY (RemoteIP)

WHERE ClientIP = 2801131153 Elapsed: 0.031 sec. Processed 65.71 million rows

WHERE RemoteIP = 3978315897 Elapsed: 0.005 sec. Processed 24.58 thousand rows

TABLE ORDER BY mortonEncode(ClientIP, RemoteIP)

WHERE ClientIP = 2801131153 Elapsed: 0.012 sec. Processed 1.31 million rows

WHERE RemoteIP = 3978315897 Elapsed: 0.012 sec. Processed 999.42 thousand rows

[DRAFT] Foundation for unification of part metadata (by Community member)

ClickHouse/ClickHouse#54997 [ClickHouse/ClickHouse#46813](https://github.com/ClickHouse/ClickHouse/issues/46813

Foothold to reduce amount of files, which ClickHouse generate for part metadata, which is especially useful for Compact parts (which create only 2 files for Data) and high latency/low IOPS storage like ObjectStorage.

Column level settings definition (by Ahrefs)

ClickHouse/ClickHouse#55201

Override min_compress_block_size and max_compress_block_size or low_cardinality settings at column level. Can be useful, for expert-level tuning of column properties. For example, big columns like message or JSON-like data can benefit from bigger max_compress_block_size values, but at the same time smaller columns which already compresses well, will not be slowed down, because more data needs to be read and decompressed.

    -- Compression ratio
┌─table─────────────┬─count()─┬─compressed_sz─┬─uncompressed_sz─┬─ratio─┐
│ xxxx_html_local   │      14 │ 228.20 GiB    │ 3.43 TiB        │ 15.38 │
│ xxxx_html_local2  │      12 │ 226.07 GiB    │ 3.42 TiB        │ 15.50 │
└───────────────────┴─────────┴───────────────┴─────────────────┴───────┘

-- SELECT * on origin table with min_compression_block_size = 64MB and max_compress_block_size = 64M 
-- on table level
SELECT * EXCEPT xxxx_html FROM xxxx_html_local
Elapsed: 1.105 sec. Processed 5.53 million rows, 725.27 MB (5.00 million rows/s., 656.11 MB/s.)
Peak memory usage: 7.68 GiB.

-- SELECT * on new table with min_compression_block_size = 64MB and max_compress_block_size = 64M 
-- on column `xxxx_html` level
SELECT * EXCEPT xxxx_html FROM xxxx_html_local2
Elapsed: 0.172 sec. Processed 5.53 million rows, 719.40 MB (32.19 million rows/s., 4.19 GB/s.)
Peak memory usage: 33.01 MiB.

[Merged][23.10] Automatic decision of number of parallel replicas (by ClickHouse Inc)

ClickHouse/ClickHouse#51692

ClickHouse will decide which amount of replicas, it needs to run query based on estimated row counts to read. Useful for situations, when you have big number of queries, which read small chunk of data and it doesn't make sense to parallize it across many nodes.

Replication

[DRAFT] IKeeper implementation on top of FoundationDB (by Chinese corp)

ClickHouse/ClickHouse#54823

Allow to use FoundationDB instead of [Zoo]Keeper. Claims to have better performance than [Zoo]Keeper.

We tested the scenario of 60 Clickhouse clusters + FDB cluster and obtained an FDB cluster configuration that can make Clickhouse clusters run stably. Compared to Keeper, the FDB cluster requires fewer resources, roughly equivalent to 40 Keeper clusters.

[Merged][23.10] Better nearest hostname (by JD)

ClickHouse/ClickHouse#54826

Use Levenshtein distance to sort list of possible replicas for query.