ClickHouse on object storage.

On a bare-metal server, the size of the dataset picks the catalog row, and the row comes with CPU and RAM you did not ask for. We priced a terabyte three ways, in public list prices, and stopped buying storage and compute as one unit.

Every table part on our object-storage node is an object in a bucket. The box holds a 2.5 TB cache, about a gigabyte of metadata and a Keeper, and until a second replica joins it is the only path to the data. Here are the three disk layers, the table definitions, what the nightly backup saves and what it skips, and how ClickHouse Cloud does the same thing behind an engine it does not release.

ClickHouse's guide to running its open-source build on object storage calls the setup more complicated and recommends its cloud. The docs then list what is missing, in pieces. We collected eight items, and for each one we say what it means on a running node and what we are doing about it, with the config or the command.

A bulk load into our object-storage ClickHouse node left 91 merges running at once, each failing on memory and being picked up again, and on R2 every part a merge writes is a set of metered uploads. Six levers make the node write fewer, larger objects. Each is here with its value, along with the one ClickHouse has retired and the pool settings that let merges finish.

Our runbook says a fresh machine, a one-gigabyte metadata snapshot and access to the bucket are enough to bring the node back. Nobody has timed it. This is the drill we are running, the failures we expect, and the stages the clock will cover.

A replicated Kafka cluster bills a byte three times: the replica disks, the traffic between zones, and the brokers that own the disks. Richard Artoul's WarpStream hands the first two to the bucket and makes the brokers stateless, and his posts print the bill by line, latency floor included. Profiled from his own published account.

Vector and full-text search live in RAM by tradition, so the bill grows with the corpus whether anyone searches it or not. turbopuffer keeps the durable copy in object storage, caches on SSD, and prints both latencies: a p90 of 444 ms cold and 10 ms warm over a million vectors. Profiled from Simon Hørup Eskildsen's own post.