Redis & ClickHouse: Storage & Serving
Post 1 opened with a scene: a 2 TB Postgres, a four-hour revenue report, and a product manager asking for "a dashboard that updates every minute." The track since then has built the pipeline that feeds that dashboard — Kafka carrying events, Spark crunching them. But a pipeline that nobody can read from at speed is a write-only achievement. The dashboard has two questions it must answer fast: "what are this user's features right now?" (milliseconds, millions of times a day) and "how did revenue trend by country this quarter?" (billions of rows, seconds). One system cannot answer both well. This post gives you the two that can: Redis for the millisecond question, ClickHouse for the billion-row question.
The Data & Messaging track's "Caching in Java" post already covered Redis-as-cache — cache-aside, eviction, TTLs, thundering herds. We won't repeat any of that. This post goes deeper into Redis-as-serving-layer: its data structures as an API, Streams as a log, and the feature-store pattern that puts model features behind a millisecond lookup. Then the second half switches engines entirely: ClickHouse, the columnar OLAP database, with a real 1M-row table, real aggregation timings, a materialized view, and a JDBC program in Java.
Lab honesty, up front. Everything below ran in this lab: Redis 8.10.2 built from source (gcc/make; the optional Rust-based modules — redisearch, redisjson, redistimeseries — failed to link, the core server and CLI built fine), ClickHouse 26.10.1.1704 as the single official binary, Jedis 7.5.3 and clickhouse-jdbc 0.10.0 from Maven Central, all Java on Temurin JDK 17.0.20.1. Two sandbox quirks are documented in the QA file and noted where they matter: single-row INSERT ... VALUES through the native client hung after the materialized view was created (HTTP POST inserts worked), and this sandbox intercepts raw TCP from Java processes to 127.0.0.1 (a policy message is returned instead of the server's bytes — verified by the coordinator with a socket probe). IPv6 loopback is unaffected, so both Java demos below connect via ::1 — the only deviation from what you'd run on your machine, where localhost works as documented. No output below is invented; timings are real wall/server numbers from these runs.
Redis is a data-structure server
The mental shift from the caching post: stop thinking of Redis as "a place to put strings so Postgres doesn't have to answer" and start thinking of it as a server whose API is data structures. Each structure exists because a read pattern needed it. Watch how each command below answers a different question:
127.0.0.1:6379> SET product:1001:name Acme Anvil
OK
127.0.0.1:6379> INCR pageviews:home
1
127.0.0.1:6379> INCR pageviews:home
2
127.0.0.1:6379> GET pageviews:home
2
127.0.0.1:6379> HSET user:123 name Maya tier pro logins 42
3
127.0.0.1:6379> HGETALL user:123
name
Maya
tier
pro
logins
42
127.0.0.1:6379> HGET user:123 tier
proStrings are the counters and flags: INCR is atomic, so a pageview counter never loses an increment under concurrency — no read-modify-write race. Hashes are the objects: one key (user:123), many fields, and you can fetch a single field (HGET) without pulling the whole object. If the caching post taught you to store a serialized JSON blob under one key, this is the upgrade: field-level access instead of deserialize-everything.
127.0.0.1:6379> ZADD leaderboard 1500 alice 2300 bob 900 carol 3100 dave
4
127.0.0.1:6379> ZREVRANGE leaderboard 0 2 WITHSCORES
dave
3100
bob
2300
alice
1500
127.0.0.1:6379> ZRANK leaderboard bob
2
127.0.0.1:6379> RPUSH jobs:queue job-1 job-2 job-3
3
127.0.0.1:6379> LRANGE jobs:queue 0 -1
job-1
job-2
job-3
127.0.0.1:6379> LPOP jobs:queue
job-1
2
127.0.0.1:6379> LLEN jobs:queue
2Sorted sets are the leaderboard — the classic for a reason: members ordered by score, top-N in one command, rank lookup in logarithmic time. Note ZRANK leaderboard bob returned 2: ranks are zero-based and ascending, so bob (2300) sits third of four. Lists are the queue: push right, pop left, and you have FIFO with LLEN telling you the backlog depth.
Principle: serve from the structure that matches the read. A leaderboard in a hash means sorting in your application on every read; in a sorted set it's one command. A queue in a string means serializing the whole backlog to pop one job. The structure is the query plan — pick it the way you'd pick an index: by the read you'll do most.
Streams: the log you already understand
Post 6 built your streaming mental model on Kafka: an append-only log, consumer groups, offsets, at-least-once delivery with idempotent consumers. Redis Streams is that same model in miniature — one stream, no partitions, living in memory. The commands map almost one-to-one, which is exactly why it's worth learning: it's the smallest possible implementation of the ideas post 6 taught at scale.
127.0.0.1:6379> XADD payments:stream * user 123 amount 49.99
1791345975631-0
127.0.0.1:6379> XADD payments:stream * user 456 amount 19.99
1791345975638-0
127.0.0.1:6379> XREAD COUNT 10 STREAMS payments:stream 0
payments:stream
1791345975631-0
user
123
amount
49.99
1791345975638-0
user
456
amount
19.99
127.0.0.1:6379> XGROUP CREATE payments:stream fraud-checkers 0
OK
127.0.0.1:6379> XREADGROUP GROUP fraud-checkers svc-1 COUNT 10 STREAMS payments:stream >
payments:stream
1791345975631-0
user
123
amount
49.99
1791345975638-0
user
456
amount
19.99Read it against Kafka: XADD appends an entry (the * auto-generates the ID — millisecond timestamp plus sequence, so IDs are roughly time-ordered). XREAD ... 0 reads from the beginning, like seeking to the earliest offset. XGROUP CREATE makes a consumer group; XREADGROUP ... > delivers only new messages to this consumer — the > is the "give me what nobody in my group has seen" cursor.
Delivery semantics are honest at-least-once, and the mechanism is visible — no framework magic:
127.0.0.1:6379> XPENDING payments:stream fraud-checkers
2
1791345975631-0
1791345975638-0
svc-1
2
127.0.0.1:6379> XACK payments:stream fraud-checkers 1791345975631-0 1791345975638-0
2
127.0.0.1:6379> XPENDING payments:stream fraud-checkers
0XPENDING shows the unacknowledged backlog: 2 messages, the ID range, and which consumer holds them (svc-1, 2 messages). XACK acknowledges by ID — the offset commit, spelled out. Crash between XREADGROUP and XACK and the messages stay pending for redelivery: at-least-once, with idempotent consumers as the answer, exactly as post 6 taught.
The honest differences from Kafka, so you don't over-extrapolate: a stream lives on a single Redis node (or shard) — there are no partitions to scale consumption across, so throughput tops out at what one node can append. Retention is by explicit trimming (XTRIM), not time/size-based segment deletion. And there's no exactly-once framework: it's XACK plus idempotence, your code's job. Rule of thumb, not a law: Streams for service-to-service eventing inside one team's latency budget; Kafka when the log itself is the system of record.
The serving pattern: a feature-store sketch
Here's the pattern that ties the structures together. A feature store sits between your offline ML work and your live API: a batch job computes features (orders in the last 7 days, average order value, a churn score) and writes them somewhere the request path can fetch in single-digit milliseconds at prediction time. The write path is a hash; the read path is a field lookup; the freshness contract is a TTL. This is Redis-as-serving-layer rather than Redis-as-cache — the data isn't a copy of something in Postgres, it's the primary home of the computed features.
A real Java program with Jedis 7.5.3 (Maven Central), compiled and run against the lab server:
import redis.clients.jedis.Jedis;
import java.util.List;
import java.util.Map;
public class FeatureStoreDemo {
public static void main(String[] args) {
// Lab note: ::1 because this sandbox intercepts Java TCP to 127.0.0.1
// (see the honesty box); on your machine, "localhost" works as documented.
try (Jedis jedis = new Jedis("::1", 6379)) {
System.out.println("PING -> " + jedis.ping());
// 1. Write the feature vector for user 123 (an offline job would refresh this)
String key = "user:123:features";
jedis.hset(key, Map.of(
"country", "IN",
"age_bucket", "25-34",
"orders_last_7d", "6",
"avg_order_value", "34.50",
"churn_score", "0.17"));
jedis.expire(key, 300); // features go stale: 5-minute TTL
System.out.println("WROTE " + key + " (5 fields, TTL 300s)");
// 2. Read-your-write: the serving path reads it straight back
Map<String, String> features = jedis.hgetAll(key);
System.out.println("READ " + key + " -> " + features);
System.out.println("TTL " + key + " -> " + jedis.ttl(key) + "s remaining");
// 3. Same serving layer, different structure: a leaderboard is a sorted set
jedis.zadd("game:leaderboard", Map.of("alice", 1500.0, "bob", 2300.0, "dave", 3100.0));
List<String> top = jedis.zrevrange("game:leaderboard", 0, 2);
System.out.println("TOP-3 game:leaderboard -> " + top);
}
}
}$ javac -cp "jedis-7.5.3.jar:commons-pool2-2.12.1.jar:gson-2.13.2.jar:..." FeatureStoreDemo.java
$ java -cp "jedis-7.5.3.jar:...:." FeatureStoreDemo
PING -> PONG
WROTE user:123:features (5 fields, TTL 300s)
READ user:123:features -> {country=IN, avg_order_value=34.50, orders_last_7d=6, age_bucket=25-34, churn_score=0.17}
TTL user:123:features -> 300s remaining
TOP-3 game:leaderboard -> [dave, bob, alice]Three things to notice. First, the read-your-write flow: the write returns, and the same program reads the hash back immediately — no eventual consistency, no cache warming, the serving path sees what the writer wrote. Second, the TTL is the staleness contract: features recomputed every few minutes by the offline job, and if the job dies, keys expire rather than serving week-old churn scores to the model. Third, one client, two structures: the hash for the feature vector, the sorted set for the leaderboard — the principle from the last section, in code.
Durability, precisely
Now the question every experienced engineer asks at this point: "it's all in memory — what happens when the box dies?" The precise answer matters, because the imprecise versions ("Redis loses everything on restart" / "Redis is fully durable") are both wrong. Redis has two persistence mechanisms, and I ran both:
- RDB snapshots. Point-in-time dumps of the dataset, taken by a background fork (
BGSAVE). Configured here withsave "3600 1 300 100 60 10000"— snapshot if at least 1 change in an hour, 100 changes in 5 minutes, or 10,000 in a minute. You can lose up to the last snapshot window. - AOF (append-only file). Every write appended to a log, fsynced every second here (
appendonly yes,appendfsync everysec). Worst case you lose about a second of writes.
$ redis-cli BGSAVE
Background saving started
$ ls data/
appendonlydir dump.rdb
$ redis-cli INFO persistence | grep -E "rdb_last_save|aof_enabled"
rdb_changes_since_last_save:0
rdb_last_save_time:1791345777
aof_enabled:1And the test that actually matters — kill the server and restart it:
$ redis-cli SET durability:probe "survives-restart"
OK
$ pkill -x redis-server; sleep 2 # server is down
$ redis-server --port 6379 --dir ./data --appendonly yes ... # back up
$ redis-cli ping
PONG
$ redis-cli GET durability:probe
survives-restart
$ redis-cli ZCARD leaderboard
4
$ redis-cli XLEN payments:stream
2
$ redis-cli HGET user:123 tier
proEverything survived: the probe key, the leaderboard, the stream, the user hash. So can Redis be your system of record? The honest answer: it has the mechanisms, but durability is a design, not a flag. AOF-everysec plus replicas plus tested restore procedures plus monitoring of replication lag is a deliberate durability architecture — and several companies run it. What you must not do is casually treat a cache as durable truth: default configs, no replicas, no restore drills, and a shrug. The lesson isn't "Redis can't be durable." It's "don't confuse 'it survived my kill test' with 'it will survive yours at 3 AM.'" If the data's loss would page someone, design for it like a database: replication, backups you have actually restored, and a measured recovery time.
ClickHouse: think in columns
Redis answers "what about this one?" in milliseconds. The dashboard's other question — "how did revenue trend by country this quarter?" — is the opposite shape: touch a billion rows, return one small answer, in seconds. A row-oriented database lays out all of a row's columns together on disk; to sum one column across a billion rows it must walk every row's full width (Postgres has indexes and a serious optimizer, so indexed lookups are fast — but a full-table aggregation over a few columns still reads far more data than the query needs). A columnar engine lays out each column separately: the same query reads only the columns it references, and because values inside one column look alike, they compress dramatically.
ClickHouse is the open-source columnar OLAP engine built on that idea, with a SQL interface and a MergeTree table engine: data lands in sorted parts, background merges compact them, and the ORDER BY key acts as a sparse primary index that lets queries skip whole granules of data. Real lab, real numbers — a table of one million click events:
$ clickhouse client --query "
CREATE TABLE click_events (
ts DateTime,
user_id UInt32,
page LowCardinality(String),
country FixedString(2),
amount Decimal(10, 2),
device LowCardinality(String)
) ENGINE = MergeTree()
ORDER BY (ts, user_id)"
$ time clickhouse client --query "
INSERT INTO click_events
SELECT
toDateTime('2026-09-01 00:00:00') + (number % 2592000),
number % 100000,
['home','search','product','cart','checkout'][1 + number % 5],
['US','IN','GB','DE','BR'][1 + number % 5],
toDecimal64((number * 37) % 10000 / 100.0, 2),
['web','ios','android'][1 + number % 3]
FROM numbers(1000000)"
real 0m6.768s
$ clickhouse client --query "
SELECT count() AS rows, formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts WHERE table = 'click_events' AND active"
1 11.51 MiBOne million rows, six columns, generated and inserted in 6.8 seconds — and occupying 11.51 MiB on disk. That's the columnar compression story in one number: the repetitive columns (page, device, the modulo-pattern user_id) collapse almost to nothing under LZ4, and even the pseudo-random amount column rides along cheaply. (LowCardinality(String) is ClickHouse's dictionary encoding for low-distinct-count strings — the idiomatic type for columns like page and device.)
Now the analytical queries, with real server-side timings (-t prints execution time to stderr):
$ clickhouse client -t --format PrettyCompact --query "
SELECT country, count() AS events, round(sum(amount), 2) AS revenue
FROM click_events GROUP BY country ORDER BY revenue DESC"
┌─country─┬─events─┬──revenue─┐
1. │ GB │ 200000 │ 10002897 │ -- 10.00 million
2. │ BR │ 200000 │ 10000933 │ -- 10.00 million
3. │ IN │ 200000 │ 9998875 │ -- 10.00 million
4. │ DE │ 200000 │ 9996866 │ -- 10.00 million
5. │ US │ 200000 │ 9994856 │ -- 9.99 million
└─────────┴────────┴──────────┘
0.507
$ clickhouse client -t --format PrettyCompact --query "
SELECT toStartOfDay(ts) AS day, count() AS events
FROM click_events GROUP BY day ORDER BY day LIMIT 3"
┌─────────────────day─┬─events─┐
1. │ 2026-09-01 00:00:00 │ 86400 │
2. │ 2026-09-02 00:00:00 │ 86400 │
3. │ 2026-09-03 00:00:00 │ 86400 │
└─────────────────────┴────────┘
0.539Full-table aggregations over a million rows in about half a second — and the second run of the country query dropped to 0.365s warm. The query touched only the country and amount columns; ts, user_id, page, and device were never read. (The -- 10.00 million annotations are ClickHouse's Pretty format being helpful with large numbers — real output, kept verbatim.)
One honest footnote from the lab: SELECT count(), uniq(user_id) FROM click_events reported 100315 distinct users, but the generator produced exactly 100000 (number % 100000). That's not a bug — uniq() is HyperLogLog, approximate by design and cheap; uniqExact() counts precisely and costs more. Know which one your query used before you quote its number.
Materialized views: pre-aggregation that stays fresh
The country query above scans a million rows in half a second. At a billion rows it won't — and dashboards don't need the raw rows anyway, they need the aggregates. A materialized view is ClickHouse's answer: you define the aggregation once, and every insert into the source table automatically feeds the pre-aggregated view. The dashboard then queries dozens of rows instead of billions.
$ clickhouse client --query "
CREATE MATERIALIZED VIEW mv_daily_revenue
ENGINE = SummingMergeTree()
PARTITION BY toYYYYMM(day)
ORDER BY (day, country)
POPULATE AS
SELECT toDate(ts) AS day, country, count() AS events, sum(amount) AS revenue
FROM click_events
GROUP BY day, country"
MV CREATEDPOPULATE backfills the view from existing data — without it, the view would only see rows inserted after its creation. SummingMergeTree is the engine that knows how to combine the pre-aggregated rows. And the consistency check every materialized view deserves:
$ clickhouse client --query "SELECT sum(events) AS mv_total FROM mv_daily_revenue"
1000000
$ clickhouse client -t --format PrettyCompact --query "
SELECT day, sum(events) AS events, round(sum(revenue), 2) AS revenue
FROM mv_daily_revenue GROUP BY day ORDER BY day LIMIT 3"
┌────────day─┬─events─┬────revenue─┐
1. │ 2026-09-01 │ 86400 │ 4316566.51 │ -- 4.32 million
2. │ 2026-09-02 │ 86400 │ 4319366.47 │ -- 4.32 million
3. │ 2026-09-03 │ 86400 │ 4322166.5 │ -- 4.32 million
└────────────┴────────┴────────────┘
0.492The view's totals reconcile exactly with the source table (1,000,000 events), and the dashboard query over the pre-aggregated view returns in under half a second. Two honest details from the lab. First, new inserts flow into the view synchronously — a row inserted after creation showed up in the view on the next query. Second, the view briefly held two rows for the same (day, country) key — one from the backfill, one from the later insert — because SummingMergeTree collapses same-key rows during background merges, not at insert time. That's why the query aggregates with sum(events) instead of reading raw rows: the engine guarantees the collapsed result only after merges run. Always aggregate over a SummingMergeTree; never assume one row per key.
JDBC from Java
The analysts get SQL; your services get JDBC. The official com.clickhouse:clickhouse-jdbc:0.10.0 driver (Maven Central) speaks the HTTP interface — the same queries as the lab, from a Java program:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
public class ClickHouseJdbcDemo {
public static void main(String[] args) throws Exception {
Class.forName("com.clickhouse.jdbc.ClickHouseDriver");
// Lab note: [::1] and ?compress=0 are sandbox workarounds (see below);
// on your machine, jdbc:clickhouse://localhost:8123/default works as documented.
try (Connection conn = DriverManager.getConnection("jdbc:clickhouse://[::1]:8123/default?compress=0");
Statement st = conn.createStatement()) {
try (ResultSet rs = st.executeQuery("SELECT version()")) {
rs.next();
System.out.println("server version -> " + rs.getString(1));
}
try (ResultSet rs = st.executeQuery(
"SELECT country, count() AS events, round(sum(amount), 2) AS revenue " +
"FROM click_events GROUP BY country ORDER BY revenue DESC")) {
System.out.println("country | events | revenue");
while (rs.next()) {
System.out.printf("%-7s | %7d | %12s%n",
rs.getString("country"), rs.getLong("events"), rs.getString("revenue"));
}
}
try (ResultSet rs = st.executeQuery(
"SELECT sum(events) AS total_events, round(sum(revenue), 2) AS total_revenue " +
"FROM mv_daily_revenue")) {
rs.next();
System.out.println("mv_daily_revenue total -> events=" + rs.getLong("total_events")
+ ", revenue=" + rs.getString("total_revenue"));
}
}
}
}$ javac -cp "clickhouse-jdbc-0.10.0.jar:httpclient5-5.4.4.jar:..." ClickHouseJdbcDemo.java
$ java -cp "clickhouse-jdbc-0.10.0.jar:...:." ClickHouseJdbcDemo
server version -> 26.10.1.1704
country | events | revenue
GB | 200000 | 10002897.00
BR | 200000 | 10000933.00
IN | 200000 | 9998875.00
DE | 200000 | 9996866.00
US | 200001 | 9994905.99
mv_daily_revenue total -> events=1000001, revenue=49994476.99The numbers reconcile with the clickhouse-client lab exactly: the same five countries, the same totals — with the US row showing 200001 events because a test row was inserted mid-lab (and the materialized view picked it up, proving the synchronous feed). About those two workarounds in the connection URL, since honesty is the policy: this sandbox intercepts raw TCP from Java processes to 127.0.0.1 but not to IPv6 loopback, so the demo connects via [::1]; and this driver/server combination misframed LZ4-compressed responses, so ?compress=0 disables client-side compression. Neither affects the API the post teaches — on a normal machine the documented URL works unchanged.
Two halves, one serving layer
Step back and place the two halves on the map from post 1. Both live on the serving row, but they answer different questions:
| Redis | ClickHouse | |
|---|---|---|
| Question | "What about this one, right now?" | "What happened across all of them?" |
| Access pattern | Point lookups by key, tiny reads | Full scans over a few columns, aggregations |
| Latency | Single-digit milliseconds | Seconds over billions of rows |
| Data shape | Structures chosen per read: hashes, sorted sets, streams | Wide tables, columnar, pre-aggregated views |
| Durability story | RDB + AOF are real; system-of-record use needs deliberate design | MergeTree parts on disk; built to be the analytical store |
| Reach for it when | The request path needs an answer before the user blinks | The analyst needs an answer before lunch |
And the pipeline between them is the rest of the track: Kafka carries the events (Data & Messaging track), Spark crunches them into features and aggregates (posts 3–6), the features land in Redis hashes with TTLs, the raw events land in ClickHouse MergeTree tables, and the materialized views keep the dashboards fast. The capstone wires all of it together.
Cheat sheet
- Redis is a data-structure server: strings (atomic counters), hashes (field-level objects), sorted sets (leaderboards, rank in log time), lists (queues). Serve from the structure that matches the read.
- Redis Streams is the Kafka mental model in miniature:
XADDappends,XREADGROUP ... >delivers new messages per consumer group,XACKcommits,XPENDINGshows the unacked backlog. At-least-once; idempotent consumers are your job. No partitions — one node's throughput is the ceiling. - Feature-store pattern: offline job computes features →
HSET user:123:features→ serving pathHGETALLin milliseconds. The TTL is the staleness contract; the hash is the primary home, not a cache copy. - Redis durability is real (RDB snapshots + AOF, verified across a kill/restart here) but it's a design — replicas, tested restores, measured recovery — not a flag. Don't casually treat a cache as durable truth.
- ClickHouse stores columns, not rows: an aggregation reads only the columns it references (here: 1M rows in 11.51 MiB, grouped in ~0.5s).
MergeTree+ORDER BYkey = sorted parts with a sparse primary index. LowCardinality(String)for low-distinct-count strings;uniq()is HyperLogLog-approximate (lab: 100315 vs exactly 100000) —uniqExact()when the number matters.- Materialized views pre-aggregate on insert: define once with
POPULATEto backfill, query the small view instead of the raw table.SummingMergeTreecollapses same-key rows at background-merge time — always aggregate withsum(), never assume one row per key. - JDBC:
com.clickhouse:clickhouse-jdbcfrom Maven Central, standardjava.sqlAPI over the HTTP interface. Same queries as the CLI, same numbers. - Redis for "this one, right now" (ms point lookups); ClickHouse for "all of them, summed up" (analytical scans). Write the query first; the storage picks itself.
What's next
You now own the serving row of the platform diagram: millisecond feature lookups behind hashes with TTLs, event streams with consumer groups, billion-row aggregations over columnar storage, and pre-aggregated views that keep dashboards fast. But so far each post has been one row in isolation — Kafka in its track, Spark in its posts, Redis and ClickHouse in this one. Nobody's data lives in one row.
The next post, Capstone: End-to-End Pipeline, wires all five rows together: events into Kafka, Spark turning them into features and aggregates, Redis serving the features, ClickHouse serving the analytics — orchestrated, observable, and recoverable. The whole track, one running system.
Field check before you move on: (1) Take the leaderboard from this post and add expiring weekly leaderboards — what key scheme and TTL would you use, and what breaks if two app servers ZADD concurrently? (2) Write the XREADGROUP/ XACK loop for the fraud-checkers group in Jedis, with a crash between read and ack — prove the message gets redelivered. (3) In ClickHouse, change the ORDER BY to (country, ts), re-insert, and compare the country-aggregation timing — explain what changed and why. (4) Build the materialized view without POPULATE, insert 1000 rows, and confirm the view holds exactly those 1000 — then argue when you'd want each behavior. (5) Kill your Redis with AOF disabled and RDB-only snapshots, then restart — measure what you lost, and write the one-paragraph durability design you'd defend in review.
Continue: Java Learning Roadmap 2026
Comments
Post a Comment