HDFS & MapReduce: The Foundations

Picture the nightly revenue report from post 1 — one big SQL query that scans 2 TB and times out. Now imagine it doesn't live in Postgres at all. The orders live in files on a hundred cheap machines, and instead of one database grinding through them, a hundred small programs each read their own slice and combine answers. That's the idea this post is built on — and the first thing you need to understand about it is where the bytes actually sit. Everything in data engineering starts with storage, and the storage this whole industry grew out of is HDFS.

HDFS and MapReduce are old technology — HDFS from 2006, MapReduce from the 2004 Google paper. You will likely never write a production MapReduce job. But here's why this post exists: every tool you'll meet later leaks this foundation. Spark's shuffle is MapReduce's shuffle. Partitioning schemes only make sense once you've watched a hot key stall a reducer. And when a Spark job mysteriously reads slowly from the lake, the answer is usually a block-placement decision made for Hadoop fifteen years ago. So we'll go hands-on: real blocks you can observe, and a real Java MapReduce job — run end to end, shuffle counters and all.

Lab honesty, up front. This lab runs Hadoop 3.3.6 in two modes on one machine: a pseudo-distributed HDFS (real NameNode and DataNode daemons on localhost — real blocks, real replication metadata) and standalone/local MapReduce (the job runs in a single JVM, no cluster, no daemons). YARN is covered conceptually — spinning up a real ResourceManager just to watch one container is theater, not learning. Everything architectural (rack awareness, multi-node replication, production YARN) is standard documented knowledge; every command, block listing, and job output below actually ran, and anything that didn't is marked [NOT RUN].

Why files, not a database

The OLTP/OLAP split from post 1 drives the whole design. A transactional database (OLTP) is optimized for many small, consistent writes — it needs indexes, locking, and row-level updates. An analytical workload (OLAP) is optimized for huge sequential scans — it wants to read gigabytes per second off disk with no locks in the way. HDFS picked the second job and committed completely: it is a filesystem, not a database, and it stores files by chopping them into big blocks. The 128–256 MB block size is a rule of thumb, not a law — it exists because spinning-disk-era Hadoop wanted each block to amortize the cost of a seek, and on modern SSDs/cloud storage the sweet spot shifts — but the principle holds: blocks are the first partitioning scheme in this track. The unit of storage, the unit of replication, and (roughly) the unit of parallelism are all the same thing. When post 3 talks about Spark partitions, this is where the idea was born.

HDFS architecture: one brain, many hands

HDFS has two kinds of daemons and a strict division of labor between them:

  • The NameNode (the brain). One per cluster (plus a standby in real deployments). It holds all the metadata: which files exist, which blocks each file is chopped into, and which DataNodes hold copies of each block. It holds no data itself — just the map. This makes it the single point of coordination, which is why real clusters protect it with high availability and why its memory (roughly a few hundred bytes per block) is the practical limit on how many files a cluster holds.
  • The DataNodes (the hands). One per machine, usually dozens to thousands. Each holds actual block data on its local disks and reports a block inventory — a block report — to the NameNode on a heartbeat. When a DataNode stops heartbeating, the NameNode notices and re-replicates its blocks elsewhere. DataNodes never talk to each other about metadata; they take orders from the NameNode and serve block reads to clients directly.
HDFS: metadata lives in one place, data lives everywhere NameNode file → [block A1, A2, A3] A1 → DataNode 1, 2, 3 DataNode 1 block A1 block B2 DataNode 2 block A2 block B1 DataNode 3 block A3 block B3 Client asks the NameNode where blocks are, then reads blocks directly from DataNodes.

A read works in two hops: the client asks the NameNode "where are the blocks of /data/orders.parquet?", gets back a list of DataNodes, and reads the bytes straight from the nearest one. The NameNode is never in the data path — which is the only reason one metadata server can front petabytes.

Blocks and replication

When you write a file, the client chops it into blocks (default 128 MB in Hadoop 3.x — configurable, and many sites run 256 MB) and pipelines each block to its replica DataNodes. The classic rule of thumb is replication factor 3: three copies, placed on two racks. That placement is rack awareness: the NameNode knows the rack topology and tries to keep one copy on the local rack for fast reads and one on a remote rack so a whole rack failure doesn't lose data. Two purposes, one mechanism — fault tolerance and locality. In this post's lab there's a single DataNode, so the honest replication factor is 1 — I'll say so in the lab, because a lab that claims 3x replication on one disk is lying.

Write-once-read-many

HDFS files are write-once: you create a file, stream bytes in, and close it. There's no random write, no in-place update — appending was added later and is still awkward. This is a deliberate trade, not a missing feature: by refusing updates, HDFS avoids all locking and cache-coherency machinery, which is what lets it scale to thousands of nodes. The cost: if your workload needs to update single records (a user profile, an order status), HDFS is the wrong tool — that's the OLTP job, and it belongs in a database. Data lakes inherit this property: Parquet files on S3 are immutable too, and "updating" means writing a new file (which is exactly what Delta Lake's transaction log manages in post 5).

YARN: the landlord (conceptual)

HDFS stores; something has to schedule. That's YARN. The mental model is a landlord, not an engine: YARN doesn't know what a MapReduce job or a Spark query is. It knows containers — a slice of CPU and RAM on a specific machine — and it hands them out to frameworks that ask.

Three roles: the ResourceManager (one per cluster) tracks all containers and decides who gets what; a NodeManager on each machine launches and monitors the containers on that machine; and each job runs an ApplicationMaster — a per-job negotiator that asks the ResourceManager for containers, tells the NodeManagers what to run in them, and restarts failed tasks. MapReduce, Spark, and Flink all learned to run on YARN as applications instead of each owning a cluster. Kubernetes now plays the same landlord role in many shops, but the idea is identical: separate "who gets machines" from "what runs on them."

Honest scope note: this post does not start a YARN cluster — standing up a ResourceManager and NodeManagers to watch a single word-count container would demonstrate nothing the concept section doesn't explain. The resource-management layer reappears in the capstone (post 10), where the full platform is wired together.

The MapReduce mental model: map → shuffle → reduce

Strip away the daemons and MapReduce is one idea: express the job as two functions and let the framework do the distribution. The map function turns each input record into key/value pairs; the framework shuffles — routes all values for the same key to the same machine, sorted — and the reduce function turns each key plus its values into the answer.

The shuffle is the whole game, and it's worth saying plainly: it's an all-to-all data movement where every mapper's output is partitioned by key, sorted, and shipped across the network to the reducer responsible for that key. Everything expensive about distributed computing lives in that sentence — network I/O, disk spills when it doesn't fit in memory, and the hot key problem: if one key has vastly more values than the others (say, counting events by country and one country is 80% of traffic), its reducer does 80% of the work while the rest idle. Data skew isn't a corner case; in real datasets it's the common case, and the shuffle is where you feel it. Remember this paragraph when post 3 explains Spark's shuffle — same idea, same skew, better engine.

Map → shuffle → reduce: word count MAP hello → (hello,1) world → (world,1) hello → (hello,1) SHUFFLE (sort + ship by key) hello → [1, 1] world → [1] all values for one key, one machine, sorted REDUCE (hello, 2) (world, 1) The framework owns everything between the arrows: partitioning, sorting, network transfer, retries. Your code is the two functions; the shuffle is the framework's — and often your job's — most expensive step. Principle: bring the code to the data. Moving terabytes to the computation is the expensive direction. MapReduce moves the (small) functions to the (big) blocks — and the shuffle is the one step where data moves anyway. Minimize it.

Principle: bring the code to the data. This is the memorable idea of the whole Hadoop era, and it survives into Spark: moving terabytes of data to the computation is the expensive direction, so the framework ships your (small) map function to the DataNode holding the (big) block. The shuffle is the one phase where data moves anyway — every shuffle byte crosses the network — so the performance art of distributed computing is, to a first approximation, minimizing the shuffle. Combiners, partitioners, and broadcast joins are all just shuffle-avoidance techniques with different names.

Why this matters for Spark: the shuffle never left

Post 1 said Spark won by avoiding MapReduce's per-stage disk materialization — pipelined execution, cached reuse, spilling only when memory runs out. That's all true, and the engine underneath is genuinely better. But here's what Spark kept: the shuffle. When a Spark job does groupByKey, reduceByKey, or a join, it performs the same all-to-all exchange MapReduce did — partition by key, sort, ship across the network. Spark just does it inside a smarter runtime, with in-memory buffering and a DAG that can pipeline the map side.

That inheritance is why this post's vocabulary carries forward intact:

  • Partitions are blocks, generalized. HDFS splits files into blocks; Spark splits RDDs and DataFrames into partitions. The unit of parallelism is still "one task per chunk of data," and the number of partitions is still the first knob you reach for when a job is slow or skewed.
  • Hot keys are still the default failure mode. One key with 80% of the values still stalls one reducer — Spark calls them tasks, not reducers, but the straggler looks identical on the UI: 199 tasks done in seconds, one task grinding for an hour. The mitigations (salting keys, two-stage aggregation, broadcast joins for small sides) are shuffle-avoidance techniques wearing new clothes.
  • repartition is a shuffle; coalesce usually isn't. This is the single most practical sentence in Spark tuning, and it only makes sense if you know what a shuffle costs. A wide transformation (anything that moves data between partitions — joins, group-bys, repartition) pays the shuffle tax; a narrow one (map, filter) doesn't. The map → shuffle → reduce diagram above is the mental model behind every Spark stage boundary you'll read in an EXPLAIN plan.

So when you write your first Spark job in post 3 and the UI shows a stage with 200 tasks and suspiciously uneven durations, you won't be debugging Spark — you'll be debugging the shuffle, an idea from 2004 that never went away.

Lab: HDFS, for real — blocks you can observe

Concepts are cheap; blocks are real. This lab runs Hadoop 3.3.6 in pseudo-distributed mode: a genuine NameNode and DataNode on one machine, talking real HDFS RPC (the daemons were started directly with hdfs --daemon start — this lab box has no SSH server, so start-dfs.sh can't do its usual ssh hop). Replication factor is 1: one DataNode, one copy, stated honestly instead of pretending at 3x. The NameNode's usual port 9000 was already taken on this box, so the cluster runs on 19000 — ports are configuration, not architecture.

$ export JAVA_HOME=~/workspace/tooling/jdk17   # Hadoop 3.3.x on JDK 17: works (the promised warning never appeared)
$ hdfs namenode -format
[...startup log elided...]
Storage directory /home/hatch/workspace/dataeng-post02-lab/hadoop-data/nn has been successfully formatted.

$ hdfs --daemon start namenode && hdfs --daemon start datanode

$ hdfs dfs -mkdir -p /user/root/input
$ hdfs dfs -put input.txt /user/root/input/
$ hdfs dfs -ls /user/root/input/
Found 1 items
-rw-r--r--   1 root supergroup         98 2026-10-07 04:10 /user/root/input/input.txt

The file is in. Now look at what HDFS actually did with 98 bytes — fsck with -files -blocks shows the block the NameNode assigned:

$ hdfs fsck /user/root/input/input.txt -files -blocks
FSCK started by root (auth:SIMPLE) from /127.0.0.1 for path /user/root/input/input.txt at Wed Oct 07 04:11:14 UTC 2026

/user/root/input/input.txt 98 bytes, replicated: replication=1, 1 block(s):  OK
0. BP-1104580282-127.0.0.1-1791346184204:blk_1073741825_1001 len=98 Live_repl=1


Status: HEALTHY
 Number of data-nodes:	1
 Number of racks:		1
[...zero-count summary lines elided: 0 corrupt, 0 missing, 0 under-replicated...]

Read it like a storage engineer: one file, one block (blk_1073741825_1001), 98 bytes, replication 1, status HEALTHY. The BP-… prefix is the block pool ID — in a federated cluster, multiple NameNodes share DataNodes, and the pool ID says which namespace owns the block. And dfsadmin -report shows the cluster the way an operator sees it:

$ hdfs dfsadmin -report
Configured Capacity: 107355308032 (99.98 GB)
[...block-health summary elided...]
-------------------------------------------------
Live datanodes (1):

Name: 127.0.0.1:9866 (localhost)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 107355308032 (99.98 GB)
DFS Used: 4205 (4.11 KB)
Non DFS Used: 18368528275 (17.11 GB)
DFS Remaining: 88113459200 (82.06 GB)
DFS Used%: 0.00%

One live DataNode, decommission status Normal, 82 GB free. This is the same report you'd read on a thousand-node cluster — the numbers just get bigger. Notice what isn't here: no mention of file contents anywhere. The NameNode knows blocks and locations; the bytes live on the DataNode. That separation is the architecture.

Lab: a real MapReduce job, end to end

Now the engine. The program below is the canonical word count — Mapper, Reducer, Driver — written against the org.apache.hadoop.mapreduce API. Three phases, exactly the ones from the diagram: the TokenizerMapper emits (word, 1), the framework shuffles, the IntSumReducer sums. The setCombinerClass line is worth noticing: the combiner is a mini-reducer that runs on the map side before the shuffle — the first shuffle-avoidance technique you'll meet, and you'll see it work in the counters.

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    // Phase 1: the Mapper. For each input line, emit (word, 1) for every word.
    public static class TokenizerMapper
            extends Mapper<Object, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);
        private final Text word = new Text();

        public void map(Object key, Text value, Context context)
                throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    // Phase 3: the Reducer. For each word, sum the 1s the shuffle delivered.
    public static class IntSumReducer
            extends Reducer<Text, IntWritable, Text, IntWritable> {

        private final IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    // Phase 0: the Driver. Wires the job together and submits it.
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Compile against Hadoop's jars, pack a jar, and run. The job reads its input from HDFS (the file we just put there) and writes back to HDFS — but it executes in local/standalone mode: no YARN, no cluster, the whole thing in one JVM. The job ID gives it away — job_local… is the LocalJobRunner's signature:

$ CP=$(hadoop classpath)
$ javac -classpath "$CP" -d classes src/WordCount.java && jar cf wc.jar -C classes .

$ hadoop jar wc.jar WordCount /user/root/input /user/root/output
[...task-attempt logs elided...]
2026-10-07 04:12:04,624 INFO mapreduce.Job: Running job: job_local1940196645_0001
2026-10-07 04:12:05,639 INFO mapreduce.Job:  map 100% reduce 0%
2026-10-07 04:12:06,640 INFO mapreduce.Job:  map 100% reduce 100%
2026-10-07 04:12:06,641 INFO mapreduce.Job: Job job_local1940196645_0001 completed successfully

The job itself took about two seconds (04:12:04 → 04:12:06 in the log) — and the counters tell the shuffle's story in numbers:

Map input records=3            # three lines in
Map output records=14          # fourteen (word, 1) pairs out of the mapper
Combine input records=14
Combine output records=8       # the combiner pre-summed: 14 pairs -> 8 partial sums
Reduce input groups=8          # the shuffle delivered 8 distinct keys...
Reduce input records=8         # ...one pre-summed value each
Shuffled Maps =1               # the all-to-all exchange, right there in the counters
Merged Map outputs=1

That's the whole pipeline in six numbers: 3 lines → 14 pairs → combiner shrinks to 8 → shuffle moves 8 → reducer sums 8 groups. The output lands back on HDFS:

$ hdfs dfs -cat /user/root/output/part-r-00000
blocks	1
hadoop	2
hdfs	1
hello	4
mapreduce	2
reduce	1
shuffle	2
sort	1

Check it against the input (hello mapreduce hello hadoop / mapreduce shuffle sort reduce / hadoop hello hdfs blocks hello shuffle): hello×4, mapreduce×2, hadoop×2, shuffle×2 — all correct. And notice the output is sorted by key. Nobody asked for sorting; the shuffle sorts because the reducer needs all values for a key grouped together, and sorted-merge is how it does it. That free sort is why MapReduce jobs (and Spark shuffles) are expensive even when you don't care about order — you pay for the sort whether you wanted it or not.

What this lab didn't do. The job ran in a single JVM against a one-node HDFS — there's no YARN here, no containers, no multi-machine shuffle. The concepts above (containers, rack-aware replication, hot keys) describe what happens when this same code meets a real cluster; nothing about this run contradicts them, but this run doesn't demonstrate them either. That's what post 3's Spark labs are for.

Cheat sheet

  • HDFS splits files into blocks (128 MB default in Hadoop 3.x is the common rule of thumb — configurable, and the right size depends on your storage), replicates them (3x across racks is the classic default; this lab honestly ran 1x on one node), and never updates in place: write-once-read-many.
  • NameNode holds metadata (file → blocks → DataNodes); DataNodes hold bytes and heartbeat. The NameNode is never in the data path.
  • Rack awareness serves two masters: fault tolerance (survive a rack failure) and locality (read from the near copy). Blocks are this track's first partitioning scheme.
  • YARN is the landlord: ResourceManager hands out containers, NodeManagers run them, the ApplicationMaster negotiates per job. Separate "who gets machines" from "what runs on them" — Kubernetes is the same idea.
  • MapReduce = two functions plus the framework's shuffle: map → (partition, sort, ship by key) → reduce. The shuffle is an all-to-all exchange and the most expensive step.
  • Bring the code to the data. Ship the small function to the big block; minimize the one phase where data moves anyway.
  • The combiner is shuffle-avoidance: pre-sum on the map side (this lab: 14 records → 8 before the shuffle).
  • The shuffle sorts whether you asked or not — you pay for the sort even when order doesn't matter.
  • Spark kept the shuffle and fixed the engine: same hot-key skew, same stage boundaries at every wide transformation. repartition shuffles; coalesce usually doesn't.
  • HDFS is OLAP-side storage (huge sequential scans); it refuses the OLTP job (single-record updates) by design. Data lakes inherit the immutability — Parquet on object storage works the same way.

What's next

You've touched the foundation: real blocks on a real (tiny) HDFS, a real MapReduce job with the shuffle visible in its counters. Everything above this layer is, in some sense, commentary on the shuffle — and the sharpest commentary is Spark, which kept the idea and rebuilt the engine around it.

The next post, Spark Core: RDDs, DataFrames & the Java API, takes the mental model you built here — partitions as blocks, the shuffle as the expensive step, hot keys as the failure mode — and puts it to work in a modern engine: lineage, transformations vs actions, and the Java-specific sharp edges (lambdas, Encoders, serialization) that the Scala-first documentation glosses over.

Field check before you move on: (1) Re-run the word count on your own input file — then remove the setCombinerClass line, re-run, and compare the Combine output records counter to the Reduce input records counter. What moved across the shuffle in each case? (2) Run hdfs fsck /user/root/output/part-r-00000 -files -blocks -locations on the lab cluster and explain, in one sentence, what -locations adds. (3) Take one "wide" Spark transformation you know (groupByKey, a join, repartition) and describe which MapReduce phase it corresponds to. (4) Your team proposes raising the HDFS block size from 128 MB to 1 GB for a 10 TB dataset of small Parquet files: argue for and against, in two sentences each — then check what actually determines the number of map tasks.

Continue: Java Learning Roadmap 2026

Comments

Popular posts from this blog

JSP Servlet Interview Questions For Freshers Series 1

What are the core classes and interfaces of hibernate 4

Java Banking Finance Services and Insurance (BFSI) domain interview questions