JavaMakeUse: Learn Java, Python, AI, Cloud & More

Joins are very important aspect in any databases and, In Hadoop MapReduce joins are also available to join the multiple datasets. Joins are always involved while preparing data for presentation from multiple tables/datasets like fetching users activities on social media, where user and user_activity two different datasets needs to be incorporate to extract the informations. We have already seen some most popular features of Hadoop MapReduce like Custom Data Types , Partitioner in MapReduce etc.

We have already seen an example of Combiner in MapReduce programming and Custom Partitioner . In this tutorial, I am going to show you an example of Map side join in Hadoop MapReduce. If you want to dig more into the deep of MapReduce, and how it works, than you may like this article on how map reduce works . MapReduce process the big data sets, and processing large data sets most of the time required joining between datasets based on common key like we almost always do while playing with any RDBMS database based on primary/foreign key concept. Joins in Hadoop MapReduce Hadoop MapReduce supports two types of joins- Map Side Join Reduce Side Join( Reduce side join example ) In this tutorial, I am going to explain you the usage of Map side join. Map side Join You can use Map side join using two different ways based on your datasets, and those depends on below conditions - Both datasets are must be divided into the same number of partitions, and must be already sorted by ...

Big data technologies are highly demanding technology in the market. And the core or you can say basic of Hadoop Map Reduce is one of the key to understand the other big data technologies, no one can ignore the basic one. To get the optimal use of Map Reduce framework, you must to have to understand the internal of Map Reduce and how it works. In my previous tutorial, I have shared a Hello world program( Word count tutorial ) in Hadoop map reduce programming. And in this tutorial, I am going to explain the internals of Map Reduce and how it works. How Map Reduce Works? MapReduce is a framework for writing application to process the big data in parallel on large clusters of commodity hardware in reliable and fault tolerant manner. MapReduce executes on top of HDFS(Hadoop Distributed File System) in two different phases called map phase and reduce phase. Most importantly map reduce programs are parallel processing, works on key value pattern, map phase works on key-value and prod...

In this tutorial, I am going to show you an example of custom partitioner in Hadoop map reduce. In my previous tutorial, you have already seen an example of Combiner in Hadoop map reduce programming and the benefits of having combiner in map reduce framework. Combiner( how combiner works in map reduce ) is one of the very very important feature of Hadoop, which really helps a lot to reduce the network traffic between map and reduce phase. What is Partitioner? The partitioner controls the partitioning of the keys of intermediate map outputs. The partitioner, which decides the output of the mapper goes to which reducer, generally it uses the default hash partitioner for the same. When there are multiple reducer, the map task partition their output, each creating one partitioner for each reduce task. Although it uses the default hash partitioner, you can also control it by using your own custom partitioner. The total number of partitioner is equals to the the total number of reduce t...

The Combiner is one of the powerful feature of Hadoop map reduce programming and used as an optimization technique. In my previous post, we have seen an example of custom data type in Hadoop at here using WritableComparable interface. In this tutorial, I am going to show you an example of map reduce programming using Combiner, although, we have already seen an example of map reduce programming in our wordcount tutorial .

In my previous post, we have already seen a step by step guide to set up an environment for Apach Hive here . And in this post going to show you a step by step guide to change your default meta store from Derby to Mysql database. What is Meta store in Hive? The Hive metastore, stores the meta data of Hive tables and partition created in Hive, in a relational database. By default relational database used to keep the meta data of Hive table is Derby database. And support only one active session at a time, which is not suitable for production use. You can't use more than one session of Hive at at time, if you are using the default(Derby) meta store. How to configure MySql Metastore for Hive? Below are the steps given to change your default Derby database to MySql for storing meta data of Hive table, before this you must have Hive installed in your machine, if not you may follow my previous post step-by-step Hive installation guide to install. Step 1. Install MySql databas...