Hadoop Storage Elasticity using LVM

Providing elastic storage to hadoop data node using logical volume

Hadoop HDFS cluster stores big data and here we need big storage. In case of adding new volumes to increase storage size, we may lose current data. Here it is hard to manage the storage also.

To achieve this requirement without data loss we can use LVM (Logical Volume Manager) to provide storage in such a way that we can increase or decrease the volume size without losing data so easily, by adding volumes as per our need.

Current Storage of HDFS cluster is around 10GB provided by one data node

In order to give more storage to this node attach external volume. Here I attached two external volumes (30GB & 20GB)

To create logical storage from these volumes first install lvm2 then create physical volumes of attached volumes.

Here two Physical Volumes (PV) are created of size 30GB & 20GB

Create Volume Group (VG) of that two PVs

VG is created of around size 50GB now we can give storage to data node from this VG as per our need. For that create Logical Volume of required size from that VG.

Here 30 GB PV created but to use it we have to format it. Then we can mount it to the data node storage directory.

PV mounted on Data Node.

HDFS cluster size is provided by that LV of size around 30GB. Now let’s increases the size of this HDFS storage.

Here I am increasing +11GB that will in total become 41 GB

Now we have formatted 30GB block and added 11GB block is remain unformatted and we can’t allocate that block for storage. So to format only remaining block we have to resize file system online that will create whole 41GB volume without losing previous data.

Now dynamically LV of size around 41GB is providing storage to HDFS!