Difference between HBase and Hadoop/HDFS

Apache Hadoop project includes four key modules

1. Hadoop Common: The common utilities that support the other Hadoop modules.
2. Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
3. Hadoop YARN: A framework for job scheduling and cluster resource management.
4. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

HBase is A scalable, distributed database that supports structured data storage for large tables. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS.

When to use HBase:

1. If your application has a variable schema where each row is slightly different
2. If you find that your data is stored in collections, that is all keyed on the same value
3. If you need random, real time read/write access to your Big Data.
4. If you need key based access to data when storing or retrieving.
5. If you have huge amount of data with existing Hadoop cluster

But HBase has some limitations

1. It can't be used for classic transactional applications or even relational analytics.
2. It is also not a complete substitute for HDFS when doing large batch MapReduce.
3. It doesn’t talk SQL, have an optimizer, support cross record transactions or joins.
4. It can't be used with complicated access patterns (such as joins)

Summary:

Consider HBase when you’re loading data by key, searching data by key (or range), serving data by key, querying data by key or when storing data by row that doesn’t conform well to a schema.

Have a look at Do's and Don't of HBase from cloudera blog.

Hadoop is basically 3 things, a FS (Hadoop Distributed File System), a computation framework (MapReduce) and a management bridge (Yet Another Resource Negotiator). HDFS allows you store huge amounts of data in a distributed (provides faster read/write access) and redundant (provides better availability) manner. And MapReduce allows you to process this huge data in a distributed and parallel manner. But MapReduce is not limited to just HDFS. Being a FS, HDFS lacks the random read/write capability. It is good for sequential data access. And this is where HBase comes into picture. It is a NoSQL database that runs on top your Hadoop cluster and provides you random real-time read/write access to your data.

You can store both structured and unstructured data in Hadoop, and HBase as well. Both of them provide you multiple mechanisms to access the data, like the shell and other APIs. And, HBase stores data as key/value pairs in a columnar fashion while HDFS stores data as flat files. Some of the salient features of both the systems are :

Hadoop

1. Optimized for streaming access of large files.
2. Follows write-once read-many ideology.
3. Doesn't support random read/write.

HBase

1. Stores key/value pairs in columnar fashion (columns are clubbed together as column families).
2. Provides low latency access to small amounts of data from within a large data set.
3. Provides flexible data model.

Hadoop is most suited for offline batch-processing kinda stuff while HBase is used when you have real-time needs.

An analogous comparison would be between MySQL and Ext4.

Hadoop uses distributed file system i.e HDFS for storing bigdata.But there are certain Limitations of HDFS and Inorder to overcome these limitations, NoSQL databases such as HBase,Cassandra and Mongodb came into existence.

Hadoop can perform only batch processing, and data will be accessed only in a sequential manner. That means one has to search the entire dataset even for the simplest of jobs.A huge dataset when processed results in another huge data set, which should also be processed sequentially. At this point, a new solution is needed to access any point of data in a single unit of time (random access).

Like all other FileSystems, HDFS provides us storage, but in a fault tolerant manner with high throughput and lower risk of data loss(because of the replication).But, being a File System , HDFS lacks random read and write access. This is where HBase comes into picture. It’s a distributed, scalable, big data store, modelled after Google’s BigTable. Cassandra is somewhat similar to hbase.