Is there a way to add nodes to a running Hadoop cluster?

Updated Answer for Cloudera using CDH 5.8.5 (Hadoop 2.6)-

To add a new node to your cluster, follow these steps on ClouderaManager UI,

  1. Click on your cluster name.
  2. Go to Hosts List.
  3. Once on the hosts page, click 'Add New Hosts to Cluster'.
  4. Enter the IP of your host and Search.
  5. Keep following the instructions and continue to next steps.
  6. Finally assign roles to your new node, for example if it's a data-node,assign only datanode related roles and continue.
  7. Finally your new node is added to your cluster. click Finish.

Here is the documentation for adding a node to Hadoop and for HBase. Looking at the documentation, there is no need to restart the cluster. A node can be added dynamically.


Following steps should help you launch the new node into the running cluster.

1> Update the /etc/hadoop/conf/slaves list with the new node-name
2> Sync the full configuration /etc/hadoop/conf to the new datanode from the Namenode. If the file system isn't shared.  
2>  Restart all the hadoop services on Namenode/Tasktracker and all the services on the new Datanode. 
3>  Verify the new datanode from the browser http://namenode:50070
4>  Run the balancer script to readjust the data between the nodes. 

If you don't want to restart the services on the NN, when you add a new node. I would say add the names ahead to the slaves configuration file. So they report as decommission/dead nodes until they are available. Following the above DataNode only steps. Again this not the best practice.