Is there a way to add nodes to a running Hadoop cluster?
Updated Answer for Cloudera using CDH 5.8.5 (Hadoop 2.6)-
To add a new node to your cluster, follow these steps on ClouderaManager UI,
- Click on your cluster name.
- Go to Hosts List.
- Once on the hosts page, click 'Add New Hosts to Cluster'.
- Enter the IP of your host and Search.
- Keep following the instructions and continue to next steps.
- Finally assign roles to your new node, for example if it's a data-node,assign only datanode related roles and continue.
- Finally your new node is added to your cluster. click Finish.
Here is the documentation for adding a node to Hadoop and for HBase. Looking at the documentation, there is no need to restart the cluster. A node can be added dynamically.
Following steps should help you launch the new node into the running cluster.
1> Update the /etc/hadoop/conf/slaves list with the new node-name
2> Sync the full configuration /etc/hadoop/conf to the new datanode from the Namenode. If the file system isn't shared.
2> Restart all the hadoop services on Namenode/Tasktracker and all the services on the new Datanode.
3> Verify the new datanode from the browser http://namenode:50070
4> Run the balancer script to readjust the data between the nodes.
If you don't want to restart the services on the NN, when you add a new node. I would say add the names ahead to the slaves configuration file. So they report as decommission/dead nodes until they are available. Following the above DataNode only steps. Again this not the best practice.