hdfs

• Hadoop
• HDFS Filesystem
• hdfs dfs
• webhdfs

• HDFS User Guide
• HDFS Commands Guide
• HDFS Snapshots

• HDFS Command Reference

• hdfs archive
• hdfs distcp
• hdfs dfs
• hdfs fs
• hdfs fsck
• hdfs fetchdt

• hdfs balancer
• hdfs daemonlog
• hdfs datanode
• hdfs dfsadmin
• hdfs namenode
• hdfs secondarynamenode

User home directory is assumed (fex, /user/beandog) unless a specific path is provided.

For some reason or another, you may need to remove an HDFS node from the cluster. Doing maintenance would be an example. The filesystem needs to be checked to see if there's any issues, then the HDFS NameNode needs to know about its removal.

Before doing anything, get a report about the node to see what it's status is.

hdfs dfsadmin -report

Name: 192.168.12.24:50010 (hadoop-node4.lan)
Hostname: hadoop-node4.lan
Decommission Status : Normal
Configured Capacity: 5487524069376 (4.99 TB)
DFS Used: 9204400128 (8.57 GB)
Non DFS Used: 279243939840 (260.07 GB)
DFS Remaining: 5199075729408 (4.73 TB)
DFS Used%: 0.17%
DFS Remaining%: 94.74%

Next, run a filesystem check to see if there's anything out of the ordinary. You can specify any directory or root partition if desired.

hdfs fsck /user/beandog/work-project
hdfs fsck /

If you want, you can a backup of a directory as well. This will take some time, since the data is stored across multiple nodes, and it has to pull parts of it from each one.

hdfs dfs -copyToLocal /user/beandog/bigdata /home/beandog/bigdata.bak

On the HDFS NameNode (the primary server that keeps track of all the metadata), edit the HDFS configuration file to now exclude that node as a place for storage.

In the hdfs-site.xml file, you'll add a new property, with the name being dfs.hosts.exclude. The value for the property is a file somewhere on the filesystem that has a list of each host that will be decommissioned, one per line.

In this case, the location I'm putting the text file is in /etc/hdfs-removed-nodes.

First, the XML file addition:

<property>
  <name>dfs.hosts.exclude</name>
  <value>/etc/hdfs-removed-nodes</value>
</property>

And the contents of hdfs-removed-nodes:

hadoop-node4.lan

Tell the NameNode to refresh the nodes:

hdfs dfsadmin -refreshNamenodes

The HDFS node will be decomissioned, which will take some time. You can view the status either through the web interface, or using hdfs dfsadmin:

hdfs dfsadmin -report

Once the node is completely decommissioned, you can remove it from the slaves file in your Hadoop configuration directory, and restart HDFS:

stop-dfs.sh
start-dfs.sh