hdfs dfs

• Hadoop
• HDFS
• hdfs dfsadmin

• HDFS Command Reference

Permissions: Note that files listed by hdfs dfs are owned by the hdfs user on the Linux system. That means that files on HDFS must be written / removed / modified by user hdfs. Using root will not work.

Removing files will happen in 6 hours by default:

15/09/08 17:42:20 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
Moved: 'hdfs://node1:8020/user/root/test1' to trash at: hdfs://node1:8020/user/root/.Trash/Current

List a directory's contents

If no argument is given, and the /user/username directory exists, it will display the user's directory.

hdfs dfs -ls

hdfs dfs -ls /

Found 8 items
drwxrwxrwx   - yarn   hadoop          0 2015-09-08 13:56 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:58 /apps
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:53 /hdp
drwxr-xr-x   - mapred hdfs            0 2015-09-08 13:52 /mapred
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:52 /mr-history
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:53 /system
drwxrwxrwx   - hdfs   hdfs            0 2015-09-08 13:57 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 17:29 /user

Create a directory

As user hdfs:

hdfs dfs -mkdir /foodir

Remove a directory

As user hdfs:

hdfs dfs -rm -R /foodir

View the contents of a file

hdfs dfs -cat /foodir/myfile.txt

Recursively view the contents of a directory

hdfs dfs -ls -R

Put a file onto the filesystem

hdfs dfs -put sample.txt

Copy a file on the HDFS filesystem to another HDFS filename

hdfs dfs -cp sample.txt sample-copy.txt

Copy a file from HDFS to local filesystem

hdfs dfs -get sample.txt /tmp

cat a file

hdfs dfs -cat sample.txt

tail the last 20 lines of a file

hdfs dfs -tail sample.txt

Add a file with a specific number of replication

In this example, set replication to 2 instead of default of 3.

hdfs dfs -D dfs.replication=2 -put sample.txt sample-rep2.txt

And verify the replication value:

hdfs fsck sample-rep2.txt

Average block replication:	2.0

Immediately empty trash

hdfs dfs -expunge

Disable safe mode

HDFS will enable safe mode if the set number of possible nodes to fail have failed. See dfs.namenode.safemode.threshold-pct in hdfs-site.xml.

hdfs dfsadmin -safemode leave