====== hdfs dfs ======

  * [[Hadoop]]
  * [[HDFS]]
  * [[hdfs dfsadmin]]

  * [[https://developer.yahoo.com/hadoop/tutorial/module2.html#commandref|HDFS Command Reference]]

** Permissions: ** Note that files listed by ''hdfs dfs'' are owned by the ''hdfs'' user on the Linux system. That means that files on HDFS must be written / removed / modified by user ''hdfs''. Using ''root'' will not work.

** Removing files ** will happen in 6 hours by default:

<code>
15/09/08 17:42:20 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
Moved: 'hdfs://node1:8020/user/root/test1' to trash at: hdfs://node1:8020/user/root/.Trash/Current
</code>

** List a directory's contents **

If no argument is given, and the ''/user/username'' directory exists, it will display the user's directory.

<code>
hdfs dfs -ls
</code>

<code>
hdfs dfs -ls /
</code>

<code>
Found 8 items
drwxrwxrwx   - yarn   hadoop          0 2015-09-08 13:56 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:58 /apps
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:53 /hdp
drwxr-xr-x   - mapred hdfs            0 2015-09-08 13:52 /mapred
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:52 /mr-history
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 13:53 /system
drwxrwxrwx   - hdfs   hdfs            0 2015-09-08 13:57 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2015-09-08 17:29 /user
</code>

** Create a directory **

As user ''hdfs'':

<code>
hdfs dfs -mkdir /foodir
</code>

** Remove a directory **

As user ''hdfs'':

<code>
hdfs dfs -rm -R /foodir
</code>

** View the contents of a file **

<code>
hdfs dfs -cat /foodir/myfile.txt
</code>

** Recursively view the contents of a directory **

<code>
hdfs dfs -ls -R
</code>

** Put a file onto the filesystem **

<code>
hdfs dfs -put sample.txt
</code>

** Copy a file on the HDFS filesystem to another HDFS filename **

<code>
hdfs dfs -cp sample.txt sample-copy.txt
</code>

** Copy a file from HDFS to local filesystem **

<code>
hdfs dfs -get sample.txt /tmp
</code>

** cat a file **

<code>
hdfs dfs -cat sample.txt
</code>

** tail the last 20 lines of a file **

<code>
hdfs dfs -tail sample.txt
</code>

** Add a file with a specific number of replication **

In this example, set replication to 2 instead of default of 3.

<code>
hdfs dfs -D dfs.replication=2 -put sample.txt sample-rep2.txt
</code>

And verify the replication value:

<code>
hdfs fsck sample-rep2.txt
</code>

<code>
Average block replication:	2.0
</code>

** Immediately empty trash **

<code>
hdfs dfs -expunge
</code>

** Disable safe mode **

HDFS will enable safe mode if the set number of possible nodes to fail have failed. See ''dfs.namenode.safemode.threshold-pct'' in ''hdfs-site.xml''.

<code>
hdfs dfsadmin -safemode leave
</code>