Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
A laboratory manual for the MCA34: Big Data Analytics course at RIT's Department of Master of Computer Applications. It covers various HDFS shell commands, including help, usage, ls, mkdir, copyFromLocal, put, df, expunge, mv, rm, tail, test, count, and setrep. These commands are essential for managing files in HDFS and are similar to UNIX file system commands.
Typology: Summaries
1 / 17
This page cannot be seen from the preview
Don't miss anything!
(Autonomous Institute, Affiliated to VTU)
rd
1) help HDFS Shell Command Syntax of help hdfs Command $ hadoop fs – help Help hdfs shell command helps hadoop developers with all the available hadoop commands and how to use them. Variations of the Hadoop fs Help Command $ hadoop fs – help ls Using the help command with a specific command lists the usage information along with the options to use the command. 2) Usage HDFS Shell Command $ hadoop fs – usage ls Usage command gives all the options that can be used with a particular hdfs command. 3) ls HDFS Shell Command Syntax for ls Hadoop Command - $ hadoop fs – ls / This command will list all the available files and subdirectories under default directory Variations of Hadoop ls Shell Command $ hadoop fs – ls / Returns all the available files and subdirectories present under the root directory. 4) mkdir- Used to create a new directory in HDFS at a given location. $ hadoop fs – mkdir /USN/D The above command will create a new directory D1 under your USN Create another directory D2 under the USN
10) Expunge This HDFS command empties the trash by deleting all the files and directories. Example - $ hadoop fs – expunge 11) Cat This is similar to the cat command in Unix and displays the contents of a file. Example - $ hadoop fs – cat /USN/D1/Sample1.txt 12) cp Copy files from one HDFS location to another HDFS location. Example – $ hadoop fs – cp /USN/D1/sample.txt /USN/D 13) mv Move files from one HDFS location to another HDFS location. Example – $ hadoop fs – mv /USN/D1/Sample1.txt /USN/D 14) rm Removes the file or directory from the mentioned HDFS location. Example – $ hadoop fs – rm - r /USN/D Deletes or removes the directory and its content from HDFS location in a recursive manner. 15) tail This hadoop command will show the last kilobyte of the file to stdout. Example – $ hadoop fs - tail /USN/D1/sample1.txt Example – $ hadoop fs - tail – f - tail /USN/D1/sample1.txt Using the tail commands with - f option, shows the last kilobyte of the file from end in a page wise format.
16) copyToLocal Copies the files to the local filesystem. This is similar to hadoop fs - get command but in this case the destination location msut be a local file reference Example - $ hadoop fs – copyToLocal /USN/D1/Sample1.txt /home/sk/USN/ 17) get Downloads or Copies the files to the local filesystem. Example - $ hadoop fs – get /USN/D1/Sample1.txt /home/sk/USN/ 18) touchz Used to create an emplty file at the specified location. Example - $ hadoop fs – touchz /USN/D1/Sample2.txt It will create a new empty file Sample2.txt in (hdfs path) 19) test Used for file test operations. Options: -
options:
4. get: This command is used to copy files from HDFS file system to the local file system, just the opposite to put command. Syntax: 1 $ hadoop fs - get [-f] [-p] Example: 1 $ hadoop fs - get /user/data/sample.txt workspace/ 5. cat: This command is similar to the UNIX cat command and is used for displaying the contents of a file on the console. Example: 1 $ hadoop fs - cat /user/data/sampletext.txt 6. cp: This command is similar to the UNIX cp command, and it is used for copying files from one directory to another directory within the HDFS file system. Example: $ hadoop fs - cp /user/data/sample1.txt /user/hadoop $ hadoop fs - cp /user/data/sample2.txt /user/test/in 7. mv: This command is similar to the UNIX mv command, and it is used for moving a file from one directory to another directory within the HDFS file system. Example: 1 $ hadoop fs - mv /user/hadoop/sample1.txt /user/text/
Syntax: 1 $ hadoop fs - setrep [-R] [-w] Example: 1 $ hadoop fs - setrep - R /user/hadoop/
11. touchz: This command can be used to create a file of zero bytes size in HDFS filesystem. Example: 1 $ hadoop fs - touchz URI 12. test: This command is used to test an HDFS file’s existence of zero length of the file or whether if it is a directory or not. options:
13. expunge: This command is used to empty the trash available in an HDFS system. Syntax: 1 $ hadoop fs – expunge Example: user@ubuntu1:~$ hadoop fs – expunge 17/10/15 10:15:22 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. 14. appendToFile: This command appends the contents of all the given local files to the provided destination file on the HDFS file system. The destination file will be created if it is not existing earlier. Syntax: 1 $ hadoop fs - appendToFile Example: user@ubuntu1:~$ hadoop fs - appendToFile derby.log data.tsv /in/appendfile user@ubuntu1:~$ hadoop fs - cat /in/appendfile 15. tail: This command is used to show the last 1KB of the file. option:
19. du: This command is used to show the amount of space in bytes that have been used by the files that match the specified file pattern. Even without the – s option, this only shows the size summaries one level deep in the directory. Options:
Option:
