Docsity
Docsity

Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity


Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan


Guidelines and tips
Guidelines and tips

Hadoop Commands Cheat Sheet, Cheat Sheet of Advanced Computer Programming

generic, hdfs and yarn commands in hadoop

Typology: Cheat Sheet

2020/2021

Uploaded on 04/27/2021

ryangosling
ryangosling 🇺🇸

4.8

(24)

250 documents

1 / 1

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
Hadoop commands cheat sheet
Generic
hadoop fs -ls <path> list files in the path of the file system
hadoop fs -chmod <arg> <file-or-dir> alters the permissions of a file where <arg> is the binary argument e.g. 777
hadoop fs -chown <owner>:<group> <file-or-dir> change the owner of a file
hadoop fs -mkdir <path> make a directory on the file system
hadoop fs -put <local-origin> <destination> copy a file from the local storage onto file system
hadoop fs -get <origin> <local-destination> copy a file to the local storage from the file system
hadoop fs -copyFromLocal <local-origin> <destination> similar to the put command but the source is restricted
to a local file reference
hadoop fs -copyToLocal <origin> <local-destination> similar to the get command but the destination is restricted
to a local file reference
hadoop fs -touchz create an empty file on the file system
hadoop fs -cat <file> copy files to stdout
Yarn commands
yarn node -list list nodes in the yarn cluster
yarn node -status <node id> status of a node (memory used, free, number of containers, etc) for <node id> (first
column from command above)
yarn application -list list of Yarn applications and their state
yarn logs -applicationId <appid> dump the logs for a particular application
Configuration commands
hdfs getconf return various configuration settings in effect
hdfs getconf -namenodes namenodes in the cluster
hdfs getconf -confkey <a.value> return the value of a particular setting (e.g. dfs.replication)
HDFS commands
hdfs dfsadmin -safemode get find out if you’re in safemode
hdfs dfsadmin -report find out how much disk space us used, free, under-replicated, etc.
Kognitio specific
kodoop sql <cluster> run an SQL session against the running server. <user> defaults to sys.
kodoop server <cluster> start start the server, incorporating any new config file changes. Memory images will
persist. If the server is currently running, this command restarts it.
kodoop server <cluster> stop stop the server. Memory images will persist so long as the cluster remains active.
kodoop server <cluster> status show the status of the server.
kodoop cluster <cluster> initialize initialize the server. Erase existing data/metadata.
kodoop cluster <cluster> stop stop the cluster’s YARN application. This will shut down everything except the edge
nodes. Memory images will be lost but internal data will persist in HDFS.
kodoop cluster <cluster> restart stop and then start again.
kodoop mgr <cluster> shell run a sub-shell configured to allow users to directly run the management commands
from the WX2 software
kodoop help find out about Kognitio on Hadoop commands
kodoop testenv check Kognitio on Hadoop environment is configured correctly
kodoop list_clusters show the currently configured Kognitio on Hadoop clusters
kodoop server <cluster> diagnose check for problems with a server
kodoop server <cluster> [auto|manual] turn automatic management on or off (defaults to on)
kodoop server <cluster> viconf change server config settings
kodoop incidents <cluster> list list of incidents (container failures, etc) the cluster has recovered from
kodoop gateway <cluster> restart restart a hung gateway (was an issue for older versions)
kodoop sql <cluster> quick SQL connection to the cluster as the sys user
This cheat sheet outlines some of the main Hadoop commands that we’ve found useful, as well
as Kognitio specific commands when used on Hadoop.

Partial preview of the text

Download Hadoop Commands Cheat Sheet and more Cheat Sheet Advanced Computer Programming in PDF only on Docsity!

Hadoop commands cheat sheet

Generic

  • hadoop fs -ls list files in the path of the file system
  • hadoop fs -chmod <file-or-dir> alters the permissions of a file where is the binary argument e.g. 777
  • hadoop fs -chown : <file-or-dir> change the owner of a file
  • hadoop fs -mkdir make a directory on the file system
  • hadoop fs -put <local-origin> copy a file from the local storage onto file system
  • hadoop fs -get <local-destination> copy a file to the local storage from the file system
  • hadoop fs -copyFromLocal <local-origin> similar to the put command but the source is restricted to a local file reference
  • hadoop fs -copyToLocal <local-destination> similar to the get command but the destination is restricted to a local file reference
  • hadoop fs -touchz create an empty file on the file system
  • hadoop fs -cat copy files to stdout Yarn commands
  • yarn node -list list nodes in the yarn cluster
  • yarn node -status status of a node (memory used, free, number of containers, etc) for (first column from command above)
  • yarn application -list list of Yarn applications and their state
  • yarn logs -applicationId dump the logs for a particular application Configuration commands
  • hdfs getconf return various configuration settings in effect
  • hdfs getconf -namenodes namenodes in the cluster
  • hdfs getconf -confkey <a.value> return the value of a particular setting (e.g. dfs.replication) HDFS commands
  • hdfs dfsadmin -safemode get find out if you’re in safemode
  • hdfs dfsadmin -report find out how much disk space us used, free, under-replicated, etc. Kognitio specific
  • kodoop sql run an SQL session against the running server. defaults to sys.
  • kodoop server start start the server, incorporating any new config file changes. Memory images will persist. If the server is currently running, this command restarts it.
  • kodoop server stop stop the server. Memory images will persist so long as the cluster remains active.
  • kodoop server status show the status of the server.
  • kodoop cluster initialize initialize the server. Erase existing data/metadata.
  • kodoop cluster stop stop the cluster’s YARN application. This will shut down everything except the edge nodes. Memory images will be lost but internal data will persist in HDFS.
  • kodoop cluster restart stop and then start again.
  • kodoop mgr shell run a sub-shell configured to allow users to directly run the management commands from the WX2 software
  • kodoop help find out about Kognitio on Hadoop commands
  • kodoop testenv check Kognitio on Hadoop environment is configured correctly
  • kodoop list_clusters show the currently configured Kognitio on Hadoop clusters
  • kodoop server diagnose check for problems with a server
  • kodoop server [auto|manual] turn automatic management on or off (defaults to on)
  • kodoop server viconf change server config settings
  • kodoop incidents list list of incidents (container failures, etc) the cluster has recovered from
  • kodoop gateway restart restart a hung gateway (was an issue for older versions)
  • kodoop sql quick SQL connection to the cluster as the sys user This cheat sheet outlines some of the main Hadoop commands that we’ve found useful, as well as Kognitio specific commands when used on Hadoop.