HDFS shell interfaces using the format operation: hadoop namenode -format
Showing a document: hadoop fs -ls / hadoop fs -ls / user
HDFS use:
It provides a shell interface, command line operations
hadoop namenode -format # format
namenode hadoop fs -ls / # print / file directory listing
hadoop fs -mkdir input # Create a directory
input hadoop fs -put hadoop-env.sh input / # upload files
hadoop fs -get input to input at hadoop-env.sh directory / abc.sh hadoop-envcomp.sh # Download the file from the input directory
hadoop fs -cat input / hadoop-env.sh # view the file input / hadoop-env.sh hadoop dfsadmin -report #dfs report
hadoop fs -ls /apps/hive/warehouse/t_log_2016
The principle mapReduce
MapReduce: divide and conquer, a big task into a plurality of smaller sub-tasks (Map), the parallel execution merge results (the reduce)
TaskTracker role 1. Perform Task 2. Task Status Report
JobTracker role 1. Scheduling 2. Assign the task, the task of monitoring the progress of implementation of 3. monitor the status of TaskTracker
map: 1 reduce segmentation of each word statistics in mind: Merge the same key in the same node
1. compile java files compiled -d javac -classpath /opt/hadoop-1.2.1/hadoop-core-1.2.1.jar:/opt/hadoop-1.2.1/lib/commons-cli-1.2.jar after address compile file
javac -classpath /usr/lib/ambari-server/hadoop-core-1.2.1.jar: /usr/lib/ambari-server/commons-cli-1.3.1.jar
/root/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1.2.3.4.0-3347
/hadoop-common-2.7.1.2.3.4.0-3347.jar
/root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1.2.3.4.0-3347/hadoop-mapreduce-client-core-2.7.1.2.3.4.0-3347.jar
/root/.m2/repository/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar
2. After the packed instruction jar -cvf packaged .jar file name certain .class
Example 3. Submit the input path to the submitted document file path hadoop hadoop fs -put: hadoop fs -put input / * input_wordcount /
4. Submit the name of the main function (main class name, main class where the method name) JAR to perform hadoop hadoop jar jar package execution path input output directory name directory name Example:
hadoop jar word_count_class/wordcount.jar WordCount input_wordcount output_wordcount
/usr/local/hadoop/share/hadoop/mapreduce