Prepare three virtual machines, ip were 192.168.220.10 (master), 192.168.220.11 (slave1), 192.168.220.12 (slave2)
Ready jdk-6u45-linux-x64.bin and hadoop-1.2.1-bin.tar.gz, placed in / usr / local / src / directory
Install the JDK (Each virtual machine installation)
1. Go to the / usr / local / src / directory, execute ./jdk-6u45-linux-x64.bin
2. Modify ~ / .bashrc, at the end of the file increased three lines
export JAVA_HOME=/usr/local/src/jdk1.6.0_45 export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib export PATH=$PATH:$JAVA_HOME/bin
3. To make the environment variables into effect, execute source ~ / .bashrc
Hadoop installation
Hadoop mounted on the machine 192.168.220.10
1. Go to the / usr / local / src / directory, unzip hadoop-1.2.1-bin.tar.gz, performing tar -zxf hadoop-1.2.1-bin.tar.gz
2. modify the configuration file
masters file
master
slaves file
slave1 slave2
core-site.xml file
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/src/hadoop-1.2.1/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://192.168.220.10:9000</value> </property> </configuration>
mapred-site.xml file
<configuration> <property> <name>mapred.job.tracker</name> <value>http://192.168.220.10:9001</value> </property> </configuration>
hdfs-site.xml file
<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>
hadoop-env.sh file, add a line at the back
export JAVA_HOME=/usr/local/src/jdk1.6.0_45
3. /usr/local/src/hadoop-1.2.1 catalog copy machine to 192.168.220.11,192.168.220.12
Configuration hostname
192.168.220.10 host configuration called master
1. Perform hostname master
2. Modify the / etc / hostname file
master
Modify 192.168.220.11 host name slave1, modify the host 192.168.220.12 named slave2
Host configuration file
Three machines host file, add the following code at the end
192.168.220.10 master 192.168.220.11 slave1 192.168.220.12 slave2
Configuring SSH
1. executed on 192.168.220.10 ssh-keygen, at ~ .ssh directory new directory, directory file id_rsa, id_rsa.pub
2. Copy id_rsa.pub as authorized_keys
cp id_rsa.pub authorized_keys
3. Run the ssh-keygen 192.168.220.11 192.168.220.12 respectively, and in
4. 192.168.220.11 and 192.168.220.12 contents are copied to the id_rsa.pub authorized_keys file 192.168.220.10 are as follows:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9mGRhFOdcoHw9GUnKQmqThNKpsyah93Dtq/d8RICGWIHDRJ3GXd0sEcb743ejwbuCMmtlhheXcU0FuyA6Cm0jvMyvDfaPKArtxl6KT7Z93uC0VDCXDRomueux81HAIVjc7ZqlXwVeYs1LITxEeJykKlFOXvK7JexWhWGdMMADwxbFMbaNsZ9EwRxcFLFtNg65FQ+u8CIV9KR3D02kemwLCsP+xiRcgs+wirQPm5JM+2cJoLsVQBz3Hk335IsEhc1Xb9Cralo8Tt8gh/ho8K/1pVjvyW1b0LkP9HGNdwVYD9wkWdEJRkryLXBEXpjk4xu+riF+N4rOzJD root@master ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn79fdfR/NjzPVD3NPj1vBBfQdVOrv7jeb4UJCOsd7xioPRiz8gOQnOmhu5C+GchbyGA+tg5pXwnNJTOO2wn32U4lOPndW0okN/wqyN4vgq/taJi7JgY/8rneBiGaIIdNIy/pAGlMwb53Qn766adetMhsxYMD2l4uxmbVVjzCRb8QP5EsAYTmmFOODzJsPm70uF3j1Q8zGavYg0wFSYR/yECQns4DBSuBJNxdGY6PskBXqurahwi5yaR3vWV1Ix4wtB6BYuQomEnGdzOSfrBMZ/yc5tXo0xmEfY7wFkize6z9Pm2E3oDoMR18YkwT1Cz6fHikVILA9cldtL root@slave1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCydYCvASzCZggks4hMqOcYSGLO2eAvocWezNOMwspTfpJ105Jumb/vf5h6cRZeckq56IvhSV6t6mytk4pZoZjjZPSmWvCwLtMRMPShNbA3BYtj5V3WRKV8ZcMrNdD//U7iHHoJm57vI/m+XO42YSYjPw7JDkb8Ij9b6zgI3fyvbSSYeXb451PlyJLHdxIzRMAaZDSbAML9e7EO8VJB9Wf9bXpow4+VeP33it3kgMNUlHQtyqduSwYGxVVtGsUTJkxnuRsbWeeA1/pp8MNFKUgBTMALTVHByglgZqwGcbblJxsG832PIZNRECIFqorm6odftjnT4DR7/0yR root@slave2
5. Copy 192.168.220.10 authorized_keys file to the machine 192.168.220.11,192.168.220.12
Once configured three machines can visit each other without a password
Well, here on the Hadoop cluster configuration is over, let's use the next bar
Formatting in the namenode 192.168.220.10
/Usr/local/src/hadoop-1.2.1/bin into the directory, execute
./hadoop namenode -format
appear
19/08/04 15:15:21 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master/192.168.220.10 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.6.0_45 ************************************************************/ 19/08/04 15:15:21 INFO util.GSet: Computing capacity for map BlocksMap 19/08/04 15:15:21 INFO util.GSet: VM type = 64-bit 19/08/04 15:15:21 INFO util.GSet: 2.0% max memory = 1013645312 19/08/04 15:15:21 INFO util.GSet: capacity = 2^21 = 2097152 entries 19/08/04 15:15:21 INFO util.GSet: recommended=2097152, actual=2097152 19/08/04 15:15:22 INFO namenode.FSNamesystem: fsOwner=root 19/08/04 15:15:22 INFO namenode.FSNamesystem: supergroup=supergroup 19/08/04 15 : 15 : 22 INFO namenode.FSNamesystem: isPermissionEnabled = true 19 / 08 / 04 15 : 15 : 22 INFO namenode.FSNamesystem: dfs.block.invalidate.limit = 100 19 / 08 / 04 15 : 15 : 22 INFO namenode .FSNamesystem: isAccessTokenEnabled = false accessKeyUpdateInterval = 0 min (s), accessTokenLifetime = 0 min (s) 19 / 08 / 04 15 :15:22 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 19/08/04 15:15:22 INFO namenode.NameNode: Caching file names occuring more than 10 times 19/08/04 15:15:23 INFO common.Storage: Image file /usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds. 19/08/04 15:15:23 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits 19/08/04 15:15:23 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits 19/08/04 15:15:23 INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2.1/tmp/dfs/name has been successfully formatted. 19/08/04 15:15:23 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master/192.168.220.10 ************************************************************/
View the process with jps
[root@master bin]# jps 19905 JobTracker 19650 NameNode 19821 SecondaryNameNode 20202 Jps
In 192.168.220.11 view the process
9289 DataNode 9493 Jps 9391 TaskTracker
In 192.168.220.12 view the process
6823 DataNode 6923 TaskTracker 7057 Jps
Test case:
Execution ./hadoop fs -ls /
drwxr-xr-x - root supergroup 0 2019-08-04 15:15 /usr
Upload files, execute ./hadoop fs -put /root/w.txt /
View file execution ./hadoop fs -cat /w.txt, display
ddd
success