1. Installation preparation
1. Download zookeeper-3.3.1,
Address: http://www.apache.org/dist/hadoop/zookeeper/zookeeper-3.3.1/
2. JDK version: jdk-6u20-linux-i586.bin
3. Operating system: Linux
4. The default premise is that hadoop version 0.20.2 is installed:
192.168.3.131 namenode
192.168.3.132 datanode
192.168.3.133 datanode
2. Operation steps (default on namenode)
1. Copy the above files to the "/usr/" directory of Linux. At the same time, create a new directory "/zookeeper-3.3.1".
2. Install JDK, this step is omitted...
3. Unzip zookeeper to the /zookeeper-3.3.1 directory. tar -zxvf zookeeper-3.3.1.tar.gz -C /
zookeeper-3.3.1
4. Change the name of zoo_sample.cfg under the "/zookeeper-3.3.1/conf" directory to "zoo.cfg"
5. Open the zoo.cfg file and modify the configuration as follows:
dataDir=/usr/zookeeper-3.3.1/data
dataLogDir=/usr/zookeeper-3.3.1/log
clientPort=2181
initLimit=10
syncLimit=5
tickTime=2000
server.1=192.168.3.131:2888:3888
server.2=192.168.3.132:2888:3888
server.3=192.168.3.133:2888:3888
6. Create the directory specified by the dataDir parameter (here refers to "/zookeeper-3.3.1/data"), and create a file in the directory,
Name it "myid".
7. Edit the "myid" file, and enter the corresponding number on the corresponding IP machine. As on 192.168.3.131,
The content of the "myid" file is 1, and on 192.168.3.132, the content is 2
8. Generate the zookeeper.sh file under profile.d and set it as follows:
# zookeeper path
ZOOKEEPER=/usr/zookeeper-3.3.2
PATH=$PATH:$ZOOKEEPER/bin
export PATH
Then apply . ./zookeeper.sh
9. Copy the "/usr/zookeeper-3.3.1" directory to 192.168.3.132 and 192.168.3.133 respectively.
At the same time, modify the corresponding "myid" file content.
10. At this point, the installation steps are over, and then start zookeeper.
3. Start zookeeper
1. Execute shell scripts on three machines respectively. "sh /jz/zookeeper-3.3.1/bin/zkServer.sh start"
2. Start the client script: "zkCli.sh -server 192.168.1.132:2181"
3. After the execution is completed, enter the "jps" command to view the process as follows:
The display on the namenode is as follows:
29844 JobTracker
29583 NameNode
31477 HMaster
29762 SecondaryNameNode
32356 Jps
31376 HQuorumPeer
datanode ：
16812 DataNode
17032 HRegionServer
17752 HQuorumPeer
16921 TaskTracker
18461 Jps
3. Check whether it is started by entering "sh /jz/zookeeper-3.3.1/bin/zkServer.sh status",
The general return content is as follows:
leader：
JMX enabled by default
Using config: /jz/zookeeper-3.3.1/bin/../conf/zoo.cfg
Mode: leader
follower：
JMX enabled by default
Using config: /jz/zookeeper-3.3.1/bin/../conf/zoo.cfg
Mode: follower
4. Check the cluster zookeeper startup status by entering commands in the console.
Commands such as "echo ruok | nc 192.168.3.131 2181",
console output "imok"
4. Matters needing attention
When starting zookeeper on each machine through a shell script,
The error message "Cannot open channel to X at election address" may be displayed.
This is caused by other zookeeper services specified in the zoo.cfg file not being found.
The error message will disappear after the zookeeper service of all machines is started.
One machine
192.168.0.203 hd203: hadoop namenode & hbase HMaster
192.168.0.204 hd204: hadoop datanode & hbase HRegionServer & zookeeper
192.168.0.205 hd205: hadoop datanode & hbase HRegionServer & zookeeper
192.1 68.0.206 hd206: hadoop datanode & hbase HRegionServer & zookeeper
192.168. 0.202 hd202: hadoop second namenode & hbase HMaster,
a total of 5 machines (the production environment needs to install zookeeper separately)
2 operating system and software version
1 Operating system: CentOS release 5.6 (Final) x64, turn off the firewall
2 Related software:
jdk-6u24- linux-x64.bin
hadoop-0.20.2-cdh3u0.tar.gz
hbase-0.90.1-cdh3u0.tar.gz
zookeeper-3.3.3-cdh3u0.tar.gz
three installation
1 install jdk (all machines)
mkdir /usr /java
mv jdk-6u24-linux-x64.bin /usr/java
chmod 744 jdk-6u24-linux-x64.bin
./jdk-6u24-linux-x64.bin
ln -s jdk1.6.0_24 default
2 Create a cluster platform user (all machines)
useradd cbcloud
passwd cbcloud password is set to 123456
3 edit/ etc/hosts (all machines)
192.168.0.203 hd203
192.168.0.204 hd204
192.168.0.205 hd205
192.168.0.206 hd206
192.168.0.202 hd202
4 hd203 login cluster all machines without password setting (hd 203)
su - cbcloud
ssh login to this machine without password Set as follows
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Append the id_dsa.pub file on hd203 to all
The details are not in the authorized_keys file of the cbcloud user of the machine
. ssh All machines need to be confirmed for the first time.
5 Install hadoop
5.1 Create a directory (all machines)
mkdir /home/cbcloud/hdtmp
mkdir /home/cbcloud/hddata
mkdir /home/cbcloud/hdconf
chown -R cbcloud:cbcloud above directory
tar zxvf hadoop-0.20.2-cdh3u0.tar.gz -C /home/cbcloud
cd /home/cbcloud
mv hadoop-0.20 .2-cdh3u0 hadoop
chown -R cbcloud:cbcloud hadoop/
5.2 Configure environment variables vi /etc/profile (all machines)
export JAVA_HOME=/usr/java/default
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre /lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
export HADOOP_HOME=/home/cbcloud/hadoop
export HADOOP_CONF_DIR=/home/cbcloud/hdconf
export PATH= $PATH:$HADOOP_HOME/bin
Take out the hadoop configuration file directory from the source directory, so that you can upgrade
the files in the hadoop conf directory to /home/cbcloud/hdconf in the future.
5.3 编辑 hadoop 配置文件 core-site.xml
加入
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hd203:9000</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/home/cbcloud/hdtmp/dfs/namesecondary</value>
<description>Determines where on the local filesystem the DFS secondary
name node should store the temporary images to merge.
If this is a comma-delimited list of directories then the image is
replicated in all of the directories for redundancy.
</description>
</property>
<property>
<name>fs.checkpoint.period</name>
<value>60</value>
<description>Determines where on the local filesystem the DFS secondary
name node should store the temporary images to merge.
If this is a comma-delimited list of directories then the image is
replicated in all of the directories for redundancy.
</description>
</property>
5.4 编辑hdfs-site.xml
加入
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/cbcloud/hddata</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/cbcloud/hdtmp/</value>
</property>
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>10485760</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/home/cbcloud/hdconf/excludes</value>
<final>true</final>
</property>
5.5 编辑mapred-site.xml
加入
<property>
<name>mapred.job.tracker</name>
<value>hd203:9001</value>
</property>
5.6 编辑 hadoop-env.sh
export JAVA_HOME=/usr/java/default
5.7 Edit masters This file specifies the secondary name machine, hd206hd205hd204addedit slaveshd202
add
5.8 Copy hadoop and hdconf of hd203 to all machines
# scp -r /home/cbcloud/hadoop cbcloud@hd204:/home/cbcloud
# scp -r /home/cbcloud/hdconf cbcloud@hd204:/home/
cbcloud hd203 Format the hadoop file system
Execute
hadoop namenode -format
Start
start-all.sh
View the datanode machines in the cluster
Execute jps
5764 Jps
18142 DataNode
18290 TaskTracker
See the above results and start the correct
web method
http://hd203:50070/dfshealth. jsp
Note that the local PC hosts file should also be configured with
192.168.0.203 hd203
192.168.0.204 hd204
192.168.0.205 hd205
192.168.0.206 hd206
192.168.0.202 hd202
The web mode can view the cluster status and job status, etc. so far Doop installed
6 Install zookeeper (hd203)
tar zxvf zookeeper-3.3.3-cdh3u0.tar.gz -C /home/cbcloud
在hd204-hd206上
mkdir /home/cbcloud/zookeeperdata
chown -R cbcloud:cbcloud /home/cbcloud/zookeeperdata
chown -R cbcloud:cbcloud /home/cbcloud/zookeeper-3.3.3-cdh3u0
编辑 /home/cbcloud/zookeeper-3.3.3-cdh3u0/conf/zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/cbcloud/zookeeperdata
# the port at which the clients will connect
clientPort=2181
server.1=hd204:2888:3888
server.2=hd205:2888:3888
server.3=hd206:2888:3888
scp hd203 zookeeper to hd204,hd205,hd206
# scp -r /home/cbcloud/zookeeper-3.3.3-cdh3u0/ cbcloud@hd205:/home/cbcloud/
touch the myid file in the /home/cbcloud/zookeeperdata directory of hd204-206,
the contents are 1, 2, 3 and The server number is the same. chown cbcloud:cbcloud myid Start zookeeper, execute # zkServer.sh start
in the bin directory on hd204-206 After starting, check the status through # zkServer.sh status Note that executing this on centos5.6 will report Error contacting service. It is probably not running. By checking the script, it is because echo stat | nc -q 1 localhost nc version is different, there is no -q parameter, just change the script to remove -q 1. In addition, you can pass
echo stat | nc localhost 2181 to view status
7 Install hbase
7.1 Create directory (all machines)
mkdir /home/cbcloud/hbconf
chown -R cbcloud:cbcloud /home/cbcloud/hbconf
tar zxvf hbase-0.90.1-cdh3u0.tar. gz -C /home/cbcloud
cd /home/cbcloud
mv hbase-0.90.1-cdh3u0 hbase
chown -R cbcloud:cbcloud hbase/
7.2 Configure environment variables
vi /etc/profile (all machines) add the following content
export HBASE_CONF_DIR=/home /cbcloud/hbconf
export HBASE_HOME=/home/hadoop/hbase
Take out the hbase configuration file directory from the source directory to facilitate future upgrades
of hbase mv hbase conf directory files to /home/cbcloud/hbconf
7.3 Edit hbase-env. sh
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
export JAVA_HOME=/usr/java/default
export HBASE_MANAGES_ZK=false
export HBASE_HOME=/home/cbcloud/hbase
export HADOOP_HOME=/home/cbcloud/hadoop
7.4 编辑hbase-site.xml
加入
<property>
<name>hbase.rootdir</name>
<value>hdfs://hd203:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.master</name>
<value>hd203:60000</value>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
<description>The port master should bind to.</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hd204,hd205,hd206</value>
</property>
7.5 Edit regionservers
and add hbase of
hd204
hd205
hd206
scp hd203 to hd204-206,202
# scp -r /home/cbcloud/hbase/ cbcloud@hd204:/home/cbcloud
# scp -r /home/cbcloud/hbconf/ cbcloud@hd204:/home/cbcloud
su - cbcloud
starts hbase and executes $ ./
in the bin directory of hbase on hd203
start-hbase.sh
starts the second HMaster of hbase and executes $ ./hbase-daemon.sh start master
on 202 to view the Master: http://hd203:60010/master.jsp now view datanode server jps 5764 Jps 32091 HRegionServer 18142 DataNode 11221 QuorumPeerMain
18290 The above results of TaskTracker
indicate that the startup is normal.
So far, the installation of hadoop+zookeeper+hbase is complete.
Startup sequence
1.hadoop
2.zookeeper
3.hbase
4. The second HMaster
stop sequence
1. The second HMaster, kill-9 delete
2.hbase
3. Zookeeper
4. Hadoop
must be stopped in order.
If you stop zookeeper first and then stop hbase, you will basically not be able to stop (your own test results)