CentOS6.5 install Hadoop, Hbase

One, install CentOS6.5

Install CentOS using VMWare

Two, create users and groups

groupadd hadoop
useradd hadoop -g hadoop
passwd hadoop
id hadoop

Three, plan the installation path

–Note : Change of path permissions
a.
Place sub-paths under the root path of installation path /hadoop, for example, the installation path of hbase is /hadoop/hbase
b. Temporary directory
/hadoop/tmp stores temporary files
c. The software directory
used/software is commonly used Installation software

mkdir /hadoop
mkdir /software
mkdir /hadoop/tmp

4. Upload or COPY install software

1. Use FTP
2. Use VMware to share files
chown -R hadoop:hadoop hadoop
chown -R hadoop:hadoop software

Five, turn off the firewall and SELinux

1. Turn off the firewall
temporarily turn off the firewall
service iptables stop
permanently turn off
chkconfig iptables off
2. Turn off SELinux
SElinux is a Linux security enhancement tool
Modify the configuration file
vi /etc/sysconfig/selinux
Modify: SELINUX=disabled
service iptables status can view the current iptables service status.
But even if the service is running, the firewall may not work. You have to look at the firewall rule settings iptables -L
here to talk about the commands to start and close the firewall:

Take effect after restarting
Open: chkconfig iptables on
Close: chkconfig iptables off
Takes effect immediately, and becomes invalid after restart.
Open: service iptables start
Close: service iptables stop

6. Install JDK

tar -zxf jdk-7u79-linux-i586.gz -C …/hadoop/
vi /etc/profile
#######################################
export JAVA_HOME=/hadoop/jdk1.7.0_79
export PATH= $P A T H :$ JAVA_HOME/bin

Seven, planning the host

Use VMWare to clone
master
192.168.1.2
slave1 192.168.1.3 slave2 192.168.1.4

Eight, modify the host name and IP

a. Modify the host name
vi /etc/sysconfig/network
HOSTNAME=master
HOSTNAME=
slave1 HOSTNAME=slave2
b. Modify the IP
vi /etc/sysconfig/network-scripts/ifcfg-eth0
192.168.1.2 master
192.168.1.3 slave1
192.168.1.4 slave2
Service Network restart
c. modify the mapping relationship
vi / etc / hosts
192.168.1.2 Master
192.168.1.3 slave1
192.168.1.4 slave2
Service Network restart
d. The last restart it LINUX

Nine, configure secret SSH login

a. If SSH is not installed, you need to install
ssh -V in advance. Check the installed version of
ssh 127.0.0.1
netstat -a | more to see
if it started normally. If it appears:
tcp 0 0 *:ssh : LISTEN

b. Generate key information.
In the home directory
su hadoop
cd ~
ls -la
ssh-keygen -t rsa -N ""
ls -la
cd .ssh/
ls -la
found two files id_rsa id_rsa.pub
cat id_rsa in the home directory of the hadoop user .pub >> authorized_keys
c. Add to other nodes
In other nodes, respectively
su-hadoop
mkdir .ssh
cd .ssh/
vi authorized_keys
directly COPY key information
d. Test whether the master node can log in to the slave node and the
slave node Click the permissions
mkdir ~/.ssh
chmod 700 ~/.ssh
1) Modify the file "authorized_keys" under the hadoop user
chmod 600 ~/.ssh/authorized_keys
2) Use the root user to set the SSH configuration
vi /etc/ssh/sshd_config
RSAAuthentication yes # Enable RSA authentication
PubkeyAuthentication yes #Enable public key private key pairing authentication method
AuthorizedKeysFile .ssh/authorized_keys #Public key file path
Restart SSH service
service sshd restart

10. Install Hadoop

a. Install Hadoop
on the main node 1. Unzip the file
tar -zxf hadoop-2.6.0.tar.gz -C …/hadoop/ on the main node
2. Modify hadoop-2.6.0/etc/hadoop/hadoop -env.sh Obtain JDK support
export JAVA_HOME=/hadoop/jdk1.7.0_79
3. Modify hadoop-2.6.0/etc/hadoop/core-site.xml global configuration file

    hadoop.tmp.dir
    /hadoop/tmp
    A base for other temporary directories.


    fs.default.name
    hdfs://192.168.1.2:9000

4.修改hadoop-2.6.0/etc/hadoop/hdfs-site.xml
dfs.name.dir
/hadoop/hadoop-2.6.0/namenodedata
dfs.data.dir
/hadoop/hadoop-2.6.0/data
dfs.replication
2

5. Modify-2.6.0 Hadoop / etc / Hadoop / MapReduce mapred-the site.xml
6. The modified hadoop-2.6.0 / etc / hadoop / slaves from the specified node
Slave1
slave2

b. Need to distribute to the other two nodes to synchronize the file content
rsync -avz /hadoop/hadoop-2.6.0 slave1:/hadoop/
rsync -avz /hadoop/hadoop-2.6.0 slave2:/hadoop/
c. Format HDFS
bin/hdfs namenode -format
d. start hdfs
sbin/start-dfs.sh
e. test
jps
or
http://master:50070/
f. stop hdfs
sbin/stop-dfs.sh

11. Install ZooKeeper

http://zookeeper.apache.org/
a. Install ZooKeeper
tar -zxf zookeeper-3.4.6.tar.gz -C …/hadoop/ on the master node
b. Modify zoo_sample.cfg under the conf directory to zoo. cfg
cd zookeeper-3.4.6/conf/

修改zoo.cfg
dataDir=/hadoop/zookeeper-3.4.6/zookdata

Create a file myid in the data directory specified by dataDir (/hadoop/zookeeper-3.4.6/zookdata). The content of the file is a positive integer value to uniquely identify the current machine. Therefore, the value of different machines cannot be the same, you can choose 1 ,2,3
cd /hadoop/zookeeper-3.4.6/zookdata
vi myid
1
Add the following parameters to zoo.cfg
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2: 2888:3888
b. Synchronize
rsync -avz /hadoop/zookeeper-3.4.6 slave1:/hadoop/
rsync -avz /hadoop/zookeeper-3.4.6 slave2:/hadoop/
c. Modify the myid of each node after synchronization
2
3
d Start each node's server process
bin/zkServer.sh start
e. Check whether the installation is successful.
/zkServer.sh status
bin/zkCli.sh -server master:2181
ls /
quit
f. Stop ZooKeeper
bin/zkServer.sh stop

Twelve, install hbase

http://hbase.apache.org/ Make
sure that HDFS is started
a. Unzip HBASE
tar -zxf hbase-1.0.3-bin.tar.gz -C …/hadoop/ on the master node

b. Modify the configuration file
1. Modify the HBase environment variable configuration file hbase-env.sh
export JAVA_HOME=/hadoop/jdk1.7.0_79
2. Modify hbase-site.xml
hbase.rootdir # Set the directory
hdfs where the hbase database stores data : //192.168.1.2:9000/hbase
hbase.cluster.distributed # Open the hbase distribution mode
true
hbase.zookeeper.quorum
master, slave1, slave2 # Specify the zookeeper cluster node name
hbase.zookeeper.property.dataDir # refers to the zookeeper cluster data directory
/ hadoop/zookeeper-3.4.6/zookdata

3. Modify regionservers
Slave1
slave2
C from the two sync nodes.
The rsync -avz /hadoop/hbase-1.0.3 Slave1: / Hadoop /
the rsync -avz /hadoop/hbase-1.0.3 slave2: / Hadoop /

d. Start hbase
./start-hbase.sh
e. Check whether the
jps is started successfully
f. Use the client to connect to hbase
./hbase shell
g. Play with hbase, the main work in the future create'test
','cf1'
list
put 'Test', 'ROW1', 'CF1: A', 'VALUE1'
PUT 'Test', 'ROW1', 'CF1: B', 'value2'
Scan 'Test'
disable 'Test'
drop 'Test'

13. Shut down and start Hadoop, zookeeper, hbase

1. stopped hbase. Bin directory
./stop-hbase.sh
2. each node are stopped zookeeper. Bin directory
./zkServer.sh STOP
3. stopped hadoop. In sbin directory
./stop -dfs.sh

14. Note: This article has been saved in office onenote, and it was published recently. The content belongs to a slightly earlier version of the software. For software content changes during the period, please check the version update content on the official apache website; it is for reference only;