Hadoop (Learning & # 183; 2)

Hadoop

Steps:

192.168.1.110-113 host names are nn01 (zhuji) node1 node2 node3 between all hosts to communicate secret Free / etc / host

1. Installation openjdk-devel

# yum -y install java-1.8.0-openjdk-devel

# yum -y install java-1.8.0-openjdk.x86_64

2. modify the configuration file

# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

ExportJAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/jre"

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/local/hadoop/etc/hadoop"}

These two lines arranged to modify the path

3. The packet sent to the management machine hadoop nn01

# mv hadoop /usr/local/hadoop

# cd /usr/local/hadoop/etc/hadoop/

# vim slaves

Node1

Node2

Node3

# core-site.xml

vim<name>fs.defaultFS</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/var/hadoop</value>

</property>

</configuration>

# for i in 192.168.1.11{1..3}; do ssh $i mkdir /var/hadoop; done

# vim hdfs-site.xml

<name>namenode.http-address</name>

</property>

<name>dfs.namenode.secondary.http-address</name>

</property>

<name>dfs.replication</name>

</property>

</configuration>

4. The above-described configuration file <name> dfs.namenode.secondary.http-address </ name> intermediate which can copy log dahoop.hapache.org

Above can log hadoop.apache.org

Then find and install the most recent version of their own in the bottom left corner and then

In the lower left corner of the bottom

Then hadoop entire file is sent to the other nodes node1 node2 node3

# for i in 192.168.1.11{1..3}; do scp -r /usr/local/hadoop/ $i:/usr/local/; done

# ./Bin/hdfs namenode -format

# ./sbin/start-dfs.sh

# jps

# vim /usr/local/hadoop/etc/hadoop/mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

# vim yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

# cd /usr/local/hadoop

# ./Sbin/start-yarn.sh restart

# ./bin/yarn node -list

# http://192.168.1.110:50070

# http://192.168.1.110:50090

# http://192.168.1.110:8088

# http://192.168.1.111:50075

# http://192.168.1.111:8042

The success can be seen as a success even if the page

# ./Bin/hadoop -ls FS / View] [cluster file system / case

ls / View] [local file system / case

# ./Bin/hadoop FS -mkdir / oo create folders in the file system [cluster]

mkdir / oo create a folder in the file system [local]

# ./Bin/hadoop FS -put uploaded to the cluster file system

# ./bin/hadoop fs *.txt -put /oo

# ./bin/hadoop fs -ls /

# ./Bin/hadoop FS -get downloaded to the local file system

Call cluster analysis

# ./bin/hadoop jar

share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount /oo /xx

# ./bin/hadoop fs -cat /xx/*

Install two machines. Disable selinux, uninstall firewalld

192.168.1.115 " nsfgw

192.168.1.120 》 client

Note: Nfsgw Configure / etc / hosts need to nn01 host's / etc / hosts

In nn01 with nfsgw add users nsd1802 ( uid giu to be the same)

1. Stop hadoop cluster /usr/local/hadoop/sbin/stop-all.sh

In nn01 arranged on the host vim core-site.xml

<name>hadoop.proxyuser.nsd1802.groups</name>

</property>

<name>hadoop.proxyuser.nsd1802.hosts</name>

</property>

2. Synchronization configuration to node1 node2 node3 restart

In nn01 transmitter / usr / local / hadoop to nfsgw host

Host nfsgw

# rm -rf /usr/local/hadoop/logs

# mkdir /usr/local/hadoop/logs

# setfacl -m usr:nsd1802:rwx logs

In nfsgw host

# yum -y install java-1.8.0-openjdk-devel

# yum -y install java-1.8.0-openjdk.x86_64

# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<property><name>nfs.exports.allowed.hosts</name>

</property>

<name>dfs.namenode.accesstime.precision</name>

</property>

<value>/var/nfstemp</value>

</property>

<name>nfs.rtmax</name>

</property>

<name>nfs.wtmax</name>

</property>

<name>nfs.port.monitoring.disabled</name>

<value>false</value>

</property>

</configuration>

3. Create the / var / hadoop

Creating / var / nfstemp

# chown nsd1802:users /var/nfstemp

# yum remove rpcbind nfs-utils

# /usr/local/hadoop/sbin/hadoop-daemon.sh --script ./bin/hdfs start portmap

# Of -l nsd1802

# /usr/local/hadoop/sbin/hadoop-daemon.sh --script ./bin/hdfs start nfs3

# jps

1141 Nfs3

1047 Portmap

1327 Jps

4. The machine of the client

# yum -y install nfs-utilsmount -t nfs -o

# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,noatime,sync 192.168.1.115:/ /mnt/

zookeeper

• Zookeeper experiment

- build a zookeeper cluster known

- Adding observer

- Find leader

- Analog leader fault

- Analog follower fault

- Recovery

kafka cluster

• kafka What is ?

- Kafka by LinkedIn developed a distributed messaging system

- kafka is using Scala to write

- kafka is a messaging middleware

• Why use kafka

- Decoupling, redundancy, improved scalability, buffer

- guarantee the order , flexible , load shifting

- asynchronous communication

kafka cluster

• kafka role does the cluster structure

- producer: producer , responsible for publishing news

- consumer: the consumer , responsible for processing messages reading

- topic: category of the message

- Parition: Each Topic contains more than one ring Partition.

- Broker: Kafka cluster server contains more than one ring

- Kafka by Zookeeper management cluster configuration , choose Dan leader

L Kafka used to live cluster

• kafka role does the cluster structure

L Kafka used to live cluster

• kafka installing the cluster configuration

- kafka installation configuration of the cluster is dependent zookeeper 's , take construction kafka ago cluster Ho , first of all created a usable zookeeper cluster

- Installation openjdk operating environment

- distribute kafka copied to all cluster hosts kafka cluster

• kafka installing the cluster configuration

• server.properties

– broker.id

- each server broker.id will be the same as Ji

– zookeeper.connect

- zookeeper cluster address , Ji used are listed , write to part

- modify the configuration file

- throat Mai does not validate

• kafka installing the cluster configuration

- in all hosts throat Mai Service

– /usr/local/kafka/bin/kafka-server-start.sh

-daemon /usr/local/kafka/config/server.properties

- Verify

- jps command should be able to see kafka module

- netstat should see 9092 in listening

L Kafka used to live cluster

• Cluster Verification not news release

- Creating a topic

./bin/kafka-topics.sh --create --partitions 2 --replication-

factor 2 --zookeeper node3:2181 --topic mymsg

- Check existing topic

./bin/kafka-topics.sh --list --zookeeper node2:2181

- View just created topic

./bin/kafka-topics.sh --describe --zookeeper node1:2181 --topic mymsg

L Kafka used to live cluster

• Cluster Verification not news release

- in both terminals inside , producers announced , consumers read messages

- Producer

./bin/kafka-console-producer.sh --broker-list master:9092,node1:9092 --topic mymsg

- Consumers

./bin/kafka-console-consumer.sh --bootstrap-server node2:9092,node3:9092 --topic mymsg

- --from-beginning indicates read messages from beginning

L Kafka used to live cluster experiment

• use zookeeper build a kafka cluster

• Create a topic

• View the system topic Love Hu

• Analog producers announced

• Analog consumers receive messages

Hadoop (Learning & # 183; 2)

Guess you like