Chapter VII --Hvie deploy big data environment deployment and love between my roommate and Mysql kill one

A, Mysql installation

Preface :( If you have already installed the Mysql can skip this step)
Speaking Mysql dry our line, no one should not be familiar with it, but also the development of personnel required course, each installation will always encounter strange Mysql mistakes, Take my roommate XX people who, every time he is always out of some installation problems, and when I installed but did not appear any problems, in fact, it was a question of practices, the nature of your practices these issues will not find you, but if you engage in hand-finished all hands, then you are probably waiting for a "feces" mountain. Do not say, let's look at how to install it in the end.

• 1, the first step is to clean house
  since we want to install Mysql (that is, to ask someone to come under the command of its own) it must take the house clean, in order to please others;

Take a look at the room and those stains right

dpkg --list|grep mysql

Enlightened look surprised, had so many stains, to clean it before the next

sudo apt-get remove mysql-common

Atta boy less than half, and there are some, again

sudo apt-get autoremove --purge mysql-server-5.7

This pop-up (ask if you keep your data, you die, I want to ask another portal, you must take the clean, delete, ah, decisive choice yes)


I X how there is not another way to deal with urgent these diehards:

dpkg -l|grep ^rc|awk '{print$2}'|sudo xargs dpkg -P

Ha ha ha, this is the absence of it, I'm excited to see a number of

• 2, clean the house, which please the guests
  where to find it, then it must surely go to someone's home when ah please enter Mysql official website :
  
  important information I have marked in the figure.
  Depending on the configuration of their own systems, and needs to select the appropriate version, I'm here because the Ubuntu system, so we chose to download the deb package (just direct the latest version)
  
  https://dev.mysql.com/get/mysql -apt-config_0.8.15-1_all.deb

# 下载
wget https://dev.mysql.com/get/mysql-apt-config_0.8.15-1_all.deb
# 安装mysql的配置包
dpkg -i mysql-apt-config_0.8.15-1_all.deb

This will pop up a dialog box, as I mentioned in the chart, their choice is




so even if you want to install the mysql configuration into a success
because we have just the right Mysql installation package configuration information into the apt them, but not yet force, now need to source package apt conducted under the updated configuration synchronization bit

sudo apt-get update
# 更新完源后就要安装了
# 这里分别安装mysql的服务端和客户端
sudo apt-get install mysql-server mysql-client

• 3, people have to go to their tune now so he is more satisfying

vim /etc/mysql/mysql.conf.d/mysqld.cnf
# 对其添加如下配置
# 也就是将默认编码改成utf8 从而解决中文乱码问题
character_set_server=utf8
collation_server = utf8_general_ci
[client]
default_character_set=utf8

# ！！！注意 ！！！
# 修改完毕后要将mysql服务重启才能生效
sudo service mysql restart

• 4. Login test

mysql -u root -p
#输入你刚才配置的密码进行登录

-- 使用命令查看当前的编码集是否改变
SHOW variables LIKE '%character%';

Thus Congratulations, MySQL has been successfully installed, let's look Hive install it

Two, Hive installation

• 1, what is the Hive
  1) Hive is based on Hadoop (HDFS / MR) is used to manage and query results of / unstructured data warehouse;
  2) a way to store, query and analyze large stored in the Hadoop mechanism of scale data;
  3) Hive defines a simple SQL-like query language called HQL, which allows users familiar with SQL to query data;
  4) complex analysis developed in Java from defined function (UDF) to handle the built-in can not be completed to allow work;
  . 5) Hive no specific data format (separator or the like may set their own flexible);
• 2, download Hive
  Hive Tsinghua mirror

# 老三步
# 1.下载
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz
# 2.解压
tar -zxvf apache-hive-2.3.6-bin.tar.gz -C ~/opt/
# 3.创建软连接
ln -s apache-hive-2.3.6-bin/ hive

• 3, arranged Hive
  (. 1) arranged in the Hive Mysql user (because the metadata is stored in the hive which it is necessary to use MySQL MySQL, the user directly from the root of course also possible, but not recommended)

-- 创建一个hive用户，密码为 123456
CREATE　USER 'hive'@'%' IDENTIFIED BY '123459';
-- 赋予hive用户权限
GRANT ALL PRIVILEGES ON *.* TO 'hive' WITH GRANT OPTION;
-- 刷新，使权限生效
FLUSH PRIVILEGES;
-- 退出
QUIT

（2）hive-env.sh

# 原本env文件是不存在的需要自己copy过来
cp hive-env.sh.template hive-env.sh
# 编辑hive配置文件
vim ~/opt/hive/conf/hive-env.xml
# 填入内容
export HADOOP_HOME=/home/hadoop/opt/hadoop
export HIVE_CONF_DIR=/home/hadoop/opt/hive/conf
export HIVE_AUX_JARS_PATH=/home/hadoop/opt/hive/lib

（3）hive-site.xml

# 这个文件原本也是不存在的，也需要自行复制
cp hive-default.xml.template hive-site.xml
# 编辑
vim hive-site.xml

Fill in the content, pay attention to the original configuration file delete all

<configuration>
	<!--hive存放文件的默认文件格式-->
	<property>
		<name>hive.default.fileformat</name>
		<value>TextFile</value>
	</property>
	<!--连接mysql的地址-->
	<property>
		<name>javax.jdo.option.ConnectionURL</name>
		<value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>
	</property>
	<!--连接mysql时java使用的驱动-->
	<property>
		<name>javax.jdo.option.ConnectionDriverName</name>
		<value>com.mysql.cj.jdbc.Driver</value>
	</property>
	<!--连接mysql时使用的用户名，也就是刚刚新建的那个hive用户-->
	<property>
		<name>javax.jdo.option.ConnectionUserName</name>
		<value>hive</value>
	</property>
	<!--上面用户密码-->
	<property>
		<name>javax.jdo.option.ConnectionPassword</name>
		<value>123456</value>
	</property>
	<!--是否验证存储再 metastore 中的版本是否和hive jar中的版本一致，
	     设置为 false 如果上述不一致则会发出警告-->
	<property>
		<name>hive.metastore.schema.verification</name>
		<value>false</value>
	</property>
</configuration>

(4) Mysql connect the drive
or enter the MySQL official website:

# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.19-1ubuntu18.04_all.deb
# 注意下载下来的是deb包，我们只需要jar包即可，当然你也可以安装
# 这里我只是解压即可
dpkg -X mysql-connector-java_8.0.19-1ubuntu18.04_all.deb ./
# 移动jar包到 hive的lib目录中去
mv usr/share/java/mysql-connector-java-8.0.19.jar ~/opt/hive/lib/

If you can be bothered to pick up directly from my GitHub in
https://github.com/NickWike/BigdataInstaller/raw/master/mysql-connector-java-8.0.19.jar
(5) Add the environment variable

# 编辑
vim ~/.bashrc
# 末尾添加内容


# hive
export HIVE_HOME=/home/hadoop/opt/hive
export PATH=${HIVE_HOME}/bin:${PATH}

# :x 保存退出
# 刷新使之生效
source ~/.bashrc

• 4, initialization, and start the Hive
  (1) First start hadoop (start-all.sh)
  (2) new directory / user / hive in hdfs in
  hdfs the DFS -mkdir -p / user / hive
  # change permissions to 733
  hdfs the DFS - -R & lt the chmod / User /
  
  (. 3) table initialization Mysql

schematool -initSchema -dbType mysql

Open the hive test

to check the data to see, in the construction of a database look

good no problem, hive installation is over, there is the problem of a small partner, welcomed the comments area to explore issues yo, or whisper bloggers will do.