Table of contents
1.2.3 Turn off the firewall and selinux
1.2.4 Password-free configuration (also required for a single machine)
1.2.5 Modify yum source to Ali source
1.2.7 NTP clock synchronization
1.2.9 Install MySQL and create the library
3. Install your own components through Ambari
4.1.1 Download the relevant installation package
4.1.2 Upload the decompressed copy
5. Problems encountered during installation and deployment
5.1 Error: Could not open client transport with JDBC Uri
5.2 Error when starting hive service
5.3 Problems with elasticsearch username group
5.4 elasticsearch cannot import format_hdp_stack_version
5.5 kibana cannot import format_hdp_stack_version
5.6 kibana cannot import format_hdp_stack_version
5.7 hostname was not found in configuration dictionary
5.8 master and slave stop after starting
5.9 parent directory /opt/elasticsearch/master/config doesnot exist
1. Preparation
1.1 Environment Release Notes
Amber : 2.7.1.0-169
Hdp：3.0.1.0-187
Centos: 3.10.0-123.el7.x86_64
IP：192.168.31.139
There is only one virtual machine, so all adopt single-node mode.
1.2 Initial configuration
1.2.1Set the host name
echo master >> /etc/hostname
1.2.2 HOSTS file
cat > /etc/hosts <<EOF
192.168.31.139 master
EOF
1.2.3 Turn off the firewall and selinux
#Close firewall
systemctl stop firewalld.service
systemctl disable firewalld.service
#View status
firewall-cmd --state
#Close selinux
setenforce 0
sed -i ‘s/SELINUX=*/SELINUX=disabled/g’ /etc/sysconfig/selinux
sysctl -p /etc/sysconfig/selinux
1.2.4 Password-free configuration (also required for a single machine)
Execute the following command:
1）# ssh-keygen -t rsa
Press Enter until the end . Two files, id_rsa and id_rsa.pub, are generated in the /root/.ssh/ directory. The former is the private key and the latter is the public key.
2）#cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
1.2.5 Modify yum source to Ali source
- Backup first
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo_bak
- Download repo source
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
1.2.6 Configure JAVA environment variables (if the java environment is not installed, you need to install it first)
Note: By default, there is only jre environment and no jdk development environment.
View the default operating environment jre of linux7
# ls -lrt /etc/alternatives/java
Add the following configuration to .bash_profile in the root directory (java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64 is the built-in java version):
# jdk config
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64
export JRE_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.362.b08-1.el7_9.x86_64/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
Then load the environment variables
source .bash_profile
1.2.7 NTP clock synchronization
#Install ntpdate command
# yum install ntpdate -y
#Synchronize with network time
# ntpdate cn.pool.ntp.org
#Make time synchronization into a scheduled task
# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
#Restart crond service
#service crond restart
1.2.8 Install h ttpd service
Download the httpd package and dependent packages:
Link: https://pan.baidu.com/s/1roRtXBkmfM2WOoLcoDfXAQ
Extraction code: q4my
Execute the following command, follow the rpm package
rpm -ivh XXX.rpm
start up:
#systemctl start httpd.service
starts at boot:
#systemctl enable httpd.service
1.2.9 Install MySQL and create the library
1) Installation
Select the version to download via https://mariadb.org/mariadb/all-releases
Follow the prompts to create the yum source information file
#vi /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = https://mirrors.aliyun.com/mariadb/yum/10.3/centos7-amd64
gpgkey=https://mirrors.aliyun.com/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
# yum install -y MariaDB-server MariaDB-client
Start mariadb
systemctl start mariadb
and set it to start automatically at boot
systemctl enable mariadb
2) Create library ambari and hive library
create database ambari character set utf8 ;
CREATE USER 'ambari'@'%'IDENTIFIED BY 'XXXXX';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%' IDENTIFIED BY 'XXXXX';
FLUSH PRIVILEGES;
create database hive character set utf8 ;
CREATE USER 'hive'@'%'IDENTIFIED BY 'XXXXX';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'XXXXX';
3) Download the jdbc driver package
The relevant jars of mysqld can be downloaded from MySQL :: Download MySQL Connector/J (Archived Versions) and placed under /usr/share/java.
2. Deploy Ambari
2.1 Download package
Download Ambari HDP, HDP-UTILS, HDP-GPL source packages
# mkdir -p /var/www/html/ambari
将ambari 电影的包解历到var/www/html/ambari下
# tar -zxvf ambari-2.7.1.0-centos7.tar.gz -C /var/www/html /ambari/
# unzip HDP.zip -C /var/www/html/ambari/
# tar -zxvf HDP-GPL-3.0.1.0-centos7-gpl.tar.gz -C /var/www/html/ambari/
# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/ambari/
After the httpd service is started, execute the command curl http://192.168.31.139:8000/ambari/ and the following return indicates that the httpd service is normal.
2.2 Configure local sources
cd /etc/yum.repos.d/ Configure related repo files
[root@master yum.repos.d]# cat hdp.gpl.repo
#VERSION_NUMBER=3.0.1.0-187
[HDP-GPL-3.0.1.0]
name=HDP-GPL Version - HDP-GPL-3.0.1.0
baseurl=http://192.168.31.139:8000/ambari/HDP-GPL/centos7/3.0.1.0-187/
gpgcheck=1
gpgkey=http://192.168.31.139:8000/ambari/HDP-GPL/centos7/3.0.1.0-187/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[ root @ master yum . repos . d ] # cat ambari . repo
#VERSION_NUMBER=2.7.1.0-169
[ambari-2.7.1.0]
name=ambari Version - ambari-2.7.1.0
baseurl=http://192.168.31.139:8000/ambari/ambari/centos7/2.7.1.0-169/
gpgcheck=1
gpgkey=http://192.168.31.139:8000/ambari/ambari/centos7/2.7.1.0-169/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@master yum.repos.d]# cat hdp.repo
#VERSION_NUMBER=3.0.1.0-187
[HDP-3.0.1.0]
name=HDP-3.0.1.0
baseurl=http://192.168.31.139:8000/ambari/HDP/centos7/3.0.1.0-187/
gpgcheck=1
gpgkey=http://192.168.31.139:8000/ambari/HDP/centos7/3.0.1.0-187/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22/
baseurl=http://192.168.31.139:8000/ambari/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://192.168.31.139:8000/ambari/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
Then execute the following command:
#yum clean all //Clear YUM cache
#yum makecache //Create cache
#yum repolist
2.3 Install and start ambari
# yum install ambari-server -y
# ambari-server setup
Select to install according to the prompts
1. y
2. Press Enter
3. Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)
4. Enter advanced database configuration [y/n] (n)? y
5 .Enter choice (1): 3
Hostname (localhost): 192.168.31.139
Port (7306):
Database name (ambari): ambaridb
MySql schema (ambari):
Username (ambari): ambari
Enter Database Password (ambari):Amvari-123
Re-enter password:
Configuring ambari database...
--------------------------------------- -------------------
Start the service
#ambari-server start
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
Access the Web initialization configuration cluster
URL: http:192.168.31.139:8080
Account/Password: admin/admin
3. Install your own components through Ambari
1) The Ambari service is started successfully. Enter the Ambari address in the browser: http://192.168.31.139:8080. The default username is admin and the password is admin. After logging in, you will enter the following interface:
2) Click "LAUNCH INSTALL WIZARD" to create a cluster, enter the cluster name "HDPSKY" in the following interface, and customize the name.
3) Click "next", the following interface will pop up, select "Use Local Repository", then select the corresponding os in the section, and enter the source address in the Base URL of each item, which is the httpd service address configured in Section 1.2.8 .
4) Click "next" and the following interface will pop up. Select the authentication method through SSH Private Key, that is, configure mutual password exemption in Section 1.2.4. Copy the id_rsa private key on the master machine locally, click "CHOOSE FILE" to upload, and click "next".
Note: If it is a single node, you also need to configure password-free, that is, you do not need to enter a password for the ssh local address, which means the configuration is correct. Otherwise, after clicking "next", the server log will report the following error.
5) After clicking "next", the following warning message pops up, which can be ignored. The reason is that the domain name configured for hosts file analysis is master, and click "CONTINUE".
6) Select the service for installation and configuration. The default selection is sufficient. If it is not installed at this time, you can also add it for installation after the process is completed.
7) Service Master configuration
8) Slaves and Clients section configuration of the service
In the pop-up box below, enter the relevant password, which is the same as the username by default.
9) Customized configuration of services. If there is an alarm, please follow the prompts to modify the address or password.
Note that when configuring the HIVE database, select the "Existing MySQL" option because MySQL is already installed on the master node. Otherwise, the following error will be reported:
Pay attention to the red box in the picture above. You need to download the jdbc channel package. See step 3) of 1.2.9 for preparation. Just perform the following operations.
After the configuration modification is completed, you can click "TEST CONNECTION" to perform a connection test. If the following appears, it indicates success.
After the installation starts, the following prompt appears. You can click on the red box to view the log details for analysis and resolution.
4. Install elk through Ambrai
4.1 Preparation work
4.1.1 Download the relevant installation package
elasticsearch-6.4.2.tar.gz
kibana-6.4.2-linux-x86_64.tar.gz
Ambari-Elastic-Service-master.zip
download address:
https://www.elastic.co/cn/downloads/past- releases/elasticsearch-6-4-2
download address:
https://www.elastic.co/cn/downloads/past-releases/kibana-6-4-2
download address: https://github.com/BalaBalaYi/ Ambari-Elastic-Service
4.1.2 Upload the decompressed copy
1) Ambari-Elastic-Service-master.zip
creates a new /opt/es directory, uploads Ambari-Elastic-Service-master.zip to this directory and decompresses it.
Enter the decompressed directory and copy the ELASTICSEARCH-6.4.x folder and KIBANA-6.4.x folder to the /var/lib/ambari-server/resources/stacks/HDP/3.0/services directory.
Enter the /var/lib/ambari-server/resources/stacks/HDP/3.0/services directory and rename ELASTICSEARCH-6.4.x and KIBANA-6.4.x to ELASTICSEARCH and KIBANA
2) elasticsearch-6.4.2.tar.gz and kibana-6.4.2-linux-x86_64.tar.gz
create new /var/www/html/es directories, and add kibana-6.4.2-linux-x86_64.tar.gz Upload two compressed packages with elasticsearch-6.4.2.tar.gz to this directory and enter http://192.168.31.139:8080/es through the browser to test whether you can access this directory (httpd service is required, Section 1.2.8)
3) restart ambari-server
#ambari-server restart
4.2 Installation
Click "Add Service" under "Services" on the Ambari page
Tick Elasticsearch and Kibana
Select the node where the master and server services are installed
Select the node where the slave service is installed
There is a red dot in the service configuration interface indicating that it needs to be completed.
Discovery Zen Ping Unicast Hosts parameters of elasticsearch (previously selected installed node, master)
Elasticsearch Download Url parameters of elasticsearch (installation package address)
Kibana's Elasticsearch Url parameters (es usage address, configure according to the prompts)
Server Host parameters of kibana (the node selected for installation previously)
Kibana Download Url of kibana (installation package address, see 4.1.2)
Click Next and click Deploy.
5. Problems encountered during installation and deployment
5.1 Error: Could not open client transport with JDBC Uri
The configuration information in the hadoop file core-site.xml is as follows. Restart Hadoop and start hiveserver2 and beeline again.
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
5.2 Error when starting hive service
Error message: Sys DB and Information Schema not created yet
Solution:
#cd /etc/hive/
#touch /etc/hive/sys.db.createdEnter
ambari-server and restart
#ambari-server restart
5.3 Problems with elasticesearch username group
错误信息：KeyError: u'elasticsearch'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-102.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-102.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']
solve:
You need to modify a configuration parameter in the ambari-server resource, change ignore_groupsusers_create from false to true , and then manually create users and user groups .
Specific steps:
#cd /var/lib/ambari-server/resources/scripts
#python configs.py -u admin -p admin -n cluster_es -l manager.node -t 8080 -a get -c cluster-env | grep -i ignore_groupsusers_create
#python configs.py -u admin -p admin -n cluster_es -l manager.node -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true
5.4 elasticsearch cannot import format_hdp_stack_version
错误：ImportError: cannot import name format_hdp_stack_version
solve:
first step,
#cd /var/lib/ambari-server/resources/stacks/HDP/3.0/services/ELASTICSEARCH/package/scripts
#vim params.py
Delete format_hdp_stack_version in the params.py file.
The second step is to clear the ambari-agent cache of all elasticsearch slave nodes.
#rm -rf /var/lib/ambari-agent/cache/*
Enter the ambari-server side to restart
#ambari-server restart
5.5 kibana cannot import format_hdp_stack_version
The solution is the same as above.
#cd /var/lib/ambari-server/resources/stacks/HDP/3.0/services/KIBANA/package/scripts directory.
Clear the cache, restart the ambari server and agent service , and then reinstall or try again, and the following error is encountered.
5.6 kibana cannot import format_hdp_stack_version
Error: kibana's _new__() takes at least 2 arguments(1 given)
Solution: Create a kibana user on the kibana server node (the group has actually been created by the script).
useradd -g useradd
5.7 hostname was not found in configuration dictionary
错误："/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ELASTICSEARCH/package/templates/elasticsearch.master.yml.j2", line 93, in top-level template code
action.destructive_requires_name: {
{action_destructive_requires_name}}
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!
Solution:
cd /var/lib/ambari-server/resources/stacks/HDP/3.0/services/ELASTICSEARCH/configuration
Edit elasticsearch-config.xml and find that the value of the discovery.zen.ping.unicast.hosts attribute is empty.
We filled in this parameter during installation, and fill in the value of this parameter.
<name>discovery.zen.ping.unicast.hosts</name>
<value>master</value>
cd
/var/lib/ambari-server/resources/stacks/HDP/3.0/services/ELASTICSEARCH/package/scripts
Edit params.py and found that there are two ways of writing hostname=config['hostname'].
Modify both places to:
hostname = config['configurations']['elasticsearch-config']['discovery.zen.ping.unicast.hosts']
Clear the ambari-agent cache of all elasticsearch slave nodes.
The server node restarts ambari-server restart,
the slave node restarts ambari-agent restart,
restarts the elasticsearch service, and finds that the service can be started .
5.8 master and slave stop after starting
Looking at the logs, you will find these two errors.
Error message:
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
Solution:
The maximum number of files opened simultaneously by each process is too small. You can check the current number through the following 2 commands
ulimit -Hn
ulimit -Sn
modify the /etc/security/limits.conf file and add configuration (all master and slave nodes).
Error message:
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Solution:
Modify the /etc/sysctl.conf file and increase the configuration (all master and slave nodes).
vm.max_map_count=262144
Then execute the command sysctl -p to take effect
sysctl -p
5.9 parent directory /opt/elasticsearch/master/config doesnot exist
This error occurred on the other two slave nodes.
The solution is simple, just create an empty directory.
mkdir -p /opt/elasticsearch/master/config