ambari 2.7 hadoop 3.0 安装相关文档

参考官方文档逐步安装部暑，ambari 2.7 上 部署hadoop 3.0，一些错误解决笑记

Ambari 2.7 installation guide 

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html

网络相关配置

互信

ssh-keygen  // 一路回车生成公钥

ssh-copy-id <target host> //复制到目标主机

systemctl enable ntpd  //启动ntp

systemctl disable firewalld //关闭防火墙

service firewalld stop 

setenforce 0

echo umask 0022 >> /etc/profile

org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host

https://stackoverflow.com/questions/25641047/org-postgresql-util-psqlexception-fatal-no-pg-hba-conf-entry-for-host

YARN Registry DNS Start failed

ERROR dns.PrivilegedRegistryDNSStarter (PrivilegedRegistryDNSStarter.java:init(61)) - Error initializing Registry DNS
java.net.BindException: Problem binding to [hadoop-app1.novalocal:53] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:736)
	at org.apache.hadoop.registry.server.dns.RegistryDNS.openUDPChannel(RegistryDNS.java:1016)
	at org.apache.hadoop.registry.server.dns.RegistryDNS.addNIOUDP(RegistryDNS.java:925)
	at org.apache.hadoop.registry.server.dns.RegistryDNS.initializeChannels(RegistryDNS.java:196)
	at 

解决方法——————
参考：https://community.hortonworks.com/questions/224670/yarn-registry-dns-start-failed-hortonworks-3.html

YARN 设置命令 https://yarnpkg.com/lang/en/docs/cli/config/

错误原因：

安装YARN的HOST上，PORT 53 已被占用

可以使用 netstat -tnlpa | grep 53 命令查看

[root@hadoop-app1 ~]# netstat -tnlpa | grep 53
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1307/dnsmasq        
tcp        0      0 10.0.14.134:35534       10.0.14.134:53          TIME_WAIT   -                   
tcp        0      0 10.0.14.134:52538       10.0.10.94:22           TIME_WAIT   -  

进入YARN 设置文件重新设置端口 vi /usr/hdp/3.0.1.0-187/hadoop-yarn/conf/yarn-site.xml 

修改保存后用命令启动 registrydns yarn --daemon start registrydns

    <property>
      <name>hadoop.registry.dns.bind-port</name>
      <value>53</value> //改成其他端口如 54
      <hidden>true</hidden>
    </property>

然后在ambari UI 管理 yarn - start service 

这个操作有个头疼的问题，如果在ambari web选择 restart service似乎端口又会改成53,然后。。。又failed

所以把53端口的进程KILL了，再启动，似乎更实用点。。

参考：https://community.hortonworks.com/questions/222145/yarn-dns-registry-server-not-started-in-hortonwork.html

[root@hadoop-app1 bin]# sudo lsof -i -P -n |grep 53
java        680      atlas  305u  IPv6 765376      0t0  TCP 10.0.14.134:48446->10.0.11.117:2181 (ESTABLISHED)
avahi-dae  2123      avahi   12u  IPv4  18340      0t0  UDP *:5353 
dnsmasq    2950     nobody    5u  IPv4  29192      0t0  UDP 192.168.122.1:53 
dnsmasq    2950     nobody    6u  IPv4  29193      0t0  TCP 192.168.122.1:53 (LISTEN)
java       8454      druid  535u  IPv6 799296      0t0  TCP 10.0.14.134:49518->10.0.11.117:2181 (ESTABLISHED)
java       8939      druid  589u  IPv6 904539      0t0  TCP 10.0.14.134:8081->10.0.14.134:45014 (ESTABLISHED)
java       8939      druid  593u  IPv6 905141      0t0  TCP 10.0.14.134:8081->10.0.14.134:45364 (ESTABLISHED)
java       9792      druid  427u  IPv6 917355      0t0  TCP 10.0.14.134:45364->10.0.14.134:8081 (ESTABLISHED)
java      12382       hdfs  351u  IPv4 918492      0t0  TCP 10.0.14.134:8020->10.0.14.115:53366 (ESTABLISHED)
java      12382       hdfs  396u  IPv4 918489      0t0  TCP 10.0.14.134:8020-
....
[root@hadoop-app1 bin]# kill -9 2950  // 把那个nobody 又占用端口53的PID2958 kill 了

启动后YARN不再报错

==============================================

accumulo 启动失败 var/log/accumulo/accumulo-init.err' returned 255

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 75, in status
    check_process_status(pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

解决参考：https://community.hortonworks.com/articles/79327/re-initializing-apache-accumulo-under-hdp.html 

确保 ZOOKEEPER 已启动 

在accumulo 安装HOST上运行

sudo -u hdfs hdfs dfs -rm -R /apps/accumulo/data

sudo -u accumulo ACCUMULO_CONF_DIR=/etc/accumulo/conf/server accumulo init --instance-name hdp-accumulo-instance --clear-instance-name

==============================================

mysql-jdbc 安装

1. sudo yum install mysql-connector-java*
2. ls -al /usr/share/java/mysql-connector-java.jar
3. cd /var/lib/ambari-server/resources/

ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar

Configuring MySQL for Ranger

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/bi_install_ranger_db.html

==============================================

KYLIN 镜像地址

http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.2/