ubuntu14 install hadoop using hdp

1. Preparation
1) Install java
copy jdk-7u80-linux-x64.tar.gz to this machine, extract /test
vi /etc/profile
export JAVA_HOME=/test/jdk1.7.0_80
export CLASSPATH=.:$JAVA_HOME/lib :$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
source /etc/profile
2) Change hostname
vi /etc/hostname
vi /etc/hosts
127.0.1.1 Change hdp01.test to 173.168.0.121 hdp01.test 
173.168.0.121 hdp01.test 173.168.0.122 hdp02.test 3) sudo apt-get install ntp service ntp start 4) Install postgres, mysql Step 1: Install PostgreSQL, phpPgAdmin and Apache2





sudo apt-get -y install postgresql postgresql-contrib phppgadmin


Step 2: Configure PostgreSQL user
su - postgres
psql
\password
\q


Step 3: Configure Postgresql
vim /etc/postgresql/9.1/main/postgresql.conf


 
Install MySQL on Ubuntu Very simple and can be done with just a few commands.
sudo apt-get install mysql-server
sudo apt-get install mysql-client libmysql-java
sudo apt-get install libmysqlclient-dev
sudo apt-get install libmysql-java


2. Switch to root, set up ssh
1) Install ssh
dpkg -- list | grep ssh
sudo apt-get install openssh-server
2) Generate rsa keys
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3) Change root, ssh and file permissions
chmod o+w ~/ 
chmod 700 -R ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 /root
4) Change sshd_config
vi /etc/ssh/sshd_config
to /etc/ Change the PermitRootLogin prohibit-password in the ssh/sshd_config file to PermitRootLogin yes (or delete the PermitRootLogin prohibit-password, for safety and backup, just comment it out)
sudo /etc/init.d/ssh restart
5)
vi ~/ .ssh/config
host 173.168.0.121
user [YOUR_USER_NAME]
6) On the slave machine: mkdir .ssh
in the root directory to copy the id_rsa.pub on the master to .ssh cat ~/.ssh/id_rsa.pub >> ~/. ssh/authorized_keys chmod o+w ~/  chmod 700 -R ~/.ssh chmod 600 ~/.ssh/authorized_keys






chmod 755 /root


3. Download the package from the following three locations, you can download
nohup wget -c http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/ directly on windows with http 2.2.1.0/ambari-2.2.1.0-ubuntu14.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates /2.4.0.0/HDP-2.4.0.0-ubuntu14-deb.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1. 0.20/repos/ubuntu14/HDP-UTILS-1.1.0.20-ubuntu14.tar.gz > 2.log 2>&1 &
1) copy these three files to /var/www/html/hadoop
cd /var/www/html
mkdir hadoop
2) Run the http service and extract the three files to /var/www/html/hadoop


4. Add the official source to other machines on the intranet
sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list -O /etc/apt/sources.list.d/ambari. list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
5. Modify the official source configuration file (dist parent directory)
sudo vi /etc/apt/sources.list.d/ambari.list


#VERSION_NUMBER =2.2.1.0-161
#deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0 Ambari main


deb http://173.168.0.121/hadoop/AMBARI- 2.2.1.0/ubuntu14/2.2.1.0-161/ Ambari main


6. Update source and install program
sudo apt-get update
master: sudo apt-get install ambari-server
slave: sudo apt-get install ambari-agent


7. Configure ambari Service
1) sudo ambari-server setup
Enter the value of JAVA_HOME, other default
2) ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql.jar


8. Start the service
master: sudo ambari-server start
slave: sudo ambari-agent start
sudo vi /etc/ambari-agent/conf/ambari-agent.ini 
hostname=hdp01.test


9. 配置hive，ambari数据库
1）
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
CREATE USER 'hive'@'hdp01.test'IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE hive;


CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
CREATE DATABASE ambari;
USE ambari;
SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;


CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost';
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
CREATE USER 'oozie'@'hdp01.test'IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE oozie;


2）
sudo vi /etc/mysql/my.cnf 
添加'#'注释掉其中的"bind-address = 127.0.0.1"
sudo service mysql restart


10.Access the web interface using the default admin/admin login.
http://ip:8080

1) Click the icon in the upper left corner or select Launch Install Wizard
2) In the second step, Select Stack, in the place where the source is selected, fill in the path (dist parent directory)
http://173.168.0.121/hadoop/HDP/ubuntu14/2.x/ updates/2.4.0.0/
http://173.168.0.121/hadoop/HDP-UTILS-1.1.0.20/repos/ubuntu14/


11. In step 3


1) just fill in hdp01.test
sudo cat id_rsa copy
the private key to the box
2) Fill in hdp02.test and
choose to install ambari-agent manually
. After seeing both sucess, click Next .


12. Disable HDP (Transparent HugePages)
vi /etc /default/grub, add option transparent_hugepage=never
grub-mkconfig -o /boot/grub/grub.cfg to GRUB_CMDLINE_LINUX

reboot

13. hive and oozie select the existing mysql database and set the database password