foreword
Finally, it's time to start writing some articles related to big data. When I really wanted to start writing my old business, I still thought about it for a long time. One is that I don’t know where to start writing, and the other is how to write something interesting.
We often say that the process is more important than the result. There are also many people who like to start doing something after they are fully prepared. But often before I started, I slowly denied my own ideas.
Technology is like life, there are always mountains that cannot be climbed. Expectations about what will happen in the future, but not expectations about what might happen in the future. All we can do is to take every step in front of us.
So, this time I decided to start writing from the process. Put forward ideas and ideas, and then implement them. After completing a part, write down the process and insights. The expected result may not be achieved in the end, but I believe that there will always be gains in the process.
platform architecture
To play big data, wherever you go, you must have a platform. The most basic Hadoop ecology HDFS, Yarn, and Hive are required, Kafka, zookeeper must also be required, redis must also be required, and Spark and Flink clients must also be required.
How to deal with so many things?
Engage in three virtual machines? I can't bear to toss my notebook.
Engage in three servers? the cost is too high.
Looking at the only CVM with 1Core 2G configuration in hand, I can't help but fall into deep thought...
facing problems
1Core 2G wants to make so many platform components, can it be done?
Doesn't a big data cluster require multiple machines for distribution?
solution
No matter how low the machine configuration is, don't consider performance for sure. If performance is a concern, upgrade the configuration with banknote capabilities or use multiple hosts.
As for the multiple machines required for distribution, use docker containerization to solve it. On this machine, an 8-node redis cluster has been used before, and the difficulty is port mapping.
For the port, redis is okay, and there are many Hadoop port machines, so it will definitely be troublesome when mapping, so we will talk about it later.
Starting pointAmbari
The first step must be to install Hadoop. I originally planned to virtualize four dockers, and then set up Hadoop with HA and it's over. However, I just thought I got it all here, so let’s build Ambari, which can not only install various platforms online, but also monitor the operation and maintenance interface. for?
So the first goal is to install Ambari.
Ambari
At that time, when I compiled Ambari, the latest version was 2.7.5, and now it is 2.7.6. The compilation steps can be followed by the official documentation. Official document address: https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.7.5
Prepare before compiling
- Install JDK, install maven
- Deploy node，npm install bower
- Install the database, I choose MySQL, and build the database and table, the command is as follows:
create database ambari default charset=utf8;
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
grant all on ambari.* to ambari@'%';
mysql> set global validate_password_policy=LOW; set global validate_password_policy=LOW; ^C
mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> set global validate_password_length=6
-> ;
- To install python2.6 or 2.7, you also need to use the setuptools module, which can be installed through the egg method below.
# 下载链接：https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea
sh setuptools-0.6c11-py2.7.egg
Step1: Download and compile
Among the five steps given in the official document, the most difficult part is to compile, and various problems will be encountered during the compilation process. I spent a week's night solving a dozen of the more difficult problems, and then it took 40 minutes to compile successfully.
My server system is Centos, so from the command given on the official website, all I have to do is compile the source code into an rpm package and install it on the server.
In actual operation, I made two modifications to the compilation command, one is to compile in the background through nohup and &, and the other is to bypass the file permission by setting -Drat.skip.
nohup mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true &
The size of the compiled source code package has also changed from 80M to 8G:
After the compilation is successful, the rpm installation package will appear in the target directory of each submodule.
Step2: Install Ambari service
As shown in the last picture of Step1, execute install to install the server.
Step3: Set up and start the Ambari service
After executing the setup, enter the setting page, mainly to set the database.
After the setting is complete, use the start command to start.
I used restart here, and the listening port is 8000 (the default is 8080).
Step4: Install Ambari Agent
Agent is deployed on the host of the cluster to monitor the service status. Here I plan to use docker as the cluster node, so this step will be synchronized with the creation and allocation of docker in the subsequent cluster construction.
Step5: Access the service
The Amabri service can be accessed through port 8080, and the default account password is admin/admin .
epilogue
It would be best if the compilation of Ambari can be passed once. I encountered problems when compiling, and some of them were not even available to Baidu. I had to figure out a way by myself, so be patient when compiling.
Later, I will sort out some problems encountered in the compilation. Many problems in the previous compilation have not been fully recorded, and I am working hard to review them.
