Distributed search engine (Elastic Search) + message queue (RabbitMQ) deployment (mall 4)

1. Distributed search engine: Elastic Search

The goal of Elastic Search is to implement search. It is a very powerful open source search engine that can help us quickly find the content we need from massive data. When the amount of data is small, we can search the data in the relational database through indexes. However, if the amount of data is large, the search efficiency will be very low. At this time, we need a distributed search engine. It provides a distributed multi-user capable full-text search engine based on a RESTful web interface. Elastic search combines kibana, Logstash, and Beats, also known as elastic stack (ELK), and is widely used in log data analysis, real-time monitoring and other fields. Elastic search is the core of the elastic stack, responsible for storing, searching, and analyzing data.
Elastic search is a Lucene-based search server. The bottom layer of Elastic search is implemented based on Lucene. Lucene is a Java language search engine class library and a top-level project of Apache Company.
The storage in ES is like this:
an index (indeces) is equivalent to a database (database), there are multiple types in each index (equivalent to table structure), and there are multiple documents (equivalent to rows) in each index. , each document consists of multiple fields (equivalent to fields).
You can think of ES as a document-oriented database. The following diagram describes the similarities between ES and relational databases:

In enterprises, Elasticsearch and mysql are often used in combination:
1. Write operations with higher security requirements are implemented using mysql.
2. Search requirements with higher query performance requirements are implemented with elasticsearch
. 3. The two are then based on A certain way to achieve data synchronization and ensure consistency.
Characteristics of Elastic Search and MySQL:
MySQL: good at transaction type operations, which can ensure data security and consistency
Elastic Search: good at search, analysis, and calculation of massive data

2. RabbitMQ

The main task of RabbitMQ is to process massive amounts of information. It is mainly used for data storage and forwarding between internal subsystems of a distributed system. This is an application in system decoupling. It is an implementation of the AMQP protocol, supports a variety of clients, can store and forward asynchronous messages from clients, and performs well in terms of ease of use, scalability, and high availability.
Features of RabbitMQ:
1. Developed based on ErLang language, it has the advantages of high availability and high concurrency, and is suitable for cluster servers;
2. Robust, stable, easy to use, cross-platform, supports multiple languages, and has complete documentation;
3. High reliability and news Confirmation mechanism and persistence mechanism;
4. Reliability: RabbitMQ supports persistence, ensuring the stability of messages;
5. High concurrency and high availability: RabbitMQ uses Erlang as the development language. Erlang is a language developed for telephone switches. It is naturally It has the aura of high concurrency and high availability; it can be used to cut traffic peaks in high-concurrency systems and temporarily store request traffic data in RabbitMQ, thereby preventing a large amount of request traffic from directly reaching the background service and overwhelming the background service.
6. Simple cluster deployment: Erlang makes RabbitMQ cluster deployment super simple;
7. RabbitMQ is a message server that implements the AMQP standard.
RabbitMQ is a popular open source message middleware developed based on the erlang language, or a message queue system. The producer of the message puts the message to be sent into the message queue, and the receiver of the message can receive the message from the server according to the forwarding mechanism configured by RabbitMQ. RabbitMQ performs message forwarding, buffering and persistence operations according to specified forwarding rules. It is mainly used for communication between multiple servers or between subsystems of a single server. RabbitMQ is the standard configuration of distributed systems.

3. Installation and deployment:

(1) ES Install Elastic Search

1. Make sure the system has jdk1.8 and above installed

2. 

3. 

4. 

5. 

6. 

7. Add user
   useradd admin

8. Unzip
   (1) tar xf elasticsearch-5.6.7.tar.gz -C /usr/local
   (2) cd /usr/local

9. Set the soft link
   ln -sv elasticsearch-5.6.7 elasticsearch
   5. Create the folder
   mkdir -pv /usr/local/elasticsearch/{data,logs}
   mkdir -p /usr/local/elasticsearch/plugins
   mkdir -p /usr/local /elasticsearch/config/scripts

10. Set permissions
    chown -R admin.admin /usr/local/elasticsearch-5.6.7

11. Modify the configuration file
    vi /usr/local/elasticsearch/config/elasticsearch.yml
    
    
    
    
    
    cluster.name: my_application

Change the node name

node.name: node-1
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs

Modify the listening address of ES so that other machines can also access it.

network.host: 0.0.0.0
http.port: 9200

Add new parameters so that the head plug-in can access es

http.cors.enabled: true
http.cors.allow-origin: “*”

12. Edit the limit.conf file
    vi /etc/security/limits.conf
    
13. Edit the sysctl.conf file /etc/sysctl.conf
    

vm.max_map_count= 262144
sysctl -p
14. Start the es service and switch to the admin user added above
(1) The instructions are as follows
cd /usr/local
chown -R admin.admin /usr/local/elasticsearch-5.6.7
su admin
( 2) Start es
cd elasticsearch/bin
./elasticsearch
16. Possible problems

16. After installing the ik word segmenter and decompressing it, the file name is changed to ik-analysis
    and placed in /usr/local/elasticsearch/plugins.
    Switch root and execute this command
    chown -R admin.admin /usr/local/elasticsearch-5.6.7
    
    

17. Start es in the background
    cd /usr/local/elasticsearch/bin
    su admin
    ./elasticsearch -d
    

(2) RabbitMQ installation

1. Upload installation package
   

2 Unzip
tar -zxvf otp_src_22.0.tar.gz
3. Remove
mv otp_src_22.0 /usr/local/
4. Change directory
cd /usr/local/otp_src_22.0/
5. Install dependencies
yum -y install gcc glibc- devel make ncurses-devel openssl-devel xmlto perl wget gtk2-devel binutils-devel

6. Create the directory to be installed
   mkdir .../erlang
7. Configure the installation path
   chmod +7 /usr/local/erlang
   ./configure --prefix=/usr/local/erlang
   
   

When prompted that the permissions are insufficient, use the following path to configure +7 permissions for root
chmod +7 /usr/local/opt_src_22.0/Erts/configure
chmod +7 /usr/local/opt_src_22.0/Make/configure
chmod +7 /usr/local /opt_src_22.0/Lib/common-test/configure
chmod +7 /usr/local/opt_src_22.0/Lib/crypto/configure
chmod +7 /usr/local/opt_src_22.0/Lib/erl-interface/configure
chmod + 7 /usr/local/opt_src_22.0/Lib/megaco/configure
chmod +7 /usr/local/opt_src_22.0/Lib/odbc/configure
chmod +7 /usr/local/opt_src_22.0/Lib/snmp/configure
chmod +7 /usr/local/opt_src_22.0/Lib/wx/configure

8. install
   make install
   
   
   

9. Check if the installation is successfulll
   /usr/local/erlang/bin

10. Add environment variable
    echo 'export PATH=$PATH:/usr/local/erlang/bin' >> /etc/profile
    
    
    
    
    

11. Refresh environment variables
    source /etc/profile

12. Execute erl (execute in the installation directory)

13. 

Execute through valid environment variables (equivalent to executing .exe or .com .bat after having the DOS system search path)

14. Halt(). command exits

15. Since it is in tar.xz format, you need to use xz. If not, install
    yum install -y xz first.
    

16. 
    Unzip rabbitmq-server-generic-unix. Unzip tar /bin/xz -d rabbitmq-server-generic-unix-3.7.15.tar.xz for the first time.

17. Unzip
    tar for the second time -xvf rabbitmq-server-generic-unix-3.7.15.tar

18. Remove
    mv rabbitmq_server-3.7.15/ /usr/local/
    cd /usr/local/

19. Rename
    mv /usr/local/rabbitmq_server-3.7.15 rabbitmq

20. Configure the environment variable
    echo 'export PATH=$PATH:/usr/local/rabbitmq/sbin' >> /etc/profile

21. Refresh environment variables
    source /etc/profile
    
    

22. Start the stopped state
    rabbitmq-server –detached
    

rabbitmqctl stop

rabbitmqctl status

23. Enable web plug-in
    rabbitmq-plugins enable rabbitmq_management
    

24. Add a user
    rabbitmqctl add_user admin yuanfeng021
    

25. Configure permissions
    rabbitmqctl set_permissions -p "/" admin ". " ". " ".*"
    

    rabbitmqctl set_user_tags admin administrator
    
    
    26. Visit on CentOS8 machine: http://XXX.XXX.XXX:15672/ admin yuanfeng021
    
    

27. At this time, it is found that it cannot be accessed on the LAN, so
(1) check the port number opened on CentOS

It can be seen that port 15672 has not been opened
(2) Use the command firewall-cmd –add-port=xxx/tcp to open the three ports 5672, 15671, and

15672. If you want to permanently open port 15672, use the command firewall-cmd - -permanet --add-port=15672/tcp
(3) Modify the rabbitmq configuration file, rabbitmq-env

Add a line with the following content: (uppercase)
NODE_IP_ADDRESS=192.168.0.132 (the IP address of the CentOS network card where you installed rabbitmq server)

(4) Test the 15672 port number of rabbitmq-server through telnet from the computer on the LAN

The test results are as follows: It means that the 15672 port is opened normally (the cursor in the upper left corner flashes, but there is no prompt to connect the port failure)

28. Accessing rabbitmq-server from the LAN is normal, as shown below:

The access is successful, and the deployment of distributed search engine (Elastic Search) + message queue (RabbitMQ) is completed. Please criticize and correct any deficiencies.