[Spring Clound] Nacos high-availability cluster construction and use

1. Introduction to Nacos

Nacos is dedicated to helping you discover, configure and manage microservices. Nacos provides a set of easy-to-use feature sets to help you quickly realize dynamic service discovery, service configuration, service metadata, and traffic management. Nacos helps you build, deliver and manage microservice platforms more agilely and easily. Nacos is a service infrastructure for building a "service"-centric modern application architecture (such as microservice paradigm, cloud native paradigm).

nacos official website: https://nacos.io/zh-cn/docs/what-is-nacos.html

2. Nacos installation

2.1. Nacos environment dependency

Nacos is developed based on java, and its operation depends on the 64 bit JDK 1.8+ environment.

Go to the official website to download JDK: Java Downloads | Oracle

2.2. Nacos server installation

Since Nacos is open source, you can download the source code from github and compile it for installation, or you can download the official compiled installation package directly from github for installation. This article directly uses the official compiled package for installation, based on the latest official release2.0.3 version. Do not use versions below 2.x, as there have been loopholes before. Compared with the 1.x version, the performance of the 2.x version is also improved by 10 times.

Download address: https://github.com/alibaba/nacos/releases

3. Nacos deployment

3.1. Single instance deployment

Single-instance deployment is not suitable for production environments, and a single point of failure is fatal.

Linux single instance non-cluster mode startup command:

startup.sh -m standalone

Linux single instance non-cluster mode shutdown command:

shutdown.sh

Visit the nacos management page, and initialize the username and password as nacos

Access address: http://127.0.0.1:8848/nacos/index.html

The port is 8848 by default, and the ip needs to be replaced with the ip of the deployment server instance

3.2. Cluster deployment

3.2.1. Cluster Architecture

• High availability Nginx cluster
• Nacos cluster (at least three instances)
• High availability database cluster (replaces Nacos embedded database)

3.2.2. Simulated deployment

Environmental information:

system version	Machine IP	deploy application	App version
CentOS8.5	192.168.230.1	Nginx	nginx-1.18.0
CentOS8.5	192.168.230.129	Nacos	2.0.3
CentOS8.5	192.168.230.130	Nacos	2.0.3
CentOS8.5	192.168.230.131	Nacos	2.0.3
CentOS8.5	192.168.230.1	MySQL	5.7.32

Use the vmware virtual machine to simulate the above different machine environments.

Build steps:

1. Initialize the nacos database

Unzip the downloaded installation package nacos-server-2.0.3.tar.gz to the /usr/local directory, and find the nacos-mysql.sql script under /nacos/conf

Create nacos library in MySQL instance and execute sql script

Modify the Nacos configuration file to point to the MySQL instance and replace its embedded database

Find the following configuration in application.properties, the configuration is commented out by default, just cancel the comment, modify the database information to the actual database information and save it. Other nacos service instances also need to make the same modification

In order to achieve high availability, there are usually multiple MySQL database instances, and the configuration file of nacos also needs to specify the information of each MySQL instance, for example:

2. cluster configuration

Find the cluster.conf.example file under /nacos/conf and rename it to cluster.conf

Modify the cluster.conf file, delete the default ip information in it, and add the ip of each nacos instance in the nacos cluster. All nacos instances in the cluster need to do the above cluster configuration, so far the configuration of nacos is over, you can try to start three nacos instances in cluster mode.

Switch to the /nacos/bin directory and execute ./startup.sh & tail -f /usr/local/nacos/logs/start.out

Start nacos, and check the startup log, the following prompt appears to indicate that the startup is successful

At this point, you can access the console of each nacos instance through a browser, for example: http://192.168.230.131:8848/nacos

Note: If the three instances start normally in cluster mode, then accessing the management pages of the three instances respectively will display the above login page. If it cannot be accessed, the firewall may not open the port of nacos service, you can execute the following command.

[root@localhost bin]# firewall-cmd --add-port=8848/tcp --permanent
 
success
 
[root@localhost bin]# firewall-cmd --reload
 
success

If the nacos startup report has been abnormal, there are three reasons:

Reason 1: The MySQL database cannot be connected, usually because the host where the MySQL database is located does not open port 3306. Execute the following command on the MySQL host

[root@localhost bin]# firewall-cmd --add-port=3306/tcp --permanent
success
[root@localhost bin]# firewall-cmd --reload
Success

Reason 2: Since the default database connection timeout setting in the nacos configuration file application.properties is relatively short, as shown in the figure below, due to network delays and other reasons, MySQL may connect to timeout and cause nacos to start an error, so you only need to set the timeout to a long time just a few

Reason 3: The memory of the virtual machine is insufficient. When creating a virtual machine in vmvare, only 1G of memory is allocated to each virtual machine. From the startup script startup.sh of nacos, we can see that when nacos starts in cluster mode, the java heap allocated by default The memory space is 2G, so it can be judged that the nacos startup error is caused by insufficient memory of the virtual machine. After changing the memory of the virtual machine to 2G, it can start normally.

3. Nginx configuration

Modify the Nginx configuration file nginx.conf as follows:

http {
    
    
 
    include       mime.types;
 
    default_type  application/octet-stream;
 
    sendfile        on;
 
    keepalive_timeout  65;
 
    #nacos集群负载均衡
    upstream nacos-cluster {
    
    
        server 192.168.230.129:8848;
        server 192.168.230.130:8848;
        server 192.168.230.131:8848;
    }
 
    server {
    
    
        listen       80;
        server_name  192.168.230.1;
        location / {
    
    
            #root   html;
            #index  index.html index.htm;
            proxy_pass http://nacos-cluster;
        }
 
        error_page   500 502 503 504  /50x.html;
 
        location = /50x.html {
    
    
            root   html;
        }
    }
}

4. access console

Enter the ip of the host where the Nginx service is located in the browser to access:

http://192.168.230.1/nacos/#/login

Default username/password: nacos/nacos

Switch to [Cluster Management] - [Node List] to view nacos cluster node information:

4. Microservice Integration Nacos

4.1. Dependent component version selection

Since the compatibility requirements between Spring Cloud Alibaba and Spring Boot and Spring Cloud versions are very strict, if you rely on the wrong version, it is likely to cause an error when the service starts and cause the startup to fail. However, Alibaba officials have sorted out the recommended version dependencies for us:

Since the nacos we use is the latest version 2.0.3, from the above table, we can find that the version of the dependent Spring Cloud Alibaba is 2.2.7.RELEASE. Continue to query the graduation version dependency table according to the version of Spring Cloud Alibaba, as shown in the figure below, you can find that the dependent Spring Cloud version is Hoxton.SR12, and the dependent Spring Boot version is 2.3.12.RELEASE

Therefore, through the above query, it is finally determined that the microservice dependency version selection is as follows:

Spring Cloud Version	Spring Cloud Alibaba Version	Spring Boot Version	Nacos Version
Hoxton.SR12	2.2.7.RELEASE	2.3.12.RELEASE	2.0.3

Official version description: https://github.com/alibaba/spring-cloud-alibaba/wiki/%E7%89%88%E6%9C%AC%E8%AF%B4%E6%98%8E

4.2. Registration Center

First create a demo-project-parent parent project, and define the dependent components and versions in the parent pom file according to the version determined in 4.1 Dependent component version selection, as follows:

<properties>
  <java.version>1.8</java.version>
  <spring-boot.version>2.3.12.RELEASE</spring-boot.version>
  <spring-cloud.version>Hoxton.SR12</spring-cloud.version>
  <spring-cloud-alibaba.version>2.2.7.RELEASE</spring-cloud-alibaba.version>
</properties>
 
<dependencyManagement>
  <dependencies>
    <!-- spring boot 依赖 -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring-boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    
    <!-- spring cloud 依赖 -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring-cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
 
    <!-- spring cloud alibaba 依赖 -->
    <dependency>
      <groupId>com.alibaba.cloud</groupId>
      <artifactId>spring-cloud-alibaba-dependencies</artifactId>
      <version>${spring-cloud-alibaba.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

4.2.1. Service provider

Create a new service provider submodule micro-service-provider under the parent project, and introduce dependencies in the pom file:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
 
<dependency>
  <groupId>com.alibaba.cloud</groupId>
  <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
</dependency>

Modify the provider configuration file application.yml:

server:
  port: 8081
 
spring:
  application:
    name: service-provider
  cloud:
    nacos:
      discovery:
	#nacos集群配置(Nginx)
        server-addr: 192.168.230.1:80

Provider startup class:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
 
@SpringBootApplication
@EnableDiscoveryClient
public class ProviderApplication {
    
    
 
    public static void main(String[] args) {
    
    
        SpringApplication.run(ProviderApplication.class, args);
    }
 
}
Provider测试Controller类：

import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
 
@RestController
public class ProviderController {
    
    
 
    @Value("${server.port}")
    String port;
 
    @GetMapping("/hi")
    public String hi(@RequestParam(value = "name", defaultValue = "zmx",required = false) String name) {
    
    
        return "hello " + name + ", i'm provider, my port:" + port;
    }
 
}

4.2.2. Service consumers

Create a new service consumer submodule micro-service-consumer under the parent project, and introduce the following dependencies in the pom file:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
 
<dependency>
  <groupId>com.alibaba.cloud</groupId>
  <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
</dependency>
 
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
 
<!-- 需要注意的是引入openfeign，必须要引入loadbalancer，否则无法启动。-->
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

Modify the consumer configuration file:

server:
  port: 8082
 
spring:
  application:
    name: service-consumer
  cloud:
    nacos:
      discovery:
        #nacos集群配置(Nginx)
        server-addr: 192.168.230.1:80

Add the @EnableFeignClient annotation to the startup class of the project to enable the FeignClient function:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
 
@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
public class ConsumerApplication {
    
    
 
    public static void main(String[] args) {
    
    
        SpringApplication.run(ConsumerApplication.class, args);
    }
 
}

Write a feign client, FeignClient, to call the interface of the provider service:

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
 
//此处的value值对应于provider服务application配置文件中的spring.application.name
@FeignClient(value = "service-provider" )
public interface ProviderClient {
    
    
 
    @GetMapping("/hi")
    String hi(@RequestParam(value = "name", defaultValue = "zmx", required = false) String name);
 
}

Write a test interface and let the consumer call the interface of the provider service:

import net.myibc.client.ProviderClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
 
@RestController
public class ConsumerController {
    
    
 
    @Autowired
    ProviderClient providerClient;
 
    @GetMapping("/hi-feign")
    public String hiFeign(){
    
    
        return providerClient.hi("feign");
    }
}

Start the provider and consumer projects, and view them on the [Service Management]-[Service List] page of nacos. It can be seen that both services have been registered successfully:

Note: After starting all the services here, you may only be able to view a certain service in the console, or you may not be able to find the registered service. Check the nacos.log log and find the following exception:

The reason for this exception: when the nacos client is upgraded to version 2.x, the communication method of gRPC is added, and two ports are added. These two ports are automatically generated with a certain offset on the original port of nacos (8848 by default).

Description of the offset between the port and the main port:

9848: client gRPC request server port, used by the client to initiate a connection and request to the server

9849: The server gRPC requests the server port, which is used for synchronization between services, etc.

7848: Nacos cluster communication port, used for election, detection, etc. between Nacos clusters

Therefore, it is not difficult to guess that the reason for this problem is that the two ports newly added in nacos2. Unable to view all service registration instance information. Just execute the following command on the nacos host:

[root@localhost bin]# firewall-cmd --add-port=9848/tcp --permanent
success
[root@localhost bin]# firewall-cmd --add-port=9849/tcp --permanent
success
[root@localhost bin]# firewall-cmd --add-port=7848/tcp --permanent
success
[root@localhost bin]# firewall-cmd --reload
Success

Note: If the VIP/nginx proxy cluster is used, the TCP request forwarding of the two ports 9848 and 9849 needs to be configured in nginx, otherwise the client service cannot be connected to the two ports of the server when it starts, which will cause the startup to fail . If you do not use any proxy, after the host executes the above command to enable the port whitelist, you only need to configure the ip and port of the nodes of the naocs cluster in the client service application configuration file. The configuration is as follows:

Nginx configures nacos TCP forwarding configuration:

#配置nacos TCP转发
stream {
    
    
    upstream nacos1 {
    
    
        server 192.168.230.129:9848;
        server 192.168.230.130:9848;
        server 192.168.230.131:9848;
    }
 
    server {
    
    
        listen 9848;
        proxy_pass nacos1;
    }
 
    upstream nacos2 {
    
    
        server 192.168.230.129:9849;
        server 192.168.230.130:9849;
        server 192.168.230.131:9849;
    }
 
    server {
    
    
        listen 9849;
        proxy_pass nacos2;
    }
}

4.2.3. Service call

Enter http://127.0.0.1:8082/hi-feign on the browser, and the browser returns a response:

hello feign, i'm provider ,my port:8081

It can be seen that the browser's request successfully calls the interface of the consumer service, and the consumer service also successfully calls the interface of the provider service through feign.

4.2.4, load balancing

Feign uses Spring Cloud Loadbanlancer as a load balancer. You can modify the port of the provider and start a new provider service locally, then there are two provider services locally, and the ports are 8081 and 8083 respectively. Call http://127.0.0.1:8082/hi-feign multiple times on the browser, the browser will alternately display:

hello feign, i'm provider ,my port:8081
hello feign, i'm provider ,my port:8083
At this time, the number of console provider instances will become 2:

4.3. Configuration Center

4.3.1, configuration center client

Create a new nacos configuration client submodule nacos-config-client under the parent project, and introduce the following dependencies in the pom file:

<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
</dependency>

Configure the following content in the bootstrap.yml file (must be the bootstrap.yml file, not the application.yml file):

spring:
  application:
    name: nacos-config-client
  cloud:
    nacos:
      config:
        server-addr: 192.168.230.1:80
        file-extension: yaml
        prefix: nacos-config-client
  profiles:
    active: dev

In the above configuration, the address of the nacos config server is configured, and the extension of the configuration is ymal (currently only ymal and properties are supported). Note that server.port is not configured, and the attributes of server.port are configured in nacos. The above configuration corresponds to the format of dataId in Nacos. The complete format of nacos is as follows:

${
    
    prefix}-${
    
    spring.profile.active}.${
    
    file-extension}

• prefix defaults to the value of spring.application.name, and can also be configured through the configuration item spring.cloud.nacos.config.prefix.
• spring.profile.active is the profile corresponding to the current environment. For details, please refer to the Spring Boot documentation. Note: When spring.profile.active is empty, the corresponding connector - will also not exist, and the splicing format of dataId becomes $prefix.${file-extension}
• file-exetension is the data format of the configuration content, which can be configured through the configuration item spring.cloud.nacos.config.file-extension. Currently only properties and yaml types are supported.

Start nacos, log in to the console http://192.168.230.1/nacos, create a Data ID, and the complete configuration is shown in the figure:

Write a RestController and add the @RefreshScope annotation on the Controller to implement hot loading of the configuration. This annotation will only take effect when the class that needs to dynamically update the configuration is loaded. If this annotation is added to the startup class, the dynamic update in other classes will not take effect. The code is as follows:

import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
 
@RestController
@RefreshScope
public class ConfigController {
    
    
 
    @Value("${username:lily}")
    private String username;
 
    @RequestMapping("/username")
    public String get() {
    
    
        return username;
    }
}

Start the project nacos-provider, visit http://127.0.0.1:8084/username on the browser, and return the username value zhangsan3 configured on the nacos console. Change the configuration of username to lisi on the nacos console, and revisit http://127.0.0.1:8084/username without restarting the nacos-provider project, and return the modified value lisi, which can be seen as the configuration of nacos The center implements the hot loading function.

Return before modifying the username configuration in nacos:

Change the value of username in nacos to lisi and request to return again:

Note: When using some high-version SpringCloud components, the bootstrap.yml configuration file may not be found when starting the nacos configuration client service, and the following error will be reported. You only need to configure the environment variables in the system environment variables. Available: spring.cloud.bootstrap.enabled=true

4.3.2. Enable authority authentication

Usually, for the sake of security, the client service needs to pass the user name and password authentication when accessing nacos. Nacos does not enable authentication by default. If you need to enable authentication, you first need to edit the /nacos/conf/application.properties configuration file , Find the nacos.core.auth.enabled=false configuration item, as shown in the figure below, the default is false, which means that the authorization authentication is turned off, and the authentication can be enabled by changing it to true. After the modification is complete, all nacos instances need to be restarted.

Configure the nacos user name and password in the application configuration file, as shown in the figure below, first configure the service user name and password as a user that does not exist, and start the service to verify whether the permission configuration takes effect:

After starting the service, the following exceptions were reported, code=403 and unknown user, indicating that the permission verification failed:

After changing the username and password to the correct ones, start the service again, and you can find that the service starts normally:

So far, it shows that the permission authentication configuration of nacos has taken effect.