With the rapid development of microservices and cloud computing in recent years, machines have gradually changed from physical machines to virtual machines. Application services have gradually changed from a huge single application to an application cluster composed of several microservices. The traditional deployment model can no longer meet the daily update needs of development, and a management architecture suitable for microservices is required.

Technology stack and documentation

Resource scheduling framework MESOS

Application Orchestration Platform Marathon

nginx dynamically modify upstream dyups

nginx dynamically modify upstream upsync

Use Mesos for machine resource management

First, it is the management of machine resources. In the microservice architecture, the original single service is split into independent unit applications. These services are small in size and can run independently on machines with smaller configurations. For fault isolation, we will deploy these services on different virtual machines as much as possible, so that the number of machines will increase exponentially. For operation and maintenance, when each new service is deployed, it is necessary to first check whether the remaining resources of the existing machine meet the needs of the new service. Sometimes due to inaccurate evaluation, it may cause back and forth expansion, migration, or waste of resources.

In the beginning, our architecture might be like this

1.png

In order to solve the above problems, you can use MESOS (distributed resource management framework), which allows us to use the entire data center like a computer (a resource pool).
Mesos is divided into two roles, master and agent, during deployment. Of course, you can start them on the same machine.

You need to install zookeeper before installing Mesos. Mesos uses zk to achieve high availability and election, including a master leader and several backup masters to avoid downtime.

Mesos master is responsible for managing each Framework and Slave, and non-allocation of resources on Slave to each Framework.
Mesos agent is responsible for managing each Mesos Task on this node and allocating resources for each Executor (the lower version is mesos-slave).

$ cat > /tmp/bintray-mesos-el.repo <<EOF
#bintray-mesos-el - packages by mesos from Bintray
[bintray-mesos-el]
name=bintray-mesos-el
baseurl=https://dl.bintray.com/apache/mesos/el7/x86_64
gpgcheck=0
repo_gpgcheck=0
enabled=1
EOF

$ sudo mv /tmp/bintray-mesos-el.repo /etc/yum.repos.d/bintray-mesos-el.repo

$ sudo yum update

$ sudo yum install mesos

$ tree /etc/mesos-master
/etc/mesos-master/
|-- hostname
|-- ip
|-- log_dir
|-- quorum # quorum > (number of masters)/2
`-- work_dir

$ tree /etc/mesos-agent
/etc/mesos-agent/
|-- containerizers # Container type, default mesos, you can add docker, such as: mesos,docker
|-- hostname
|-- ip
|-- log_dir
|-- master # master address, the format is host:port or
zk://host1:port1,host2:port2,.../path or file:///path/to/file
|-- resources # Set the total resource size, you can Set smaller to reserve more machine resources`--
work_dir

$ cat /etc/mesos/zk # Set the storage directory of mesos in zk
zk://192.168.100.9:2181,192.168.100.110:2181,192.168.100.234:2181/mesos

$ systemctl start mesos-master
$ systemctl start mesos-slave
When the mesos service is started, the agent will report machine resources to the master node, including CPU, memory, disk, etc. When we want to publish a service, we only need to set the CPU, memory, and disk parameters of the service, and the mesos master will automatically help us choose a machine with sufficient resources to run, as shown below

2.png

We leave the startup of microservices to Mesos management, so that we only need to focus on the overall resources. MESOS provides a UI interface, you can directly access port 5050 of the mesos master to view the cluster resource usage. Overall usage and Agent node usage

3.png

4.png

After completing the above, our architecture becomes like this

5.png

Use Marathon for microservice management

Marathon is a private PaaS platform built on Mesos. It can automatically handle hardware or software failures and ensure that each application is "always online". We use Marathon to manage microservices with the following advantages: 1. It supports containers and non-containers, and is not limited by service startup type, operating system version, etc. 2. Beautiful and powerful user interface, which allows quick and convenient application configuration on the UI. 3. Supports constraints, such as allowing a mesos agent node to run only one application. 4. Support health check. You can configure monitoring and checking of http, https, tcp, and command types. 5. Complete REST API, easy to integrate and write scripts. This is essential for later integration.

Add the repository

$ sudo rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-2.noarch.rpm

Install packages

$ sudo yum -y install mesos marathon

marathon and mesos zk path

$ cat /etc/default/marathon
MARATHON_MESOS_USER="root"
MARATHON_MASTER="zk://192.168.100.9:2181,192.168.100.110:2181,192.168.100.234:2181/mesos"
MARATHON_ZK="zk://192.168.200.9:1181,192.168.100.110:2181,192.168.100.234:2181/marathon"

After systemctl start marathon is
started, you can directly access port 8080 of marathon to see a beautiful and powerful UI interface.

6.png

Let's take the springboot application as an example to create an application on marathon

7.png

8.png

9.png

10.png

When we update the application, marathon will create a new application with the same number of instances, and replace the old node after the health check is passed, so there is no need to worry about online accidents caused by the shutdown of the old service while the new service is not started. So far, we can easily and quickly create, upgrade, expand, and shrink daily applications on marathon. When the service health check fails or the machine goes down, marathon will automatically start the hung application on other nodes, which greatly improves high availability.

11.png

Use nginx upsync/dyups module for smooth changes

When our microservices can be randomly allocated on different machines, a new headache arises. Nginx does not know the change of the back-end node, and it is impossible to manually modify the upstream node and reload nginx every time, so the cost is too high. Our solution is to connect with the microservice registry. When the service is registered or unregistered, the registry will be updated. The nginx upsync/dyups module can be used to dynamically modify the ability of upstream nodes to synchronize and achieve smooth changes. If the registration center used is consul, it is recommended to use the upsync module, so that no development is required, and only a simple nginx configuration is needed to achieve the effect we want. It supports consul kv, consul_services, consul_health, and upsync also supports etcd. It is recommended to use the consul_health interface. The upsync module is not a built-in module of nginx. You need to recompile and add this module when you use it.

wget 'http://nginx.org/download/nginx-1.8.0.tar.gz'
tar -xzvf nginx-1.8.0.tar.gz
cd nginx-1.8.0/

./configure --add-module=/path/to/nginx-upsync-module
make
make install
configuration file example

http {
upstream test {
upsync 127.0.0.1:8500/v1/health/service/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul_health strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;

    include /usr/local/nginx/conf/servers/servers_test.conf;
}

upstream bar {
    server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3;
}

server {
    listen 8080;

    location = /proxy_test {
        proxy_pass http://test;
    }

    location = /bar {
        proxy_pass http://bar;
    }

    location = /upstream_show {
        upstream_show;
    }

}

}
When upsync cannot meet our needs or the registry is not consul or etcd, we can consider using the nginx dyups module. Dyups only provides upstream additions, deletions, and modification interfaces, and the comparison and modification with the registry requires us to complete the work through scripts. Although this method is more troublesome, it is highly customizable and supports http, C, and lua API, which can basically meet the needs of most scenarios.

The dyups module also needs to be added when nginx is compiled

$ git clone git: //github.com/yzprofile/ngx_http_dyups_module.git

to compile as a static module

$ ./configure --add-module=./ngx_http_dyups_module

to compile as a dynamic module

$ ./configure --add-dynamic-module=./ngx_http_dyups_module
示例配置

http {

include conf/upstream.conf;

server {
    listen   8080;

    location / {
        # The upstream here must be a nginx variable
        proxy_pass http://$dyups_host;
    }
}

server {
    listen 8088;
    location / {
        return 200 "8088";
    }
}

server {
    listen 8089;
    location / {
        return 200 "8089";
    }
}

server {
    listen 8081;
    location / {
        dyups_interface;
    }
}

}
Special attention, when using dyups, the upstream of proxy_pass must be the nginx variable, otherwise it will not take effect, remember.

14.png

Overall review

After the above adjustments, we got the following optimizations

Server resources are automatically allocated and used rationally
Improve the high availability of microservices
Reduce OPS labor costs, easier to manage and maintain

Build a high-availability architecture suitable for microservices

Add the repository

Install packages

marathon and mesos zk path

to compile as a static module

to compile as a dynamic module

Guess you like