1-- Nginx server] [High Performance - 8 HTTP Dynamic Load Balancing

1 What is the dynamic load balancing

The traditional load balancing, if the Upstreamparameters change, every time need to re-load the nginx.conffile, so scalability is not very high, so we can use dynamic load balancing to achieve Upstream 可配置化, 动态化, 无需人工重新加载 nginx.conf.

This is similar to a distributed configuration center.

2 dynamic load balancing implementation

Every discovery configuration changes need to reload nginx, restart Nginx.

1. Consul + Consul-template
2. Consul+ OpenRestyAchieve dynamic load balancing without having to reload
3. Consul+ upsync+ NginxAchieve dynamic load balancing without having to reload

3 Common server registration and discovery frame

Common service discovery framework Consul, Eureka, ZooKeeperand Etcd.

ZooKeeperIt is one of the oldest of this type of project, which originated in Hadoop. It is very mature, reliable, used by many large companies (YouTube, eBay, Yahoo, etc.).

etcdIt is the use of a HTTPprotocol 键/值on the storage system, which is a distributed system configuration and functional levels can be used to build service discovery system. It is easy to deploy, install and use, provide reliable data persistence features. It is also very safe and complete documentation.

4 ConsulQuick Start

ConsulIs an open source distributed service registration and discovery system, through HTTP APImay make service registration, discovery very simple to implement, it supports the following features.

• 服务注册: The service implementers can HTTP APIor DNSthe way the service is registered Consul.
• 服务发现: Service consumers can HTTP APIor DNSfashion, from Consulaccess to services IPand PORT.
• 故障检测: Supports such as TCP, HTTPhealth checks and other methods mechanism, thereby automatically removed when the service is faulty.
• K/V存储: Use K/V存储dynamic distribution center, which uses HTTPlong polling trigger to achieve change and configuration changes.
• 多数据中心: Support for multiple data center, you can register and find services in the data center, which is to support local consumption only room service, multi-data center cluster also avoid single points of failure in a single data center.
• Raft算法: ConsulUsing the Raftalgorithm clusters data consistency.

By Consulcan manage service registration and discovery, then there is a need and Nginxdeployed on the same machine Agentto implement Nginxconfiguration changes and Nginxrestart function. We have Confdor Consul-templatetwo options, but Consul-templateis Consulofficial, we choose it. Using HTTPlong polling trigger to achieve change and configuration changes (using Consulthe watchcommand implementation). In other words, we use Consul-templateto achieve a configuration template, and then pull Consulconfiguration rendering template to generate Nginxthe actual configuration.

5 Nginx+ Consul+ UpSyncprinciple

ConsulServer Load balancing is used to store configuration

ConsulWeb As a visual profile modification interface

NginxWill pass UpSyncinterval reading ConsulServerthe above profile, a profile will also cached locally, so as not to cause service outages Center configuration is not available.

6 Consulenvironment to build

1. Download consul_0.7.5_linux_amd64.zip

wget https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_linux_amd64.zip

2. Extract consul_0.7.5_linux_amd64.zip

unzip consul_0.7.5_linux_amd64.zip

If the decompression error

-bash: unzip: 未找到命令

Solution

yum -y install unzip

3. Do ./consulthe following message appears on the installation is successful

./consul

[root@weaver-1 ~]# ./consul
usage: consul [--version] [--help] <command> [<args>]

Available commands are:
    agent          Runs a Consul agent
    configtest     Validate config file
    event          Fire a new event
    exec           Executes a command on Consul nodes
    force-leave    Forces a member of the cluster to enter the "left" state
    info           Provides debugging information for operators
    join           Tell Consul agent to join cluster
    keygen         Generates a new encryption key
    keyring        Manages gossip layer encryption keys
    kv             Interact with the key-value store
    leave          Gracefully leaves the Consul cluster and shuts down
    lock           Execute a command holding a lock
    maint          Controls node or service maintenance mode
    members        Lists the members of a Consul cluster
    monitor        Stream logs from a Consul agent
    operator       Provides cluster-level tools for Consul operators
    reload         Triggers the agent to reload configuration files
    rtt            Estimates network round trip time between nodes
    snapshot       Saves, restores and inspects snapshots of Consul server state
    version        Prints the Consul version
    watch          Watch for changes in Consul

[root@weaver-1 ~]# 

4. Start consul

./consul agent -dev -ui -node=consul-dev -client=192.168.153.11

5. temporarily turn off the firewall

systemctl stop firewalld

6. The browser access: http://192.168.153.11:8500/

7. Use PostMan registration Http Service

http://192.168.153.11:8500/v1/catalog/register

{
    "Datacenter":"dc1",
    "Node":"tomcat",
    "Address":"192.168.153.11",
    "Service":{
        "Id":"192.168.153.11:8001",
        "Service":"test",
        "tags":[
            "dev"
        ],
        "Port":8001
    }
}

{
    "Datacenter":"dc1",
    "Node":"tomcat",
    "Address":"192.168.153.11",
    "Service":{
        "Id":"192.168.153.11:8002",
        "Service":"test",
        "tags":[
            "dev"
        ],
        "Port":8002
    }
}

• Datacenter: Specifies the data center
• Address: Specify IP Services
• Service.Id: Specifies the unique identification
• Service.Service: Specifies the service grouping
• Service.tags: Specifies the service tag (such as test environment, advance environment)
• Service.Port: Specifies the service port.

7. discovery Http Service

http://192.168.153.11:8500//v1/catalog/service/item_jd_tomcat

7 nginx-upsync-module

Note: Before clearing Nginxthe environment, re-install.

7.1 nginx-upsync-moduleIntroduction

UpsyncSina Weibo is based on open source Nginxto achieve three modules dynamically configured. Nginx-Upsync-ModuleFunction is to pull Consulthe rear end of the serverlist, and dynamically update Nginxrouting information. This module does not depend on any third-party modules. ConsulAs Nginxthe DBuse of Consulthe KVservice, each Nginx Workprocess independent of each to pull upstreamconfiguration, and update their routing.

7.2 nginx-upsync-moduleInstallation

download file

cd /usr/local/

1. Download Nginx

wget http://nginx.org/download/nginx-1.9.10.tar.gz

Role: to achieve reverse proxy, load load library

2. Download consul

wget https://releases.hashicorp.com/consul/0.7.1/consul_0.7.1_linux_amd64.zip

Role: dynamic load balancing are configured to achieve registration

3. Download nginx-upsync-module

wget https://github.com/weibocom/nginx-upsync-module/archive/master.zip

Role: nginxdynamic access to the latest upstreaminformation

Extracting installer

unzip master.zip

unzip consul_0.7.1_linux_amd64.zip

If the decompression error

-bash: unzip: 未找到命令

Solution

yum -y install unzip

installation Nginx

Decompression Nginx

tar -zxvf nginx-1.9.10.tar.gz

Configuration Nginx

groupadd nginx

useradd -g nginx -s /sbin/nologin nginx

mkdir -p /var/tmp/nginx/client/

mkdir -p /usr/local/nginx

Compile Nginx

cd nginx-1.9.10

./configure   --prefix=/usr/local/nginx   --user=nginx   --group=nginx   --with-http_ssl_module   --with-http_flv_module   --with-http_stub_status_module   --with-http_gzip_static_module   --with-http_realip_module   --http-client-body-temp-path=/var/tmp/nginx/client/   --http-proxy-temp-path=/var/tmp/nginx/proxy/   --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/   --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi   --http-scgi-temp-path=/var/tmp/nginx/scgi   --with-pcre --add-module=../nginx-upsync-module-master

make && make install

Compilation is being given

./configure: error: SSL modules require the OpenSSL library.

Solution

yum -y install openssl openssl-devel

Upstream Dynamic Configuration

## 动态去consul 获取注册的真实反向代理地址
upstream test {
    server 127.0.0.1:11111;
    upsync 192.168.153.11:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
    upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;
}

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_pass http://test;
        index  index.html index.htm;
    }
	
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

• upsyncFrom instruction specifies consulwhich path upstream pull server configuration;
• upsync_timeoutConfiguration from the consulpull timeout upstream server configuration;
• upsync_intervalConfiguration from the consulpull interval upstream server configuration;
• upsync_typeSpecifies consulconfiguration server;
• strong_dependencyDisposed nginxat startup whether mandatory dependency configuration server, if configured on, when the pull configuration failed nginxstartup also fails.
• upsync_dump_pathSpecified from consulthe upstream server persisted to the pulled position, so that even if the consulserver is a problem, there is a local backup.

Note: Replace consulRegistry Address

create upsync_dump_path

mkdir /usr/local/nginx/conf/servers/

upsync_dump_pathSpecified from consulthe upstream server persisted to the pulled position, so that even if the consulserver is a problem, there is a local backup.

start up consul

Temporarily turn off the firewall

systemctl stop firewalld

/usr/local/consul agent -dev -ui -node=consul-dev -client=192.168.153.11

Start both Tomcatservices

/root/tomcat-1/bin/startup.sh

/root/tomcat-2/bin/startup.sh

start up Nginx

/usr/local/nginx/sbin/nginx

Adding nginx UpstreamServices

1. Use postmentransmission putrequest

http://192.168.153.11:8500/v1/kv/upstreams/test/192.168.153.11:8001 

http://192.168.153.11:8500/v1/kv/upstreams/test/192.168.153.11:8002

test:

Modify the load balancing information parameters

{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}

{"weight":2, "max_fails":2, "fail_timeout":10, "down":0}

Test weights can be found in force.