Bowen outline:
- A, varnish Profile
- Two, Varnish How It Works
- Third, the deployment of varnish cache server
A, varnish Profile
Varnish is open source, high-performance HTTP server and reverse proxy cache server, which functions like Squid server, can be used for HTTP caching. Varnish can be installed on any web front end, which configure the cache contents. Compared with the traditional squid, varnish with higher performance, faster, easier management and many other advantages. Some companies have used it as an alternative to the old version of the squid in a production environment to provide better results in the same cache server costs, Varnish is an optional service as one of CDN cache servers.
The main characteristics of the varnish is as follows:
- Cache location: You can use a disk memory can also be used. If you are using SSD disk would recommend doing RAID1;
- Log storage: logs are also stored in memory. Storage Policy: fixed size, recycling;
- It supports the use of virtual memory;
- Precise time management mechanism to cache time attribute control;
- State engine architecture: done on different engines for different caching and proxy data for processing. Can design different control statements by the specific configuration language to determine cache data in different ways in different locations of the packets are processed through a specific rule in a particular place;
- Cache Management: binary format heap cache data management, timely clean up the data.
varnish compared to squid, is a reverse proxy cache server, and is open source, high varnish stability and access to fast, because Squid cache data is read from the hard disk, the data stored and Varnish in memory, read directly from the memory, to avoid the frequent exchange of memory, disk file, so Varnish is relatively more efficient, varnish can support more concurrent connections, because of fast varnish TCP connection release than squid; varnish also through management port, use regular expressions to remove part of a batch of cache, while the squid is impossible; squid belong to a single process using a single-core CPU, but Varnish is to open a multi-process handling by fork form to do, so it is reasonable to used to process all nuclear corresponding request.
He said a lot of varnish above advantages, but varnish is not perfect, there are two main disadvantages:
1, varnish Crash or restart the process once, cached data will be fully released from memory, then all requests are
sent to the back-end server under high concurrency, will cause great stress backend server;
2, in the varnish using a single url if requested by the HA / F5 load balancing, etc., it is different for each request falls varnish servers, resulting in the request will be penetrated to the back end; and the same request cached on multiple servers, can also cause waste of resources varnish cache, resulting in performance degradation;
Varnish disadvantage solution::
for a disadvantage: a large amount of access start recommended varnish memory cache way in the case, but later need to talk to more than one squid / nginx server. The main front of the varnish in order to prevent service, if the server is restarted, the large number of requests penetration varnish, so squid / nginx can serve as a second layer on CACHE, but also to make up for the restart will release varnish cache in memory problems;
for weaknesses two: you can do on the url hash load balancing, make a single request to the url request fixed on a single server varnish;
Two, Varnish How It Works
Varnish master process is responsible for starting the work, the master process reads the configuration file, according to the space designated (eg administrator assigned a 2G memory) to create storage space, to create and manage child process;
then child processes to handle follow-up tasks, it will allocate some threads to perform different tasks, such as: http accept the request, allocate storage space for the cache object, remove expired cache objects, free space, defragmentation.
http request process is as follows:
1, there is a dedicated thread receives http request, has been listening for requests port, when a request came, responsible for evoking a worker thread to handle the request, the worker thread will analyze uri http requests, know this What you want to request it to the cache to find whether there is the object, and if so, put the cache object directly back to the user, and if not, will be transferred to the back-end server processes the request, and wait for the results obtained from the backend worker results content, first the content stored as a cache object to the buffer space (for quick response to the next request object), and then returns the content to the user
Cache allocation process is as follows:
When there is a need to cache objects, according to the size of this object, find the most suitable for the size of free blocks to free buffer zone, after finding the object put into them, if the object does not fill this free block, put the remaining space as a new free block, if not free buffer zone in place, we must first delete some cache to make room, remove the least recent use doctrine.
Cache release process is as follows:
There is a thread to be responsible for releasing a cache of work, he regularly checks the cache lifetime of all objects, if an object has not been accessed in a specified period of time, put the object is deleted, the release of its cache footprint, freeing up space after check whether the adjacent memory space is free, and if so, integrated into a larger free block, finishing implement space debris.
More varnish properties, please venue to varnish official website .
Third, the deployment of varnish cache server
Preparing the environment:
- Three centos 7.5 server, IP respectively 192.168.20.5,20.4,20.3;
- Which is IP192.168.20.5 varnish cache server, and the other two for the back-end web server, respectively, to prepare a different page file (I am here to change web content to its IP), in order to verify its cache effect;
- Download I provide source package varnish, varnish and uploaded to the server.
1, began to deploy the installation varnish:
[root@varnish ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
#下载阿里镜像站的repo文件
[root@varnish ~]# yum -y install libedit-devel pcre-devel python-docutils #安装依赖包
[root@varnish ~]# cd /usr/src #切换至指定目录
[root@varnish src]# rz #上传我提供的varnish源码包
[root@varnish src]# tar zxf varnish-4.0.3.tar.gz #解包
[root@varnish src]# cd varnish-4.0.3/ #进入解压后的目录
[root@varnish varnish-4.0.3]# ./configure && make && make install #编译安装
[root@varnish varnish-4.0.3]# cp etc/example.vcl /usr/local/var/varnish/ #复制模板到指定路径
[root@varnish varnish-4.0.3]# cd !$ #切换到varnish安装目录下
[root@varnish varnish]# vim example.vcl #编辑其模板配置文件，根据自己所需功能，编写以下内容：
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
import directors;
import std;
# Default backend definition. Set this to point to your content server.
probe backend_healthcheck {
.url="/"; #访问后端服务器根路径
.interval = 5s; #请求时间间隔
.timeout = 1s; #请求超时时间
.window = 5; #指定轮询次数5次
.threshold = 3; #如果出现3次失败则表示后端服务器宕机
}
backend web1 { #定义后端服务器
.host = "192.168.20.4"; #要转向主机（即后端主机）的 IP 或域名
.port = "80"; #指定后端服务器的端口号
.probe = backend_healthcheck; #健康检查调用backend_healthcheck定义的内容
}
backend web2 {
.host = "192.168.20.3";
.port = "80";
.probe = backend_healthcheck;
}
acl purgers { #定义访问控制列表
"127.0.0.1";
"localhost";
"192.168.20.0/24";
!"192.168.20.4";
}
sub vcl_init { #调用 vcl_init 初始化子程序创建后端主机组，即 directors
new web_cluster=directors.round_robin(); #使用 new 关键字创建 drector 对象,使用 round_robin(轮询) 算法
web_cluster.add_backend(web1); #添加后端服务器节点
web_cluster.add_backend(web2);
}
sub vcl_recv {
set req.backend_hint = web_cluster.backend(); #指定请求的后端节点web_cluster定义的后端节点
if (req.method == "PURGE") { #判断客户端的请求头部是否是PURGE
if (!client.ip ~ purgers) { #如果是，再判断客户端的IP地址是不是在ACL访问控制列表中.
return (synth(405, "Not Allowed.")); #如果不是，返回给客户端405状态码并且返回定义的页面.
}
return (purge); #如果是ACL定义的，则交给purge处理.
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "PATCH" &&
req.method != "DELETE") { #判断客户端的请求类型
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
return (pass); #如果不是GET及HEAD则交给pass.
}
if (req.url ~ "\.(php|asp|aspx|jsp|do|ashx|shtml)($|\?)") {
return (pass); #当客户端访问的是.php等结尾的交给pass处理.
}
if (req.http.Authorization) {
return (pass); #当客户端请求的页面类型是需要认证的，交给pass处理
}
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(bmp|png|gif|jpg|jpeg|ico|gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)$") {
unset req.http.Accept-Encoding; #取消客户端接收的压缩类型
} elseif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip"; #如果有gzip类型，标记gzip类型.
} elseif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding; #其他未定义的页面也取消客户但接收的压缩类型.
}
}
if (req.url ~ "\.(css|js|html|htm|bmp|png|gif|jpg|jpeg|ico|gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)($|\?)") {
unset req.http.cookie; #取消客户端的cookie值.
return (hash); #将请求转发给hash子程序，也就是查看本地缓存.
}
if (req.restarts == 0) { #判断客户端是不是第一次请求
if (req.http.X-Forwarded-For) { #如果是第一次请求，设置获取客户端的IP地址.
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
return (hash);
}
sub vcl_hash {
hash_data(req.url); #查看客户端请求的页面，并且进行hash
if (req.http.host) {
hash_data(req.http.host); #设置客户端的主机
} else {
hash_data(server.ip); #设置服务器的IP
}
return (lookup);
}
sub vcl_hit {
if (req.method == "PURGE") { #如果是HIT并且当客户端请求的类型是PURGE返回的200的状态码，并返回相应页面.
return (synth(200, "Purged."));
}
return (deliver);
}
sub vcl_miss {
if (req.method == "PURGE") {
return (synth(404, "Purged.")); #如果是miss返回404
}
return (fetch);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.CXK = "HIT-from-varnish"; #设置http头部X-Cache =hit
set resp.http.X-Cache-Hits = obj.hits; #返回命令的次数
} else {
set resp.http.X-Cache = "MISS";
}
unset resp.http.X-Powered-By; #取消显示web版本
unset resp.http.Server; #取消显示varnish服务
unset resp.http.X-Drupal-Cache; #取消显示缓存的框架
unset resp.http.Via; #取消显示文件内容来源
unset resp.http.Link; #取消显示HTML的超链接地址
unset resp.http.X-Varnish; #取消显示varnish的id
set resp.http.xx_restarts_count = req.restarts; #设置客户端请求的次数
set resp.http.xx_Age = resp.http.Age; #显示缓存文件的时长
#set resp.http.hit_count = obj.hits; #显示缓存命中的次数
#unset resp.http.Age;
return (deliver);
}
sub vcl_pass {
return (fetch); #将后端服务器返回的数据缓存到本地
}
sub vcl_backend_response {
set beresp.grace = 5m; #缓存额外宽限时间
if (beresp.status == 499 || beresp.status == 404 || beresp.status == 502) {
set beresp.uncacheable = true; #当后端服务器相应状态码是449等，不缓存
}
if (bereq.url ~ "\.(php|jsp)(\?|$)") {
set beresp.uncacheable = true; #当是PHP的页面不缓存
} else {
if (bereq.url ~ "\.(css|js|html|htm|bmp|png|gif|jpg|jpeg|ico)($|\?)") {
set beresp.ttl = 15m; #当是上面结尾的，缓存15分钟
unset beresp.http.Set-Cookie;
} elseif (bereq.url ~ "\.(gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)($|\?)") {
set beresp.ttl = 30m; #缓存30分钟
unset beresp.http.Set-Cookie;
} else {
set beresp.ttl = 10m; #生存时间10分钟
unset beresp.http.Set-Cookie;
}
}
return (deliver);
}
sub vcl_purge {
return (synth(200,"success"));
}
sub vcl_backend_error {
if (beresp.status == 500 ||
beresp.status == 501 ||
beresp.status == 502 ||
beresp.status == 503 ||
beresp.status == 504) {
return (retry); #如果状态码是上述其中之一，则重新请求
}
}
sub vcl_fini {
return (ok);
}
#编辑完成后，保存退出后即可。
[root@varnish varnish]# varnishd -f /usr/local/var/varnish/example.vcl -s malloc,200M -a 0.0.0.0:80
#启动varnish服务，监听本机所有IP的80端口，-f为指定vcl文件，-s是指定用来存放缓存的容量(Before Google browser to access the press "F12") client access to test certain features:
Press the "F5" refresh:
Access to the configuration file that we specified in the header information, and the status code 304.
Verify ACL clear the cache configuration:
To clear the cache operating in a host 192.168.20.4 (varnish configuration of this IP is not allowed to clear the cache):
[root@localhost ~]# curl -X "PURGE" 192.168.20.5 #清除varnish的缓存You get the following error message:
Cleared cache operation (192.168.20.3 host) on the varnish allows the IP, will see the following success message:
Additional:
No comments above complete profile is as follows:
vcl 4.0;
import directors;
import std;
probe backend_healthcheck {
.url="/";
.interval = 5s;
.timeout = 1s;
.window = 5;
.threshold = 3;
}
backend web1 {
.host = "192.168.20.4";
.port = "80";
.probe = backend_healthcheck;
}
backend web2 {
.host = "192.168.20.3";
.port = "80";
.probe = backend_healthcheck;
}
acl purgers {
"127.0.0.1";
"localhost";
"192.168.20.0/24";
!"192.168.20.4";
}
sub vcl_init {
new web_cluster=directors.round_robin();
web_cluster.add_backend(web1);
web_cluster.add_backend(web2);
}
sub vcl_recv {
set req.backend_hint = web_cluster.backend();
if (req.method == "PURGE") {
if (!client.ip ~ purgers) {
return (synth(405, "Not Allowed."));
}
return (purge);
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "PATCH" &&
req.method != "DELETE") {
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
if (req.url ~ "\.(php|asp|aspx|jsp|do|ashx|shtml)($|\?)") {
return (pass);
}
if (req.http.Authorization) {
return (pass);
}
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(bmp|png|gif|jpg|jpeg|ico|gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)$") {
unset req.http.Accept-Encoding;
} elseif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elseif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
if (req.url ~ "\.(css|js|html|htm|bmp|png|gif|jpg|jpeg|ico|gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)($|\?)") {
unset req.http.cookie;
return (hash);
}
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
return (hash);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
sub vcl_hit {
if (req.method == "PURGE") {
return (synth(200, "Purged."));
}
return (deliver);
}
sub vcl_miss {
if (req.method == "PURGE") {
return (synth(404, "Purged."));
}
return (fetch);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.CXK = "HIT-from-varnish";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Drupal-Cache;
unset resp.http.Via;
unset resp.http.Link;
unset resp.http.X-Varnish;
set resp.http.xx_restarts_count = req.restarts;
set resp.http.xx_Age = resp.http.Age;
#set resp.http.hit_count = obj.hits;
#unset resp.http.Age;
return (deliver);
}
sub vcl_pass {
return (fetch);
}
sub vcl_backend_response {
set beresp.grace = 5m;
if (beresp.status == 499 || beresp.status == 404 || beresp.status == 502) {
set beresp.uncacheable = true;
}
if (bereq.url ~ "\.(php|jsp)(\?|$)") {
set beresp.uncacheable = true;
} else {
if (bereq.url ~ "\.(css|js|html|htm|bmp|png|gif|jpg|jpeg|ico)($|\?)") {
set beresp.ttl = 15m;
unset beresp.http.Set-Cookie;
} elseif (bereq.url ~ "\.(gz|tgz|bz2|tbz|zip|rar|mp3|mp4|ogg|swf|flv)($|\?)") {
set beresp.ttl = 30m;
unset beresp.http.Set-Cookie;
} else {
set beresp.ttl = 10m;
unset beresp.http.Set-Cookie;
}
}
return (deliver);
}
sub vcl_purge {
return (synth(200,"success"));
}
sub vcl_backend_error {
if (beresp.status == 500 ||
beresp.status == 501 ||
beresp.status == 502 ||
beresp.status == 503 ||
beresp.status == 504) {
return (retry);
}
}
sub vcl_fini {
return (ok);
}
In fact, if you want to achieve varnish cache function, the basic definition of the following can be realized (example.vcl file has the following contents can):
vcl 4.0;
import directors;
probe backend_healthcheck {
.url = "/";
.timeout = 1s;
.interval = 5s;
.window = 5;
.threshold = 3;
}
backend web1 {
.host = "192.168.20.3";
.port = "80";
.probe = backend_healthcheck;
}
backend web2 {
.host = "192.168.20.4";
.port = "80";
.probe = backend_healthcheck;
}
sub vcl_init {
new web_cluster = directors.round_robin();
web_cluster.add_backend(web1);
web_cluster.add_backend(web2);
}
sub vcl_recv {
set req.backend_hint = web_cluster.backend();
}-------- end of this article so far, thanks for reading --------