CDN简介
CDN（Content Delivery Network）是构建在网络之上的内容分发网络,依靠部署在各地的边缘服务器，通过中心平台的负载均衡、内容分发、调度等功能模块，使用户就近获取所需内容，降低网络拥塞，提高用户访问响应速度和命中率。CDN的关键技术主要有内容存储和分发技术。
CDN的基本原理是广泛采用各种缓存服务器，将这些缓存服务器分布到用户访问相对集中的地区或网络中，在用户访问网站时，利用全局负载技术将用户的访问指向距离最近的工作正常的缓存服务器上，由缓存服务器直接响应用户请求。
搭建CDN系统可以选择3种主流方案：
1. squid 2. varnish 3. Nginx+memcache
Varnish简介
Varnish是一款高性能的开源HTTP加速器,Varnish的功能与Squid服务器相似，都可以用来做HTTP缓存。 Verdens Gang 使用3台Varnish代替了原来的12台Squid,性能优越于Squid
Varnish工作原理图
VCL (Varnish Configuration Language)处理流程图
处理过程大致分为如下几个步骤:
(1)Receive 状态,也就是请求处理的入口状态,根据 VCL 规则判断该请求应该是 Pass 或Pipe,或者进入 Lookup(本地查询)。
(2)Lookup 状态,进入此状态后,会在 hash 表中查找数据,若找到,则进入 Hit 状态,否则进入 miss 状态。
(3)Pass 状态,在此状态下,会进入后端请求,即进入 fetch 状态。
(4)Fetch 状态,在 Fetch 状态下,对请求进行后端的获取,发送请求,获得数据,并进行本地
的存储。
(5)Deliver 状态, 将获取到的数据发送给客户端,然后完成本次请求
加速工作过程：
当varnish上没有缓存数据时：
client -> dns -> server -> cdn -> cache -> client
当varnish上没有缓存数据时：
client -> dns -> cdn -> cache -> client
实验环境
三台虚拟机+一台真实主机
主机名 IP 服务
server1 172.25.8.1 Varnish
server2 172.25.8.2 Apache
server3 172.25.8.3 Apache
foundation8 172.25.8.250 Client实验一： 配置varnish服务器
server1：
1.下载安装varnish软件包
http://varnish-cache.org/releases/
[root@server1 ~]# ls
varnish-3.0.5-1.el6.x86_64.rpm varnish-libs-3.0.5-1.el6.x86_64.rpm
[root@server1 ~]# yum install * -y安装完成后生成/etc/varnish/目录
[root@server1 ~]# cd /etc/varnish/
[root@server1 varnish]# ls
default.vcl secret
[root@server1 varnish]# ll
total 8
-rw-r--r-- 1 root root 3119 Dec 2 2013 default.vcl
-rw------- 1 root root 37 Jul 28 17:42 secret
[root@server1 varnish]# vim /etc/sysconfig/varnish 文件内容：
7 # Maximum number of open files (for ulimit -n) #最大打开文件数
8 NFILES=131072
9
10 # Locked shared memory (for ulimit -l)
11 # Default log size is 82MB + header #最大内存数
12 MEMLOCK=82000
13
14 # Maximum number of threads (for ulimit -u) #最大线程数
15 NPROCS="unlimited"
2.修改varnish的监听端口
66 VARNISH_LISTEN_PORT=803.修改配置文件
[root@server1 varnish]# cat /etc/passwd #安装varnish后会生成一个varnish用户
varnish:x:498:499:Varnish Cache:/var/lib/varnish:/sbin/nologin
[root@server1 varnish]# vim /etc/security/limits.conf
varnish - nofile 1317072
varnish - memlock 82000
varnish - nproc unlimited
4.开启服务
[root@server1 varnish]# /etc/init.d/varnish start
Starting Varnish Cache: [ OK ][root@server1 ~]# netstat -antlp #查看端口
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1085/varnishd
# 显示varnish的监听端口为80
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 910/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 986/master
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 1084/varnishd
# 80端口已打开
tcp 0 0 172.25.8.1:22 172.25.8.250:60974 ESTABLISHED 1050/sshd
tcp 0 0 :::80 :::* LISTEN 1085/varnishd
tcp 0 0 :::22 :::* LISTEN 910/sshd
tcp 0 0 ::1:25 :::* LISTEN 986/master
server2：
下载apache并开启服务
[root@server2 ~]# yum install httpd -y
[root@server2 ~]# /etc/init.d/httpd start编辑发布页面内容
[root@server2 ~]# vim /var/www/html/index.html
www.westos.orgserver1：
修改varnish监听网络服务器的ip
[root@server1 ~]# vim /etc/varnish/default.vcl
backend default {
.host = "172.25.8.2";
.port = "80";
}
[root@server1 ~]# /etc/init.d/varnish reload #重新加载
Loading vcl from /etc/varnish/default.vcl
Current running config name is boot
Using new config name reload_2018-07-29T09:28:19
VCL compiled.
available 0 boot
active 0 reload_2018-07-29T09:28:19
DoneClient客户端测试：
实验二：查看缓存命中情况
server1：
[root@server1 ~]# vim /etc/varnish/default.vcl
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT from westos cache";
}
else {
set resp.http.X-Cache = "MISS from westos cache";
}
return (deliver);
}
[root@server1 varnish]# /etc/init.d/varnish reload #重新加在varnish
Loading vcl from /etc/varnish/default.vcl
Current running config name is reload_2018-07-29T09:28:19
Using new config name reload_2018-07-29T09:43:10
VCL compiled.
available 0 boot
available 2 reload_2018-07-29T09:28:19
active 0 reload_2018-07-29T09:43:10
Done
Client客户端测试：
添加本地解析：
[root@foundation8 images]# vim /etc/hosts
172.25.8.1 www.westos.org[root@foundation8 ~]# curl www.westos.org -I
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Sun, 29 Jul 2018 01:11:37 GMT
ETag: "a02db-f-5721906d01f82"
Content-Type: text/html; charset=UTF-8
Content-Length: 15
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 01:43:47 GMT
X-Varnish: 2053658677
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS from westos cache # 第一次访问未命中
[root@foundation8 ~]# curl www.westos.org -I
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Sun, 29 Jul 2018 01:11:37 GMT
ETag: "a02db-f-5721906d01f82"
Content-Type: text/html; charset=UTF-8
Content-Length: 15
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 01:43:51 GMT
X-Varnish: 2053658678 2053658677
Age: 4
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT from westos cache # 后续访问命中varnish默认缓存时间120s，超过120s自动清除缓存
实验三：通过 varnishadm 手动清除缓存
varnishadm ban.url .*$ #清除所有
varnishadm ban.url /index.html #清除 index.html 页面缓存
varnishadm ban.url /admin/$ #清除 admin 目录缓存server1进行清除缓存:
[root@server1 varnish]# varnishadm ban.url /index.htmlClient进行测试:
[root@foundation8 ~]# curl 172.25.8.1/index.html -I
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Sun, 29 Jul 2018 01:11:37 GMT
ETag: "a02db-f-5721906d01f82"
Content-Type: text/html; charset=UTF-8
Content-Length: 15
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 01:51:20 GMT
X-Varnish: 2053658731
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS from westos cache
实验四：定义多个不同域名站点的后端服务器
[root@server1 ~]# vim /etc/varnish/default.vcl
backend web1 {
.host = "172.25.8.2";
.port = "80";
}
backend web2 {
.host = "172.25.8.3";
.port = "80";
}
## 当访问 www.westos.org或者westos.com 域名时从 web1 上取数据,
## 访问 bbs.westos.org 域名时到 web2 取数据,访问其他页面报错
sub vcl_recv {
if (req.http.host ~ "^(www.)?westos.org") {
set req.http.host = "www.westos.org";
set req.backend = web1;
} elsif (req.http.host ~ "^bbs.westos.org") {
set req.backend = web2;
} else {error 404 "westos cache";
}
}
[root@server1 varnish]# /etc/init.d/varnish reload #重新加载配置文件
Loading vcl from /etc/varnish/default.vcl
Current running config name is reload_2018-07-29T09:43:10
Using new config name reload_2018-07-29T10:07:35
VCL compiled.
available 0 boot
available 0 reload_2018-07-29T09:28:19
available 2 reload_2018-07-29T09:43:10
active 0 reload_2018-07-29T10:07:35
Doneserver2:配置网络服务器
开启httpd服务，编写测试页并重启服务
[root@server2 ~]# vim /var/www/html/index.html
www.westos.orgserver3:配置网络服务器
安装并开启服务
[root@server3 ~]# yum install httpd -y
[root@server3 ~]# /etc/init.d/httpd start编辑默认发布界面：
[root@server3 ~]# vim /var/www/html/index.html
bbs.westos.orgClient进行测试:
添加本地解析:
[root@foundation8 varnish]# vim /etc/hosts
172.25.8.1 www.westos.org bbs.westos.org westos.org访问测试：
方法1：
方法二：
[root@foundation8 ~]# curl 172.25.8.1/index.html -I
HTTP/1.1 404 westos cache
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 398
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 02:23:42 GMT
X-Varnish: 2053658751
Age: 0
Via: 1.1 varnish
Connection: close # 链接关闭
X-Cache: MISS from westos cache
[root@foundation8 ~]# curl www.westos.org/index.html -I
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Sun, 29 Jul 2018 01:11:37 GMT
ETag: "a02db-f-5721906d01f82"
Content-Type: text/html; charset=UTF-8
Content-Length: 15
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 02:24:05 GMT
X-Varnish: 2053658753 2053658752
Age: 3
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT from westos cache
[root@foundation8 ~]# curl bbs.westos.org/index.html -I
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Sun, 29 Jul 2018 02:01:01 GMT
ETag: "9ff7a-f-57219b778fded"
Content-Type: text/html; charset=UTF-8
Content-Length: 15
Accept-Ranges: bytes
Date: Sun, 29 Jul 2018 02:24:23 GMT
X-Varnish: 2053658756 2053658755
Age: 3
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT from westos cache
实验五：负载均衡和健康状况
server1：
[root@server1 ~]# vim /etc/varnish/default.vcl
backend web1 {
.host = "172.25.8.2";
.port = "80";
}
backend web2 {
.host = "172.25.8.3";
.port = "80";
}
## 把多个后端聚合为一个组,并检测后端健康状况
## Round Robin（中文翻译为轮询调度）是一种以轮询的方式依次将一个域名
## 解析到多个IP地址的调度不同服务器的计算方法。
director westos round-robin {
{.backend = web1;}
{.backend = web2;}
}
sub vcl_recv {
if (req.http.host ~ "^(www.)?westos.org") {
set req.http.host = "www.westos.org";
set req.backend = westos; ##调用轮询方法,将访问分发至后端轮询组
return (pass); ##轮叫,不进行缓存
} elsif (req.http.host ~ "^bbs.westos.org") {
set req.backend = web2;
} else {error 404 "westos cache";
}
[root@server1 varnish]# /etc/init.d/varnish reload #重新加载varnish
Loading vcl from /etc/varnish/default.vcl
Current running config name is reload_2018-07-29T14:29:11
Using new config name reload_2018-07-29T14:36:06
VCL compiled.
available 0 boot
available 0 reload_2018-07-29T09:28:19
available 0 reload_2018-07-29T09:43:10
available 0 reload_2018-07-29T10:07:35
available 2 reload_2018-07-29T14:29:11
active 0 reload_2018-07-29T14:36:06
Doneservser2：
[root@server2 ~]# vim /var/www/html/index.html
www.westos.org -server2server3：
NameVirtualHost *:80
<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName bbs.westos.org
</VirtualHost>
<VirtualHost *:80>
DocumentRoot /www1
ServerName www.westos.org
</VirtualHost>
[root@server3 ~]# mkdir /www1
[root@server3 ~]# cd /www1/
[root@server3 www1]# ls
[root@server3 www1]# vim index.html
www.westos.org -server3
[root@server3 www1]# /etc/init.d/httpd restart #重启http服务
Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.8.3 for ServerName
[ OK ]测试虚拟主机，做域名解析
[root@server3 www1]# vim /etc/hosts
172.25.8.3 server3 www.westos.org bbs.westos.org
测试：
[root@server3 www1]# curl www.westos.org
www.westos.org -server3
[root@server3 www1]# curl bbs.westos.org
bbs.westos.orgClient客户端测试：
网页测试：
shell测试：
varnish只起到调度作用，但是始终不缓存
注释return (pass);
[root@server1 ~]# vim /etc/varnish/default.vcl
[root@server1 varnish]# /etc/init.d/varnish reload
Loading vcl from /etc/varnish/default.vcl
Current running config name is reload_2018-07-29T14:36:06
Using new config name reload_2018-07-29T15:17:43
VCL compiled.
available 0 boot
available 0 reload_2018-07-29T09:28:19
available 0 reload_2018-07-29T09:43:10
available 0 reload_2018-07-29T10:07:35
available 0 reload_2018-07-29T14:29:11
available 2 reload_2018-07-29T14:36:06
active 0 reload_2018-07-29T15:17:43
Doneclient测试：
server2和server3不再轮叫，访问120次以后自动切换下一个
varnish既调度也缓存
实验六：搭建CDN推送平台
server1:推送平台
1.安装apache php，并修改端口号
[root@server1 ~]# yum install httpd php -y因为varnish占用的80端口,所以修改httpd的端口为8080
[root@server1 ~]# vim /etc/httpd/conf/httpd.conf
2.下载bansys安装包并解压：
http://code.google.com/p/varnish-php-bansys/
[root@server1 ~]# ls
bansys.zip
[root@server1 ~]# yum install -y unzip #准备解压
[root@server1 ~]# unzip bansys.zip -d /var/www/html/ #解压到指定位置/var/www/html(apche指定默认发布目录)
[root@server1 ~]# cd /var/www/html
[root@server1 html]# ls
bansys3.把解压后的文件放入apache默认发布目录
[root@server1 html]# cd bansys/
[root@server1 bansys]# ls
class_socket.php config.php index.php purge_action.php static
[root@server1 bansys]# mv * .. #将bansys文件里的内容移动到/var/www/html
[root@server1 bansys]# cd ..
[root@server1 html]# ls
bansys config.php purge_action.php
class_socket.php index.php static4.修改配置文件
[root@server1 html]# vim config.php #删除或注释数据库部分内容
<?php
//可定义多个主机列表
$var_group1 = array(
'host' => array('172.25.8.1'),
'port' => '80',
);
//varnish群组定义
//对主机列表进行绑定
$VAR_CLUSTER = array(
'www.westos.org' => $var_group1,
);
//varnish版本
//2.x和3.x推送命令不一样
$VAR_VERSION = "3";
5.开启httpd服务
[root@server1 html]# /etc/init.d/httpd restart #重启httpd服务
Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.8.1 for ServerName
[root@server1 html]# netstat -antlp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1085/varnishd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 910/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 986/master
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 1084/varnishd
tcp 0 0 172.25.8.1:22 172.25.8.250:60974 ESTABLISHED 1050/sshd
tcp 0 0 :::8080 :::* LISTEN 1506/httpd
# httpd端口已经改为8080
tcp 0 0 :::80 :::* LISTEN 1085/varnishd
tcp 0 0 :::22 :::* LISTEN 910/sshd
tcp 0 0 ::1:25 :::* LISTEN 986/master 6.浏览器访问测试
bansys 有两种工作模式：telnet 模式和 http 模式。
1. telnet 模式需要关闭 varnish服务管理端口的验证,注释掉/etc/sysconfig/varnish 文件中的 “ -S ${VARNISH_SECRET_FILE}”这行,重启 varnish 服务即可。
2. http 模式需要对 varnish 做以下设置：
[root@server1 html]# vim /etc/varnish/default.vcl
acl westos { #设置访问控制
"127.0.0.1"; #本地网址
"172.25.8.0"/24; #子网掩码、网段
}
backend web1 {
.host = "172.25.8.2";
.port = "80";
}
backend web2 {
.host = "172.25.8.3";
.port = "80";
}
sub vcl_recv {
if (req.request == "BAN") {
if (!client.ip ~ westos) {
error 405 "Not allowed.";
}
ban("req.url ~ " + req.url);
error 200 "ban added";
}
[root@server1 ~]# /etc/init.d/varnish reload #重新加载服务在未推送前：
开始推送：
刷新页面：
