DAY 70 WEB Cache - Squid Proxy Server Application

Forward proxy: instead of the client, send a request to the server.

Reverse proxy: The proxy server forwards the request to multiple servers.

Introduction to Squid Proxy Server

Squid mainly provides functions of cache acceleration and application layer filtering control.

The working mechanism of the proxy (caching web page objects, reducing repeated requests)

1. Instead of the client computer requesting data from the website, the real IP address of the user can be hidden.
2. Save the obtained web page data (static web elements) into the cache and send it to the client, so that the next time the same data is requested for a fast response.

 The concept and function of proxy server

A proxy server is a server between the client and the original (resource) server. In order to obtain content from the original server, the client sends a request to the proxy server and specifies the target original server, and then the proxy server forwards the request to the original server and will get content is returned to the client.

Caching proxies are crucial to the web, especially for large and highly loaded web sites. Caching can be used as an important means of performance optimization, which can greatly reduce the load on the back-end server. Usually static resources, that is, resources that are less frequently updated, such as pictures, css or js, etc. are cached, so that each time the browser is refreshed, it is not necessary to re-request, but read from the cache, which can reduce server traffic. pressure

Its main functions are:

• Resource acquisition: replace the client to achieve resource acquisition from the original server;
• Accelerated access: the proxy server may be closer to the original server, thus playing a certain role in acceleration;
• Cache function: the proxy server saves the resources obtained from the original server, so that the client can quickly obtain them;
• Hiding the real address: The proxy server replaces the client to obtain the original server resources, thereby hiding the real information of the client.

The most basic function: to improve the speed of web access and hide the real IP address of the client.

Types of Squid Proxies

Traditional proxy: suitable for Internet forward proxy, the address and port of the proxy server need to be specified on the client computer.

Transparent proxy: The client does not need to specify the address and port of the proxy server, but redirects the Web access to the proxy server through the default route and firewall policy.

Reverse proxy: If the requested resource is cached in the Squid reverse proxy server, the requested resource will be returned directly to the client; otherwise, the reverse proxy server will request the resource from the background WEB server, and then return the requested response to the client, and also cache the response locally for use by the next requester.

Installation and operation of Squid service

Compile and install Squid

 #关闭防火墙和selinux
 [root@yuji ~]# systemctl disable --now firewalld
 [root@yuji ~]# setenforce 0
 ​
 #安装环境依赖包
 [root@yuji ~]# yum -y install gcc gcc-c++ make 
 #解压squid安装包
 [root@yuji ~]# tar zxvf squid-3.5.28.tar.gz -C /opt/ 
 #切换到源码包目录，设置安装路径和安装模块
 [root@yuji ~]# cd /opt/squid-3.5.28 
 ​
 [root@yuji ~]# ./configure --prefix=/usr/local/squid \      #指定安装目录路径
 --sysconfdir=/etc \                         #指定配置文件路径
 --enable-arp-acl \                          #MAC地址管控，防止客户端使用IP欺骗
 --enable-linux-netfilter \                  #使用内核过滤
 --enable-linux-tproxy \                     #支持透明模式
 --enable-async-io=100 \                     #异步IO，提升存储性能。先写入缓存，再写入硬盘
 --enable-err-language="Simplify_Chinese" \  #错误信息的显示语言
 --enable-underscore \                       #允许URL中有下划线
 --disable-poll \                            #关闭默认使用 poll 模式
 --enable-epoll \        #开启epoll模式提升性能，epoll模式可以支持IO多路复制，异步非阻塞
 --enable-gnuregex                           #使用GNU正则表达式
 ​
 ​
 [root@yuji ~]# make -j2 && make install     #开2核编译安装
 ​
 [root@yuji ~]# ln -s /usr/local/squid/sbin/* /usr/local/sbin/   #优化路径，使系统直接能够识别squid命令
 [root@yuji ~]# useradd -M -s /sbin/nologin squid             #创建squid用户，不创建家目录，不可登录系统
 [root@yuji ~]# chown -R squid:squid  /usr/local/squid/var/   #修改属主属组，此目录用于存放缓存文件

Modify the configuration file of Squid

 [root@yuji ~]# vim /etc/squid.conf
 ......
 --56行--插入
 http_access allow all    #放在 http_access deny all 之前，允许任意客户机使用代理服务，控制规则自上而下匹配
 http_access deny all
 http_port 3128           #用来指定代理服务监听的地址和端口(默认的端口号为 3128)
 --61行--插入
 cache_effective_user squid      #添加，指定程序用户，用来设置初始化、运行时缓存的账号,否则启动
 cache_effective_group squid     #添加，指定账号基本组
 coredump_dir /usr/local/squid/var/cache/squid     #指定缓存文件目录。默认有这一行，一般不做修改

 Squid's run control

 #检查配置文件语法是否正确
 [root@yuji ~]# squid -k parse
 ​
 #启动 Squid，第一次启动 Squid 服务时，会自动初始化缓存目录
 [root@yuji ~]# squid -z        #-z 选项用来初始化缓存目录
 [root@yuji ~]# squid           #启动 squid 服务
 ​
 [root@yuji ~]# netstat -anpt | grep "squid"    #查看是否启动成功

 Create a Squid service script

vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"
 
case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
     else
       echo "正在启动 squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "正在关闭 squid..."
      $0 start &> /dev/null
      echo "正在启动 squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "用法：$0{start|stop|status|reload|check|restart}"
   ;;
esac
-------------------------------------------------------------------------
#2345是默认自启动级别，如是-代表任何级别都不自启动；90是启动优先级，25是停止优先级，优先级范围是0－100，数字越大，优先级越低。
chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on
chkconfig --list squid

 

 Build a traditional proxy server

Squid proxy server: 192.168.137.10/24

web server: 192.168.137.20/24

Client: 192.168.137.1/24

proxy server 192.168.52.110

关闭防护墙与selinux
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
----------------------------------------
vim /etc/squid.conf
--63行--插入
cache_mem 64 MB
#指定缓存功能所使用的内存空间大小，便于保持访问较频繁的WEB对象，容量最好为4的倍数，单位为MB，建议设为物理内存的1/4
 
reply_body_max_size 100 MB 
#允许用户下载的最大文件大小，以字节为单位，当下载超过指定大小的Web对象时，浏览器的报错页面中会出现“请求或访问太大”的提示默认设置0表示不进行限制
 
maximum_object_size 4096 KB
#允许保存到缓存空间的最大对象大小，以KB为单位，超过大小限制的文件将不被缓存，而是直接转发给用户
----------------------------------------------------------------------------------
service squid restart     #重启服务
 
 
#生产环境中还需要修改防火墙规则
iptables -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT
 
#查看 Squid 访问日志的新增记录中的缓存命中情况
tail -f /usr/local/squid/var/logs/access.log
        TCP_MEM_HIT/200
 
#查看 Web 访问日志的新增记录
tail -f /var/log/httpd/access_log
在浏览器输入Web服务器IP地址访问，查看Web服务器访问日志，显示的是由代理服务器替客户机在访问。

 modify firewall rules

 web server 192.168.52.100

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
yum install httpd -y
systemctl start httpd

install apache-web

 start service

 Client's proxy configuration

Use a browser to access and view the web server access log before proxying

 

 Proxy to client

Open the browser (try to use IE browser), Tools --> Internet Options --> Connections --> LAN Settings --> Open proxy server (address: Squid server IP address, port: 3128) 

After configuring the proxy, visit the web server multiple times 

 

 View the cache hits in the newly added records of the Squid access log

tail -f /usr/local/squid/var/logs/access.log 

Build a transparent proxy server

 

Squid server: internal network ens33: 192.168.137.30, external network ens36: 12.0.0.254

Web server: 12.0.0.100

Client: 192.168.137.30

Configure squid server

1.1 Add network card

Settings --> Add --> Network Adapter --> OK

ifconfig  #查看网卡名
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens36
vim ifcfg-ens36
------------------------------
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens36
DEVICE=ens36
ONBOOT=yes
IPADDR=12.0.0.254
NETMASK=255.255.255.0
------------------------------------
vim ifcfg-ens33
----------------------------
注释掉DNS与Gateway
----------------------
systemctl restart network

1.2 Edit the Squid server configuration file

vim /etc/squid.conf
......
http_access allow all
http_access deny all
--60行--修改添加提供内网服务的IP地址，和支持透明代理选项 transparent
http_port 192.168.137.10:3128 transparent
 
#重启squid服务
systemctl restart squid
 
#开启路由转发，实现本机中不同网段的地址转发
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -p
 
#修改防火墙规则
iptables -F
iptables -t nat -F
iptables -t nat -I PREROUTING -i ens33 -s 192.168.137.0/24 -p tcp --dport 80 -j REDIRECT --to 3128	#用于转发http协议
iptables -t nat -I PREROUTING -i ens33 -s 192.168.137.0/24 -p tcp --dport 443 -j REDIRECT --to 3128	#用于转发https协议
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

2. Web server 12.0.0.100 configuration 

2.1 install apache

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
yum install httpd -y
systemctl start httpd

2.2 Modify the network card of the web server

vim /etc/sysconfig/network-scripts/ifcfg-ens33
--------------------------------------------------
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a3484a07-0b78-4735-9302-e04ba6617e8b
DEVICE=ens33
ONBOOT=yes
IPADDR=12.0.0.100
NETMASK=255.255.255.0
GATEWAY=12.0.0.254
DNS1=8.8.8.8
--------------------------------------------------------
systemctl restart network
ifconfig

3. Client 192.168.137.30 settings

3.1 Configure the client's IP and gateway

  3.2 Turn off the proxy server

 3.3 IE browser accesses the web server

3.4 Squid server view access log

tail -f /usr/local/squid/var/logs/access.log

3.5 View new records in the Web access log

It shows that the external network port of the proxy server is used instead of the client to access

tail -f /var/log/httpd/access_log 

6. ACL access control

In the configuration file squid.conf, ACL access control is implemented through the following two steps:

(1) Use the acl configuration item to define the conditions that need to be controlled;

(2) Use the http_access configuration item to control "allow" or "deny" access to the defined list.

1. Define the access control list

格式：
acl 列表名称 列表类型 列表内容 …

vim /etc/squid.conf
......
acl localhost src 192.168.137.10/32 				#源地址为 192.168.137.10
acl MYLAN src 192.168.137.0/24 192.168.1.0/24		#客户机网段
acl destinationhost dst 192.168.80.13/32			#目标地址为 192.168.52.120
acl MC20 maxconn 20									#最大并发连接 20
acl PORT port 21									#目标端口 21
acl DMBLOCK dstdomain .qq.com						#目标域，匹配域内所有站点
acl BURL url_regex -i ^rtsp:// ^emule://			#以 rtsp://、emule:// 开头的目标 URL地址，-i表示忽略大小写
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$		#以 .mp3、.mp4、.rmvb 结尾的目标 URL 路径
acl WORKTIME time MTWHF 08:30-17:30					#时间为周一至周五 8:30~17:30，“MTWHF”为每个星期的英文首字母

Edit the configuration file to define the control access list 

acl src 192.168.137.30/32
http_access deny myhost

restart service

systemctl restart squid

Use the client to access the web server

Seven, Squid log analysis

#安装图像处理软件包
yum install -y pcre-devel gd gd-devel
 
mkdir /usr/local/sarg
tar zxvf sarg-2.3.7.tar.gz -C /opt/
 
cd /opt/sarg-2.3.7
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \							#配置文件目录，默认是/usr/local/etc
--enable-extraprotection							#额外安全防护
 
----------------------------------------------------------------------------------------------------------
./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection
make && make install
 
vim /etc/sarg/sarg.conf
--7行--取消注释
access_log /usr/local/squid/var/logs/access.log		#指定访问日志文件
--25行--取消注释
title "Squid User Access Reports"					#网页标题
--120行--取消注释，修改
output_dir /var/www/html/sarg						#报告输出目录
--178行--取消注释
user_ip no											#使用用户名显示
--184行--取消注释，修改
topuser_sort_field connect reverse					#top排序中，指定连接次数采用降序排列，升序是normal
--190行--取消注释，修改
user_sort_field connect reverse						#对于用户访问记录，连接次数按降序排序
--206行--取消注释，修改
exclude_hosts /usr/local/sarg/noreport				#指定不计入排序的站点列表的文件
--257行--取消注释
overwrite_report no									#同名同日期的日志是否覆盖
--289行--取消注释，修改
mail_utility mailq.postfix							#发送邮件报告命令
--434行--取消注释，修改
charset UTF-8										#指定字符集UTF-8
--518行--取消注释
weekdays 0-6										#top排行的星期周期
--525行--取消注释
hours 0-23											#top排行的时间周期
--633行--取消注释
www_document_root /var/www/html						#指定网页根目录
 
 
#添加不计入站点文件，添加的域名将不被显示在排序中
touch /usr/local/sarg/noreport
 
ln -s /usr/local/sarg/bin/sarg /usr/local/bin/
sarg --help
 
#验证
yum install httpd -y
systemctl start httpd
 
#运行
sarg				#启动一次记录
 
 
浏览器访问 http://192.168.80.10/sarg ，查看sarg报告网页。
 
#添加计划任务，执行每天生成报告
vim /usr/local/sarg/report.sh
#/bin/bash
#Get current date
TODAY=$(date +%d/%m/%Y)
#Get one week ago today
YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y)
/usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $YESTERDAY-$TODAY &> /dev/null
find ./ -type d -a -name "$(env LANG=en_US.UTF-8 date -d "30 day ago" +%Y%b%d)-$(env LANG=en_US.UTF-8 date -d "29 day ago" +%Y%b%d)" | xargs rm -rf
exit 0
 
 
chmod +x /usr/local/sarg/report.sh
 
crontab -e
0 0 * * * /usr/local/sarg/report.sh
 

Eight, Squid reverse proxy

If the requested resource is cached in the Squid reverse proxy server, the requested resource will be returned directly to the client; otherwise, the reverse proxy server will request the resource from the background Web server, and then return the response to the request to the client. At the same time, the response is also cached locally for use by the next requester.

 Working Mechanism:

• Cache web page objects to reduce repeated requests
• Rotate Internet requests or assign weights to intranet Web servers
• Proxy user requests to prevent users from directly accessing the web server and improve security

1. Configure squid server

1.1 Modify the configuration file

vim /etc/squid.conf
......
--60行--修改，插入
http_port 192.168.137.10:80 accel vhost vport
cache_peer 192.168.137.40 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
cache_peer 192.168.52.40 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
cache_peer_domain web1 web2 www.my.com
 
#表示对www.my.com的请求，squid向192.168.137.40和192.168.137.20的80端口发出请求
 
------------------------------------------------------------------------
http_port 80 accel vhost vport 
 ##squid从一个缓存变成了一个Web服务器反向代理加速模式，这个时候squid在80端口监听请求，同时和web server的请求端口(vhost vport)绑定，这个时候请求到了squid，squid是不用转发请求的，而是直接要么从缓存中拿数据、要么向绑定的端口直接请求数据。
 accel ：反向代理加速模式。
 vhost ：支持域名或主机名来表示代理节点。
 vport ：支持IP和端口来表示代理节点。
 ​
 parent ：代表为父节点，上下关系，非平级关系。
 80 ：代理内部web服务器的80端口。
 0 ：没有使用icp（电信运营商），表示就一台squid服务器。
 no-query ：不做查询操作，直接获取数据。
 originserver ：指定是源服务器。
 round-robin ：指定 squid 通过轮询方式将请求分发到其中一台父节点。
 max_conn ：指定最大连接数。
 weight ： 指定权重。
 name ：设置别名。

1.2 Clear firewall rules

iptables -F
iptables -t nat -F

1.3 Start squid service

systemctl stop httpd       #防止 httpd 服务使用的 80 端口号和 squid 反向代理配置的监听端口冲突
netstat -natp | grep 80
systemctl restart squid

2. Web server configuration

2.1web1 server configuration

turn off firewall

systemctl stop firewalld.service 
setenforce 0

 Install apache and configure the homepage of the webpage

yum install -y httpd
systemctl start httpd
echo "this is test01" >> /var/www/html/index.html
systemctl restart httpd

2.2web2 server configuration

turn off firewall

systemctl stop firewalld.service 
setenforce 0

 Install apache and configure the homepage of the webpage

yum install -y httpd
systemctl start httpd
echo "this is test02" >> /var/www/html/index.html
systemctl restart httpd

3. Client domain name mapping configuration

3.1 Modify the C:\Windows\System32\drivers\etc\hosts file

192.168.52.110 www.my.com

3.2 The browser does not open the proxy to access http://www.my.com

3.3 View logs

tail -f /usr/local/squid/var/logs/access.log