Continuous web output: application of squid server

One, Squid proxy server

Squid mainly provides the functions of cache acceleration and application layer filtering control.

1.1 The working mechanism of the agent

1. Instead of the client requesting data from the website, the user's real IP address can be hidden.
2. Save the obtained web page data (static web element) in the cache and send it to the client, so as to respond quickly when the same data is requested next time.

1.2 Types of Squid proxy

Traditional proxy: applicable to the Internet, the address and port of the proxy server need to be specified on the client.
Transparent proxy: The client does not need to specify the address and port of the proxy server, but redirects Web access to the proxy server for processing through the default route and firewall policy.
Reverse proxy: If the requested resource is cached in the Squid reverse proxy server, the requested resource will be directly returned to the client: otherwise the reverse proxy server will request the resource from the background WEB server, and then return the response to the request To the client, the response is also cached locally for use by the next requester.

2. Basic deployment

2.1 Turn off the firewall

systemctl stop firewalld
systemctl disable fi rewalld
setenforce 0

2.2 Compile and install Squid

yum -y install gcc gcc-c++ make
tar zxvf squid-3.5.28.tar.gz -C /opt/
cd /opt/squid-3.5.28

. /configure --prefix=/usr/local/squid \    #指定安装目录路径
--sysconfdir=/etc \    #指定配置文件路径
--enable-arp-acl \    #MAC地址管控，防止客户端使用IP欺骗
--enable-linux- -netfilter \    #使用内核过滤
--enable-l inux-tproxy \    #支持透明模式
--enable-async-io=100 \    #异步I0，提升存储性能
--enable-err- language="Simplify_ Chinese" \    #错误信息的显示语言
--enable-underscore \    #允许URL中有下划线
--disable-poll \    #关闭默认使用poll 模式
--enable-epoll \    #开启epoll模式提升性能
-- enable-gnuregex    #使用GNU正则表达式


./configure --prefix=/usr/local/squid \
--sysconfdir=/etc \
--enable-arp-acl \
--enable-linux-netfilter \
--enable-linux-tproxy \
--enable-async-io=100 \
--enable-err-language="Simplify_Chinese" \
--enable-underscore \
--disable-poll \
--enable-epoll \
--enable-gnuregex

make && make install

ln -s /usr/local/squid/sbin/* /usr/local/sbin/
useradd -M -s /sbin/nologin squid
chown -R squid:squid /usr/local/squid/var/    #此目录用于存放缓存文件

2.3 Modify Squid's configuration file

vim /etc/squid.conf
......
--56行--插入
http_access allow all    #放在http_access deny all之前，允许任意客户机使用代理服务，控制规则自上而下匹配
http_access deny all
--61行--插入
http_port 3128    #用来指定代理服务监听的地址和端口(默认的端口号为3128)
cache_effective_user squid    #添加，指定程序用户，用来设置初始化、运行时缓存的账号,否则启动不成功
cache_effective_group squid    #添加，指定账号基本组
coredump_dir /usr/ocal/squid/var/cache/squid    #指定缓存文件目录

2.4 Squid operation control

#检查配置文件语法是否正确
squid -k parse

#启动Squid, 第一次启动Squid服务时，会自动初始化缓存目录
squid -z    #一z选项用来初始化缓存目录
squid    #启动squid 服务
netstat -anpt | grep "squid"

2.5 Create Squid service script

vim /etc/init.d/squid
#!/bin/bash
#chkconfig:2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
	start)
		netstat -natp | grep squid &> /dev/nu1l
		if [ $? -eq 0 ]
		then
			echo "squid is running"
			else
			echo "正在启动squid..."
			$CMD
		fi
;;
stop)
	$CMD -k kill &> /dev/null
	rm -rf $PID &> /dev/null
;;
status)
	[ -f $PID ] &> /dev/null
		if [ $? -eq 0 ]
			then
				netstat -natpI grep squid
			else
				echo "squid is not running"
		fi
;;
restart)
	$0 stop &> /dev/null
	echo "正在关闭squid...'
	$0 start &> /dev/null
	echo "正在启动squid...
;;
reload)
	$CMD -k reconfigure
;;
check)
	$CMD -k parse
;;
*)
echo "用法: $0{start | stop | status | reload | check | restart}"
;;
esac
#2345是默认自启动级别，如-是代表任何级别都不自启动; 90是启动优先级，25是停止优先级，优先级范围是0一100， 数字越大，优先级越低。
chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on
chkconfig --list squid

Third, build a traditional proxy server

vim /etc/squid.conf
......
http_access allow all
http_access deny all
http_port 3128
cache_effective_user squid
cache_effective_group squid
--63行--插入
cache_mem 64 MB
#指定缓存功能所使用的内存空间大小，便于保持访问较频繁的WEB对象，容量最好为4的倍数，单位为MB，建议设为物理内存的1/4
reply_body_max_size 10 MB
#允许用户下载的最大文件大小，以字节为单位，当下载超过指定大小的web对象时，浏览器的报错页面中会出现“请求或访问太大”的提示,默认设置0表示不进行限制
maximum_object_size 4096 KB
#允许保存到缓存空间的最大对象大小，以KB为单位，超过大小限制的文件将不被缓存，而是直接转发给用户

service squid restart
systemctl restart squid

#生产环境中还需要修改防火墙规则
iptables -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

web server

yum install -y httpd
systemctl start httpd.service 

systemctl stop firewalld.service 
setenforce 0
systemctl disable firewalld.service 

#客户机的代理配置
打开浏览器，工具-->Internet选项-->连接-->局域网设置-->开启代理服务器(地址: Squid服务器IP地址，端口:3128)
#查看Squid访问日志的新增记录
tail -f /usr/local/squid/var/logs/access.log

#查看Web访问日志的新增记录
tail -f /var/log/httpd/access_log
在浏览器输入Web服务器IP地址访问，查看Web服务器访问日志，显示的是由代理服务器替客户机在访问。

Fourth, build a transparent proxy server

equipment	IP address
Squid server	Dual network cards, internal network ens33: 192.168.238.20 external network ens36: 12.0.0.1
Web server	12.0.0.12
Client computer	192.168.238.77

Squid server configuration

vim /etc/squid.conf
......
http_access allow all
http_access deny all
--60行--修改添加提供内网服务的IP地址，和支持透明代理选项transparent
http_port 192.168.238.20:3128 transparent

systemctl restart squid

#开启路由转发，实现本机中不同网段的地址转发
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -p

#修改防火墙规则
iptables -F
iptables -t nat -F
iptables -t nat -I PREROUTING -i ens33 -s 192.168.238.0/24 -p tcp --dport 80 -j REDIRECT --to 3128
#用于转发http协议
iptables -t nat -I PREROUTING -i ens33 -s 192.168.238.0/24 -p tcp --dport 443 -j REDIRECT --to 3128
#用于转发https协议
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

Web server configuration

yum install -y httpd
systemctl start httpd
关闭客户机的浏览器之前设置的代理服务器的功能后访问http://12.0.0.12
#查看Squid访问日志的新增记录
tail -f /usr/local/squid/var/logs/access.log
#查看Web访问日志的新增记录，显示的是由代理服务器的外网口代替客户机在访问
tail -f /var/log/httpd/access_log

Five, ACL access control

In the configuration file squid.conf, ACL access control is achieved through the following two steps:
(1) Use the acl configuration item to define the conditions that need to be controlled;
(2) Use the http_access configuration item to "allow" or "allow" the defined list Deny the control of access.
1. Define an access control list

格式:
acl 列表名称 列表类型 列表内容...

方法一
vim /etc/squid.conf
......
acl localhost src 192.168.238.20/32					#源地址为192.168.238.20
acl MYLAN src 192.168.238.0/24						#客户机网段
acl destinationhost dst 192.168.238.50/32			#目标地址为192.168.238.50
acl MC20 maxconn 20									#最大并发连接20
acl PORT port 21									#目标端口21
acl DMBLOCK dstdomain .qq.com						#目标域，匹配域内所有站点
acl BURL url_regex -i ^rtsp:// ^emule://			#以rtsp://、emule://开头的URL，-i表示忽略大小写
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$		#以.mp3、.mp4、.rmvb 结尾的URL 路径
acl WORKTIME time MTWHF 08:30-17:30					#时间为周一至周五8:30~17:30，“MTWHF" 为每个星期的英文首字母

方法二
#启动对象列表管理
mkdir /etc/squid
vim /etc/squid/dest.list
192.168.238.50
192.168.238.0/24

vim /etc/squid.conf
......
acl destinationhost dst "/etc/squid/dest.list"	#调用指定文件中的列表内容
http_access deny(或allow) destinationhost	#注意，如果是拒绝列表，需要放在http_access allow all 前面
systemctl restart squid
浏览器访问Web服务器http://192.168.238.50，显示被访问被拒绝。

Six, Squid log analysis

#安装图像处理软件包
yum install -y pcre-devel gd gd-devel
mkdir /usr/local/sarg
tar zxvf sarg-2.3.7.tar.gz -C /opt/
cd /opt/sarg-2.3.7
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \	#配置文件目录，默认是/usr/ loca/etc
--enable-extraprotection	#额外安全防护

./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection
make && make install

vim /etc/sarg/sarg.conf
--7行--取消注释
access_log /usr/local/squid/var/logs/access.log  #指定访问日志文件
--25行--取消注释
title "Squid User Access Reports"     #网页标题
--120行--取消注释，修改
output_dir /var/www/html/sarg      #报告输出目录
--178行--取消注释
user_ip no           #使用用户名显示
--184行--取消注释，修改
topuser_sort_field connect reverse     #top排序中，指定连接次数采用降序排列，升序是normal
--190行--取消注释，修改
user_sort_field connect reverse      #对于用户访问记录，连接次数按降序排序
--206行--取消注释，修改
exclude_hosts /usr/local/sarg/noreport    #指定不计入排序的站点列表的文件
--257行--取消注释
overwrite_report no         #同名同日期的日志是否覆盖
--289行--取消注释，修改
mail_utility mailq.postfix       #发送邮件报告命令
--434行--取消注释，修改
charset UTF-8          #指定字符集UTF-8
--518行--取消注释
weekdays 0-6          #top排行的星期周期
--525行--取消注释
hours 0-23           #top排行的时间周期
--633行--取消注释
www_document_root /var/www/html      #指定网页根目录

#添加不计入站点文件，添加的域名将不被显示在排序中
touch /usr/local/sarg/noreport
ln -s /usr/local/sarg/bin/sarg /usr/local/bin/
sarg --help

#运行
sarg	#启动一次记录
#验证
yum install httpd -y
systemctl start httpd
浏览器访问http://192.168.238.20/sarg，查看sarg报告网页。

#添加计划任务，执行每天生成报告
vim /usr/local/sarg/report.sh
#/bin/bash
#Get current date
TODAY=$(date +%d/%m/%Y)
#Get one week ago today
YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y)
/usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $YESTERDAY-$TODAY &> /dev/null
exit 0

chmod +x /usr/local/sarg/report.sh
crontab -e
0 0 * * * /usr/local/sarg/report.sh

Seven, Squid reverse proxy

If the requested resource is cached in the Squid reverse proxy server, the requested resource will be directly returned to the client: otherwise the reverse proxy server will request the resource from the backend Web server, and then return the response to the request to the client. At the same time, the response is cached locally for the next requester to use.
Working mechanism:
●Cache webpage objects to reduce repeated requests
●Rotate Internet requests or assign them to the intranet Web server according to weight
●Proxy user requests to prevent users from directly accessing the Web server and improve security

vim /etc/squid.conf
......
--60行--修改，插入
http_port 192.168.238.20:80 accel vhost vport
cache_peer 192.168.238.30 parent 80 0 no-query originserver round-robin max_ conn=30 weight=1 name=web1
cache_peer 192.168.238.50 parent 80 0 no-query originserver round-robin max_ conn=30 weight=1 name=web2
cache_peer_domain web1 web2 www.xyw.com
#表示对www.xyw.com的请求，squid向192.168.80.11和192.168.80.12的80端口发出请求

http_port 80 accel vhost vport
#squid从一个缓存变成了一个Web服务器反向代理加速模式，这个时候squid在80端口监听请求，同时和web server的请求端口(vhost vport)绑定，这个时候请求到了squid，squid是不用转发请求的，而是直接要么从缓存中拿数据要么向绑定的端口直接请求数据。
accel :反向代理加速模式
vhost：支持域名或主机名来表示代理节点
vport：支持IP和端口来表示代理节点

parent：代表为父节点，上下关系，非平级关系
80：代理内部web服务器的80端口
0：没有使用icp，表示就一台squid服务器.
no-query：不做查询操作，直接获取数据
originserver：指定是源服务器
round-robin：指定squid通过轮询方式将请求分发到其中一台父节点
max_conn：指定最大连接数
weight：指定权重
name：设置别名

systemctl stop httpd
systemctl restart squid

#后端节点服务器设置
yum install -y httpd
systemctl start httpd
#节点1：
echo "this is 30" >> /var/www/html/index.html
#节点2：
echo "this is 50" >> /var/www/html/index.html
#客户机的域名映射配置
修改C:\Windows\System32\drivers\etc\hosts 文件
192.168.238.20 www.xyw.com
#客户机的代理配置
打开浏览器，工具-->Internet选项-->连接-->局域网设置-->开启代理服务器(地址: Squid服务器IP地址，端口: 80)
浏览器访问http://www.xyw.com

service squid reload