Squid proxy server proxy, ACL access control, Squid log analysis tool Sarg and Squid reverse proxy

1. Overview of Caching Proxy

1.1 Basic types of agents

• Traditional proxy: applicable to the internet, the server needs to be clearly specified
• Transparent proxy: The client does not need to specify the address and port of the proxy server, but redirects web access to the proxy server through the default route and firewall policy
  . The benefits
  of using a proxy increase the speed of web access
  hide the real IP address of the client

1.2 Working mechanism

• Cache web page objects to reduce repeated requests
  

Second, the establishment of traditional agents

2.1 Experimental environment

web: 192.168.233.140
squid: ens33 192.168.233.127
client: 192.168.233.30

2.2 Squid installation

• Prepare squid-3.4.6 compressed package

tar zxvf squid-3.4.6.tar.gz -C /opt
cd  /opt
cd squid-3.4.6/

 ./configure --prefix=/usr/local/squid  \
--sysconfdir=/etc \
--enable-arp-acl \          ## 配置ACL
 --enable-linux-netfilter \  ## 过滤表
 --enable-linux-tproxy \      ## 支持透明代理的功能模块
 --enable-async-io=100 \        ## 吞吐量
 --enable-err-language="Simplify_Chinese" \  ## 字符集
 --enable-underscore \  ## 支持url中带有下划线的字符
 --enable-poll \  ## poll模块  内核模块
 --enable-gnuregex \    ## 支持正则
make &&make install

2.3 Squid optimization

[root@localhost squid-3.4.6]# ln -s /usr/local/squid/sbin/* /usr/local/sbin/
[root@localhost squid-3.4.6]# useradd -M -s /sbin/nologin squid
[root@localhost squid-3.4.6]# chown -R squid.squid /usr/local/squid/var/

[root@localhost squid-3.4.6]# vim /etc/squid.conf
# http_access deny all
 http_access allow all

# Squid normally listens to port 3128
http_port 3128
cache_effective_user  squid
cache_effective_group  squid
coredump_dir  /usr/local/squid/var/cache/squid

[root@localhost squid-3.4.6]# squid -k parse  ## 检查配置
[root@localhost init.d]# squid -z  ## 初始化
[root@localhost squid-3.4.6]# squid  ## 开启服务
[root@localhost squid-3.4.6]# netstat -atnp | grep 3128
tcp6       0      0 :::3128                 :::*                    LISTEN      56583/(squid-1) 
[root@localhost squid-3.4.6]# cd /etc/init.d/
[root@localhost init.d]# vim squid      ## 配置service启动脚本
#!/bin/bash
#chkconfig: 2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
start)
        netstat -natp | grep squid &> /dev/null
        if [ $? -eq 0 ]
        then
        echo "squid is running"
        else
        echo "正在启动 squid...."
        $CMD
        fi
        ;;
stop)
         $CMD -k kill &> /dev/null
        rm -rf $PID &> /dev/null
        ;;
status)
        [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
        then
                netstat -natp | grep squid
        else
                echo "squid is not running"
        fi
        ;;
restart)
        $0 stop &>  /dev/null
        echo "正在关闭 squid...."
        $0 start  &> /dev/null
        echo "正在启动 squid..."
        ;;
reload)
        $CMD -k reconfigure
        ;;
check)
        $CMD -k parse
        ;;
*)
        echo "用法：$0{start|stop|status|reload|check|restart}"
        ;;
esac
[root@localhost init.d]# chkconfig --add squid
[root@localhost init.d]# chkconfig --level 35 squid on
[root@localhost init.d]#  chmod +x squid 
[root@localhost init.d]# service squid start

2.4 Modification of configuration file and firewall configuration

[root@localhost init.d]# vim /etc/squid.conf
# http_access deny all
 http_access allow all
http_port 3128
cache_effective_user  squid
cache_effective_group  squid
cache_mem 64 MB      ## 指定缓存
reply_body_max_size 10 MB   ## 每一次下载单个文件最大为10 MB
maximum_object_size 4096 KB  ## 如果里面的文件大于4MB  就不缓存  直接转发给用户
[root@localhost init.d]# iptables -t 
[root@localhost init.d]# iptables -t nat -F
[root@localhost init.d]# setenforce 0
[root@localhost init.d]# iptables -I INPUT -p tcp --dport 3218  -j ACCEPT

2.5 Web server opens website service

yum -y install httpd
systemctl start httpd

2.6 Set proxy server on client browser

• Set the proxy server 192.168.233.127 port 3128 on the server to
  access 192.168.233.140
  

2.7 Client access website

• Check the access log of the web server after successfully visiting the website

root@localhost ~]# cat /var/log/httpd/access_log 
192.168.233.127 - - [06/Sep/2020:13:47:16 +0800] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://192.168.233.140/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"
192.168.233.127 - - [06/Sep/2020:13:47:16 +0800] "GET /noindex/css/fonts/Light/OpenSans-Light.ttf HTTP/1.1" 404 240 "http://192.168.233.140/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"

The experiment was successful

Three, the construction of transparent agency

3.1 Environmental preparation

web: 192.168.233.127
squid: ens33 192.168.233.140
ens36 192.168.100.1
Client: 192.168.100.10

3.2 Squid configuration file modification

[root@localhost ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward=1
[root@localhost ~]# sysctl -p
[root@localhost ~]# vim /etc/squid.conf
http_port  192.168.100.1:3128 transparent
cache_effective_user  squid
cache_effective_group  squid
[root@localhost ~]# service squid reload

3.3 Firewall configuration

[root@localhost ~]# iptables -t nat -I PREROUTING -i ens36 -s 192.168.100.0/24 -p tcp --dport 80 -j REDIRECT --to 3128
[root@localhost ~]# iptables -t nat -I PREROUTING -i ens36 -s 192.168.100.0/24 -p tcp --dport 443 -j REDIRECT --to 3128
[root@localhost ~]# iptables -I INPUT -p tcp --dport 3128 -j ACCEPT 

3.4 web server configuration

• Add static route

[root@localhost ~]# route add   -net  192.168.100.0/24  gw 192.168.233.127

• The client turns off the proxy server to
  access 192.168.233.140
• View web server log

WebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"
192.168.233.127 - - [06/Sep/2020:14:11:03 +0800] "GET /noindex/css/fonts/Light/OpenSans-Light.ttf HTTP/1.1" 404 240 "http://192.168.233.140/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"
192.168.233.127 - - [06/Sep/2020:14:11:03 +0800] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://192.168.233.140/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"

Four, ACL access control

4.1 ACL access control method

• Define the list according to the source address, target URL, file type, etc.

acl  列表名称 列表类型  列表内容

• Restrictions on the defined acl list

http_access  allow或deny  列表名称……

4.2 ACL rule priority

• When a user accesses the proxy server, Squid will match all the rule lists defined in Squid in order. Once the match is successful, it will stop the match immediately
• When none of the rules match, Squid will use the opposite rule to the last one

4.3 Common ACL list types

Control statement	Corresponding type
src	source address
dst	target address
port	port
dstdomain	Target domain
time	interview time
maxconn	Maximum concurrent connections
url_regex	The target URL address supports regular
Urlpath_regex	The entire target URL address supports regular

4.4 acl experiment

• For the sake of simplicity, continue to operate in the traditional proxy mode just done
• Browser to set proxy site

[root@localhost ~]# vim /etc/squid.conf
acl 222  src 192.168.233.30
http_access deny 222
[root@localhost ~]# service  squid  reload

注释掉刚刚在配置文件添加的项
[root@localhost ~]# service  squid  reload
重载服务  又可以访问了
[root@localhost ~]# mkdir /etc/squid
[root@localhost ~]# cd /etc/squid/
[root@localhost squid]# vim src.list
192.168.233.30

[root@localhost squid]# vim /etc/squid.conf
acl srchost src "/etc/squid/src.list"
http_access deny srchost 
[root@localhost ~]# service squid reload
再次用客户机访问web服务器  发现  又不能访问了

Five, Squid log analysis tool Sarg

• Turn off the rules set during the ACL test so that the client can access the web server normally

5.1 Install Sarg

准备安装包sarg-2.3.7.tar.gz 
[root@localhost ~]# yum -y install gd gd-devel    ## gd 图像处理工具  
[root@localhost ~]# mkdir /usr/local/sarg
[root@localhost ~]# tar zxvf sarg-2.3.7.tar.gz -C /opt/
[root@localhost ~]# cd /opt/sarg-2.3.7/
[root@localhost sarg-2.3.7]# ./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection  ## 额外安全防护
[root@localhost sarg-2.3.7]# make && make install

5.2 Modify Sarg configuration file

[root@localhost sarg-2.3.7]# cd /etc/sarg/
[root@localhost sarg]# vim sarg.conf 

  7 access_log /usr/local/squid/var/logs/access.log      ## 需确认默认路径下是否有这个日志文件
25 title "Squid User Access Reports"       ## 网页标题
120 output_dir /var/www/html/squid-reports    ## 报告输出目录
178 user_ip no        ## 使用用户名显示？
184 topuser_sort_field connect  reverse ## top排序中有连接次数、访问字节、降序排列 升序是normal
190 #user_sort_field BYTES reverse   ## 一定要注释掉  用户访问记录  连接次数、访问字节按降序排序
206 exclude_hosts  /usr/local/sarg/noreport       ## 不计入排序的站点文件
257 overwrite_report no        ## 同名日志是否覆盖
289 mail_utility mailq.postfix   ## 发送邮件报告命令
434 charset UTF-8	## 使用字符集
518 weekdays 0-6	## top排行的星期周期
525 hours 0-23	## top排行的时间周期
633 www_document_root /var/www/html  ## 网页根目录

[root@localhost sarg]# touch /usr/local/sarg/noreport  ## 创建不计入排序的站点空文件   添加的域名将不显示在排序中

5.3 Optimize start Sarg service

root@localhost sarg]# ln -s /usr/local/sarg/bin/sarg  /usr/local/bin/
[root@localhost sarg]# sarg
SARG: 纪录在文件: 1866, reading: 100.00%
SARG: 成功的生成报告在 /var/www/html/squid-reports/2020Sep06-2020Sep07
[root@localhost ~]# yum -y install httpd   # 安装http服务
[root@localhost ~]# systemctl start httpd    ## 开启服务   因为apapche服务的站点目录在/var/www/html 目录下，所有可以直接访问sarg生成的文件
访问
http://192.168.233.127/squid-reports/2020Sep06-2020Sep07/index.html

5.4 Periodic task generation report

[root@localhost ~]# sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/squid-reports/ -z -d $(date -d "1 day ago" +%d/%m/%Y)-$(date +%d/%m/%Y)
SARG: TAG: access_log /usr/local/squid/var/logs/access.log
SARG: TAG: title "Squid User Access Reports"
SARG: TAG: output_dir /var/www/html/squid-reports
SARG: TAG: user_ip no
SARG: TAG: topuser_sort_field connect  reverse
SARG: TAG: exclude_hosts  /usr/local/sarg/noreport
SARG: TAG: overwrite_report no
SARG: TAG: mail_utility mailq.postfix
SARG: TAG: charset UTF-8
SARG: TAG: weekdays 0-6
SARG: TAG: hours 0-23
SARG: TAG: www_document_root /var/www/html
SARG: 纪录在文件: 2633, reading: 100.00%
SARG: 期间被日志文件覆盖: 06/09/2020 - 07/09/2020
SARG: (info) date=07/09/2020
SARG: (info) period=2020 9月 06-2020 9月 07
SARG: (info) outdirname=/var/www/html/squid-reports//2020Sep06-2020Sep07
SARG: (info) Dansguardian report not produced because no dansguardian configuration file was provided
SARG: (info) No redirector logs provided to produce that kind of report
SARG: (info) Authentication failures report not produced because it is empty
SARG: (info) Redirector report not generated because it is empty
SARG: 成功的生成报告在 /var/www/html/squid-reports//2020Sep06-2020Sep07

## 周期性计划任务每天生成报告   crontab

Six, Squid reverse proxy

• Under the traditional proxy, we
  must pay attention to httpd service and squid to preempt port 80
  web2: 192.168.233.101
  web1: 192.168.233.140
  squid: 192.168.233.127
  client: 192.168.233.30

6.1 web1 website page settings

[root@localhost ~]# yum -y install httpd
[root@localhost ~]# vim /var/www/html/index.html
<h1> this is web2 </h1>
[root@localhost ~]# systemctl start httpd

6.2 web2 website page settings

[root@localhost ~]# yum -y install httpd
[root@localhost ~]# vim /var/www/html/index.html
<h1>this is web1 </h1>
[root@localhost ~]# systemctl restart httpd

6.3 Modify Squid configuration file

[root@localhost ~]# vim /etc/squid.conf
http_port  192.168.233.127:80 accel vhost vport  ## 监听的虚拟服务 端口80  地址为Squid的地址
cache_peer 192.168.233.140 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
cache_peer 192.168.233.101 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
## 地址、端口采用round-robin 轮询模式  源服务  最大并发连接为30  权重都设置为1    名字分别为web1 web2 no-query不做查询操作，直接获取数据
cache_peer_domain web1 web2   www.abcd.com   ## 代理域客户端包含 web1 web2    域名为www.abcd.com
[root@localhost ~]# systemctl stop httpd
[root@localhost ~]# service squid reload

6.4 Client Settings

• Change the proxy server port to 80
• Enter C:\Windows\System32\drivers\etc
  and write
  192.168.233.127 www.abcd.com in the host file

Visit www.abcd.com