Explain Squid proxy server in detail (how to build traditional proxy, transparent proxy, ACL control, squid log analysis, squid proxy)

1. Overview of Caching Proxy

1. The working mechanism of Web proxy

Cache web page objects to reduce repeated requests

2. The basic types of agents

Traditional proxy: applicable to the Internet, the server needs to be clearly specified

Transparent proxy: The client does not need to specify the address and port of the proxy server, but redirects Web access to the proxy server through the default route and firewall policy

3. The benefits of using a proxy

Improve web access speed

Hide the real IP address of the client

2. How to build a traditional agency

Host	IP address	Main software
Squid proxy server	192.168.100.11	squid
Web site service	192.168.100.12	httpd
win10	192.168.100.13	Browser

1. Install dependencies

yum install gcc gcc-c++ -y

2. Compile and install Squid service

tar xf squid-3.5.23.tar.gz
cd squid-3.5.23/

./configure \
--prefix=/usr/local/squid \
--sysconfdir=/etc \       #指定配置文件位置
--enable-arp-acl \        #支持acl访问控制列表
--enable-linux-netfilter \  #打开网络筛选
--enable-linux-tproxy \    #支持透明代理
--enable-async-io=100 \  #io优化
--enable-err-language="Simplify_Chinese" \  #报错显示简体中文
--enable-underscore \   #支持下划线
--enable-poll \               #默认使用poll模式，开启epoll模式时提升性能
--enable-gnuregex       #支持正则表达式

make && make install

ln -s /usr/local/squid/sbin/* /usr/local/sbin/
useradd -M -s /sbin/nologin squid
chown -R squid.squid /usr/local/squid/var/

3. Modify configuration files to optimize startup items

vim /etc/squid.conf
http_access allow all    #56行添加此项，表示允许所有IP访问
#http_access deny all   #注释原有的
http_port 3128
cache_effective_user squid     #添加指定用户squid
cache_effective_group squid   #添加指定组squid
coredump_dir /usr/local/squid/var/cache/squid

squid -k parse   //检查配置文件语法
squid -z   //初始化缓存目录
squid    //启动服务
netstat -ntap |grep 3128   #检测是否启动成功

4. Add service to service management

cd /etc/init.d/
vim squid

#!/bin/bash
#chkconfig: 2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
       else
       echo "正在启动 squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "正在关闭 squid..."
         $0 start &> /dev/null
      echo "正在启动 squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "用法：$0{start|stop|status|reload|check|restart}"
   ;;
esac

chmod +x /etc/init.d/squid

chkconfig --add squid
chkconfig --level 35 squid on

5. Configure traditional proxy

vim /etc/squid.conf

http_port 3128
cache_effective_user squid
cache_effective_group squid
cache_mem 64 MB      #缓存空间大小定义为64 MB
reply_body_max_size 10 MB  #允许下载的最大文件大小，默认0表示不进行限制
maximum_object_size 4096 KB  #允许保存到缓存空间的最大对象的大小，以KB为单位，超过限制不会缓存，直接转到web端

[root@localhost ~]# iptables -F
[root@localhost ~]# iptables -F -t nat
[root@localhost ~]# setenforce 0
[root@localhost ~]# iptables -I INPUT -p tcp --dport 3128 -j ACCEPT
[root@localhost ~]# systemctl restart squid

6. Install httpd service on the web server

systemctl stop firewalld.service
yum -y install httpd
systemctl start httpd

#Visit the website on the client to see if it is normally accessed

#Check the content of the log file, the visiting IP is still the client's own IP

[root@localhost ~]# cat /var/log/httpd/access_log

7. Set up squid proxy, check the change of visiting IP in the log file

Note: (1) Clear the browser cache of the client first

(2) Manually set the proxy-enable the use of proxy server-set the proxy address and port-save

(3) Visit the apache server again to check the changes in the log file; the visiting IP displayed at this time becomes the Squid proxy server address

3. Transparent proxy

在搭建的传统代理基础上做如下修改：
Squid 配置双网卡内网ens33  外网ens36

Host	IP address
Squid proxy server	192.168.50.11 (internal network), 192.168.100.11 (external network)
Web site service	192.168.100.12 (external network)
Test machine	192.168.50.13 (intranet)

Test machine hangs proxy to access Web site

web server

route add -net 192.168.50.0/24 gw 192.168.100.11

Squid proxy server

vi /etc/sysctl.conf
net.ipv4.ip_forward=1		#开启路由功能

vim /etc/squid.conf

http_port 192.168.50.11:3128 transparent

service squid reload

iptables -t nat -I PREROUTING -i ens37 -s 192.168.50.0/24 -p tcp --dport 80 -j REDIRECT --to 3128
 
iptables -t nat -I PREROUTING -i ens37 -s 192.168.50.0/24 -p tcp --dport 443 -j REDIRECT --to 3128

iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

Access web without proxy

4.ACL control

Squid provides a powerful proxy control mechanism. By reasonably setting ACL (Access Control List, access control list) and restricting it, it can filter based on various conditions such as source address, destination address, access URL path, and access time.

1. Modify in the configuration file

vim /etc/squid.conf
acl localhost src 192.168.50.13/32
http_access deny localhost
重启服务
systemctl restart squid

Visit again, the following interface appears

Some other restrictions

vi /etc/squid.conf
acl localhost src 192.168.100.13/32			#针对固定源IP地址
acl MYLAN src 192.168.100.0/24				#针对某一网段
acl destionhost dst 192.168.175.130/32		#针对具体的目标IP地址
acl MC20 maxconn 20							#访问的最大并发连接数量
acl BURL url_regex -i ^rtsp:// ^emule://	#正则表达式的访问协议
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$		#访问的文件资源末尾
acl work time MTWHFAC 08:30-17:30			#访问时间

http_access deny destionhost   #拒绝列表（注意置顶）

Enable object list management

mkdir /etc/squid     #启用对象列表管理

vim dest.list
   
   192.168.175.150
   192.168.175.140
   192.168.175.130   #目标web

vim /etc/squid.conf
   
   acl destionhost dst "/etc/squid/dest.list"


http_access deny destionhost   #拒绝列表（注意置顶）

5. Squid log analysis

1. Compile and install sarg software

[root@server1 ~]# yum install -y gd gd-devel
mkdir /usr/local/sarg

tar zxvf sarg-2.3.7.tar.gz -C /opt/

cd /opt/sarg-2.3.7

./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \
--enable-extraprotection            #额外安全防护

make && make install

2. Modify the configuration file

cd /etc/sarg/

vim sarg.conf
7/ access_log /usr/local/squid/var/logs/access.log    //指定访问日志文件
25/ title "Squid User Access Reports"      //网页标题
120/ output_dir /var/www/html/squid-reports    //报告输出目录
178/ user_ip no          //使用用户名显示
206/ exclude_hosts /usr/local/sarg/noreport   //不计入排序的站点列表文件
184/ topuser_sort_field connect reverse   //top排序中有连接次数、访问字节、降序排列 升序是normal
（注释掉）190/ user_sort_field reverse    //用户访问记录 连接次数、访问字节按降序排序
257/ overwrite_report no   //同名日志是否覆盖
289/ mail_utility mailq.postfix   //发送邮件报告命令
434/ charset UTF-8   //使用字符集
518/ weekdays 0-6   //top排行的星期周期
525/ hours 0-23   //top排行的时间周期
633/ www_document_root /var/www/html  //网页根目录

3. Modify the test

#添加不计入站点文件，添加的域名将不被显示在排序中
touch /usr/local/sarg/noreport

ln -s /usr/local/sarg/bin/sarg /usr/local/bin/


[root@localhost sarg]# sarg 
SARG: 纪录在文件: 242, reading: 100.00%
SARG: 成功的生成报告在 /var/www/html/squid-reports/2018Jul21-2018Jul21

yum install httpd -y

systemctl start httpd
systemctl stop firewalld

http://192.168.175.128/squid-reports

周期性计划
crontab -e
*/1 * * * * /usr/local/bin/sarg			每分钟生成一次，用于测试

#周期性计划任务执行每天生成报告crontab
sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/squid-reports/ -z -d $(date -d "1 day ago" +%d/%m/%Y)-$(date +%d/%m/%Y)

Then use the client to visit http://192.168.100.11/squid-reports

6.squid proxy

Reverse proxy, in fact, the client is unaware of the proxy, because the client can access without any configuration, we only need to send the request to the reverse proxy server, and the reverse proxy server selects the target server to obtain the data. When returning to the client, the reverse proxy server and the target server are external servers, exposing the address of the proxy server and hiding the real server IP address.

Because httpd occupies port 80, the httpd service in the squid server must be closed

1 Introduction

Tradition and transparency are for the client, using Squid to speed up access to web services, or the company's internal restrictions on employees' Internet behavior. Squid's service object in reverse proxy mode is a web server. Squid hides the real web server IP, speeds up customer access, and has a load balancing function.
The setting of reverse proxy requires three steps to complete:

• DNS resolution
• SQUID configuration
• Port forwarding

2. Project planning

One squid server
Two web servers, web1: 192.168.100.12 web2: 192.168.100.13
One win10 client

3. Web server deployment

//安装httpd
yum install httpd -y

//设置网页内容
echo "this is test02 web" > /var/www/html/index.html    #web1换一下数字
route add -net 192.168.50.0/24 gw 192.168.100.11  #静态路由

//开启web服务
systemctl start httpd

4. Squid proxy configuration

//设置防火墙规则
systemctl start firewalld
iptables -L    #查看防火墙规则
iptables -F
iptables -t nat -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

vim /etc/squid.conf
#去掉透明代理设置反向代理
http_port 192.168.100.11:80 accel vhost vport

#节点服务器1最大访问30，权重1，别名web1
cache_peer 192.168.100.12 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
#节点服务器2最大访问30，权重1，别名web2
cache_peer 192.168.100.13 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
#访问yun.com匹配web1，web2节点
cache_peer_domain web1 web2 www.yun.com

service squid restart

5. Set up hosts on the client

vi /etc/hosts

192.168.100.11 www.yun.com

Visit www.yun.com

Two pages alternate