Squid reverse proxy, sarg log analysis (ACL policy configuration)

Preface

Specific squid environment to build, please click on the link below, there will no longer introduce
squid environmental structures .

1. Squid's ACL access control

Squid provides a powerful proxy control mechanism. By reasonably setting ACL (Access Control List, access control list) and restricting it, it can filter based on various conditions such as source address, destination address, access URL path, and access time.

Steps of ACL access control:

1. Use the acl configuration item to define the conditions that need to be controlled.
Definition format: acl list name, list type, list content...
2. "Allow" or "deny" access to the defined list through the http_access configuration item

Two, two ways to set ACL

2.1, modify directly in squid configuration file

vim /etc/squid.conf
acl hostlocal src 192.168.10.160/32        #针对固定的源ip地址
acl MYLAN src 192.168.220.0/24              #针对某一网段
acl destionhost dst 192.168.220.130/32   #针对具体的目标ip地址
acl MC20 maxconn 20                             #访问的最大并发连接数量
acl BURL url_regex -i ^rtsp:// ^emule://    #正则表达式的访问协议
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$   #访问的文件资源末尾
acl work time MTWHF 08:30-17:30          #访问时间
MTWHF：周一到周五
MTWHFAS：周一到周日
AC ：周一 周三

http_access denyhostlocal     #拒绝列表 (注意置顶)

Specific placement

2.2. Create a new file and declare the file location in the configuration file

mkdir /etc/squid      #启用对象列表管理
vim sha.list
       192.168.100.160  #目标web
   
vim /etc/squid.conf
    acl hostlocal scr "/etc/squid/dest.list"
  
    http_access deny hostlocal    #拒绝列表 (注意置顶)

Specific placement

Three, sarg log analysis

The sarg software is used to analyze the logs of the squid service.
On the basis of the previous blog, do the Squid reverse proxy experiment. For the specific steps to install Squid, please refer to the previous blog link: Introduction to Squid Agent----Traditional Agent, Squid Agent---- -Professional introduction to transparent proxy mode
squid proxy setting method

3.1, sarg installation

//图像处理
yum -y install gd gd-devel httpd 

//解压安装包
tar zxvf sarg-2.3.7.tar.gz -C /opt/

cd /opt/sarg-2.3.7/
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \
--enable-extraprotection         #额外安全防护
 
make && make install

3.2, modify the configuration file

vi /etc/sarg/sarg.conf
对应行号去掉注释即开启
7 access_log /usr/local/squid/var/logs/access.log              ###squid的访问日志位置
25 title "Squid User Access Reports"                                     ###网页标题
120 output_dir /var/www/html/squid-reports                     ###分析报告的存放位置
178 user_ip no                                                                         ###不使用IP代替用户ID
184 topuser_sort_field BYTES reverse                                   ###升序排列
190 user_sort_field BYTES reverse                           //用户访问记录，连接此时，访问字节按降序排序
206 exclude_hosts /usr/local/sarg/noreport                        ###设置不生成报告的主机
257 overwrite_report no                                                              //同名日志是否覆盖
289 mail_utility mailx                                                               ###指定发邮件命令
434 charset UTF-8                                                                      //使用字符集
518 weekdays 0-6                                                                   ###指定top排序星期周期
523 hours 7-12,14,16,18-20                                                   ###指定top排序时间周期
633 www_document_root /var/www/html                           ###网页根目录

touch /usr/local/sarg/noreport            ###建立不生成报告的主机列表文件

ln -s /usr/local/sarg/bin/sarg /usr/local/bin/

//Start sarg log analysis and squid web service

yum install httpd -y 
root@squid sarg]# systemctl start httpd
[root@squid sarg]# systemctl start httpd
[root@squid sarg]# systemctl stop firewalld
[root@squid sarg]#

[root@squid sarg]# sarg
SARG: 纪录在文件: 237, reading: 100.00%
SARG: 成功的生成报告在 /var/www/html/squid-reports/2020Sep07-2020Sep07
[root@squid sarg]#

3.3, verification

3.3.1. Enter in the browser of the client machine: http://192.168.10.3/squid-reports/index.html

Change the browser address and re-visit the 192.168.100.3 webpage
Regenerate the report

sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/squid-reports/ -z -d $(date -d "1 day ago" +%d/%m/%Y)-$(date +%d/%m/%Y)

Check that one more record was generated

3.3.2, squid server set periodic tasks

crontab -e 
*/1 * * * * /usr/local/bin/sarg    #这里为了效果快速呈现出，设置了每隔1分钟

Four. Traditional proxy mode to build Squid reverse proxy

4.1 Basic configuration method

To implement the two most basic options for reverse proxy, modify squid.conf
1. Add the vhost option in the http_port configuration in squid.conf to support virtual host mapping

vi /etc/squid/squid.conf
http_port squid server IP vhost
2. Specify the web server host location
Use cache_peer configuration to specify the web server host location, the format is as follows:
cache_peer web server IP server type http port icp port [optional]
server type generally used The parent (parent server) icp port user connects to the adjacent ICP cache server

Server (usually another squid host), if 0 is not used; optional cache

Some attachment parameters, such as originserver, means that the server serves as the original

Host, weight=n specifies the priority of the server, n is an integer, the larger the value, the higher the priority

, The default is 1.; max-conn=n specifies the maximum number of connections from the reverse proxy host to the web server.

4.2 Specific placement

4.2.1 Configuration method

The reverse proxy uses port 80. Please note that port 80 should not conflict or be called by other programs

去掉透明代理设置反向代理通过80端口，访问虚拟主机，虚拟节点
http_port 192.168.100.3:80 accel vhost vport        
节点服务器1最大访问30，权重1，别名web1
cache_peer 192.168.100.4 parent 80 0 no-query originserver round-robin max-conn=30 weight=1 name=web1     
cache_peer 192.168.100.5 parent 80 0 no-query originserver round-robin max-conn=30 weight=1 name=web2
访问yun.com匹配web1，web2节点
cache_peer_domain web1 web2 www.yun.com

4.2.2 squid -k parse check syntax

4.2.3 Modify the client's local hosts file

4.2.4 Set the port of the browser proxy server to 80

4.2.5 Access test, the test is successful

5. Set up reverse proxy in transparent mode

5.1 Enable routing function

[root@squid network-scripts]# vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@squid network-scripts]# sysctl -p  //立即生效

5.2 Modify the configuration file

vim /etc/squid.conf

5.3 Clients use IP to access Squid

5.4 The client uses www.yun.com to access squid

5.4.1 Modify configuration file

Add a new line cache_peer_domain web1 web2 www.yun.com

vim /etc/squid.conf

http_port 80 accel vhost
cache_peer 192.168.100.4 parent 80 0 no-query originserver round-robin max-conn=30  weight=1 name=web1
cache_peer 192.168.100.5 parent 80 0 no-query originserver round-robin max-conn=30  weight=1 name=web2
cache_peer_domain web1 web2 www.yun.com
cache_effective_user squid
cache_effective_group squid
cache_mem 64 MB
reply_body_max_size 10 MB
maximum_object_size 4096 KB

5.4.2 Client Settings

Modify the local hosts file

5.4.3 Browser access to squid test

6. Squid server configuration file description

Squid's main configuration file is /etc/squid/squid.conf. All Squid settings are configured in this file. Let's explain the configuration options of this file.

http_port　3128　　　　　　//设置监听的IP与端口号

cache_mem 64 MB　　　　　　//额外提供给squid使用的内存，squid的内存总占用为 X * 10+15+“cache_mem”，其中X为squid的cache占用的容量（以GB为单位），
　　　　　　　　　　 　　　　//比如下面的cache大小是100M，即0.1GB，则内存总占用为0.1*10+15+64=80M，推荐大小为物理内存的1/3-1/2或更多。
maximum_object_size 4 MB 　　//设置squid磁盘缓存最大文件，超过4M的文件不保存到硬盘

minimum_object_size 0 KB 　　//设置squid磁盘缓存最小文件

maximum_object_size_in_memory 4096 KB 　　//设置squid内存缓存最大文件，超过4M的文件不保存到内存

cache_dir ufs /var/spool/squid 100 16 256 　　//定义squid的cache存放路径 、cache目录容量（单位M）、一级缓存目录数量、二级缓存目录数量

logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh        //log文件日志格式

access_log /var/log/squid/access.log combined　　//log文件存放路径和日志格式

cache_log /var/log/squid/cache.log 　　//设置缓存日志

logfile_rotate 60　　 //log轮循 60天

cache_swap_high 95　　//cache目录使用量大于95%时，开始清理旧的cache

cache_swap_low 90　　 //cache目录清理到90%时停止。

acl localnet src 192.168.1.0/24　　//定义本地网段

http_access allow localnet　　//允许本地网段使用

http_access deny all　　//拒绝所有

visible_hostname squid.david.dev　　//主机名

cache_mgr mchina_tang@qq.com　　//管理员邮箱