Linux Shell provides two very useful commands to crawl web pages, which are wget and curl
curl and wget use the proxy
curl 支持 http、https、socks4、socks5
wget supports http, https
Agent Example:
#!/bin/bash
#
# curl 支持 http、https、socks4、socks5
# wget 支持 http、https
#
# 米扑代理示例：
# http://proxy.mimvp.com/demo2.php
#
# 米扑代理购买：
# http://proxy.mimvp.com
#
# mimvp.com
# 2015-11-09
# http代理格式 http_proxy=http://IP:Port
# https代理格式 https_proxy=http://IP:Port
{'http': 'http://120.77.176.179:8888'}
curl -m 30 --retry 3 -x http://120.77.176.179:8888 http://proxy.mimvp.com/exist.php # http_proxy
wget -T 30 --tries 3 -e "http_proxy=http://120.77.176.179:8888" http://proxy.mimvp.com/exist.php # http_proxy
{'https': 'http://46.105.214.133:3128'}
curl -m 30 --retry 3 --proxy-insecure -x http://46.105.214.133:3128 -k https://proxy.mimvp.com/exist.php # https_proxy
wget -T 30 --tries 3 --no-check-certificate -e "https_proxy=http://46.105.214.133:3128" https://proxy.mimvp.com/exist.php # https_proxy
# curl 支持socks
{'socks4': '101.255.17.145:1080'}
curl -m 30 --retry 3 --socks4 101.255.17.145:1080 http://proxy.mimvp.com/exist.php
{'socks5': '82.164.233.227:45454'}
curl -m 30 --retry 3 --socks5 82.164.233.227:45454 http://proxy.mimvp.com/exist.php
# wget 不支持socks<br/>
wget profile settings proxy
vim ~/.wgetrc
http_proxy=http://120.77.176.179:8888:8080
https_proxy=http://12.7.17.17:8888:8080
use_proxy = on
wait = 30
wget -T 30 --tries 3 http://proxy.mimvp.com<br/>
Shell set up temporary local agent
# proxy no auth
export http_proxy=http://120.77.176.179:8888:8080
export https_proxy=http://12.7.17.17:8888:8080
# proxy auth
export http_proxy=http://username:[email protected]:8888:8080
export https_proxy=http://username:[email protected]:8888:8080
# 取消设置
unset http_proxy
unset https_proxy<br/>
Shell set the system global proxy
# 修改 /etc/profile，保存并重启服务器
sudo vim /etc/profile # 所有人有效
或
sudo vim ~/.bashrc # 所有人有效
或
vim ~/.bash_profile # 个人有效
# proxy no auth
export http_proxy=http://120.77.176.179:8888:8080
export https_proxy=http://12.7.17.17:8888:8080
# proxy auth
export http_proxy=http://username:[email protected]:8888:8080
export https_proxy=http://username:[email protected]:8888:8080
source /etc/profile
或
source ~/.bashrc
或
source ~/.bash_profile
sudo reboot<br/>
Acting meters flutter example
M flutter proxy example, contains about a dozen programming languages or scripts Python, Java, PHP, C # , Go, Perl, Ruby, Shell, NodeJS, PhantomJS, Groovy, Delphi, easy language and so on, through a large number of instances can run, explain in detail the correct way to use a proxy IP for easy web crawling, data acquisition, automated testing, and other fields. Acting meters flutter example, tests using proxy IP, all from meters flutter agents, covering more than 120 countries and 34 Chinese provinces and cities, support for http, https, socks4, socks5, etc. meters flutter agents example the official website: HTTP: // proxy.mimvp.com/demo2.php
<br/>
Reference Recommended :
Example of use of m flutter agents (recommended)
mimvp-Proxy-Demo (GitHub)
the Linux crawl pages instance (the shell awk +)
LinuxIP agent screening system (shell + proxy)