wget crawl website pages

The corresponding installation named

yum -y install wget

yum -y install setup

yum -y install perl

wget -r -p -np -k -E http://www.xxx.com whole gripping station

wget -l 1 -p -np -k http://www.xxx.com gripping a first stage

-r recursive crawl
after crawl -k amendment links for local browsing

http://blog.sina.com.cn/s/blog_669fb0c3010137bq.html

wget -m -e robots = off -k -E "http://www.abc.net/"
can be downloaded to the station's local current working directory, generate accessible, complete image.

Explained:

-m // mirror, the whole point is to grab
-e robots = off // ignore the robots protocol, mandatory, rogue crawl
-k // convert absolute URL link to the local relative URL
-E // all text / html documents with .html extension save

Guess you like

Origin www.cnblogs.com/aibabel/p/11700762.html