wget common
1. wget -O modify the download file name to the new name
E.g
wget -O zlib.tar.gz http://zlib.net/zlib-1.2.11.tar.gz
2. wget can only deal with a user name / password way to limit access to the site, you can use two parameters:
-http-user = USER HTTP user settings
-http-passwd = PASS Set HTTP password
The need to do certificate certified sites, you can only use the other download tools, such as curl.
E.g
wget --user = username --password = password http://abc.com/abc.tar.gz
See detailed Reprinted from https://www.cnblogs.com/cindy-cindy/p/6847502.html article
Linux wget to download files is a tool that is used in the command line. For Linux users is an essential tool, especially for network administrators often have to download some software or restore the backup to the local server from a remote server. If we use the virtual host to handle such matters we can only start with the remote server to download to our computer disk, then use the ftp tool to upload to the server. This will not only waste of time and energy, it is not no way to do. And to the Linux VPS, you can download it directly to the server without going through the uploading this step. wget tools small but functional, it supports breakpoint download function, supports FTP and HTTP download, support proxy servers and set up convenient and simple. Below we explain in the form of examples of how to use wget.
1, using wget to download a single file
The following example is downloading a file from the network and stored in the current directory
wget http://cn.wordpress.org/wordpress-3.1-zh_CN.zip
Progress bar is displayed in the download process, comprising (percentage of download completed, the bytes have been downloaded, the current download speed, remaining download time).
2, using wget -O download and save the file with a different name
wget will default to the last line with "/" behind the characters to command, usually for the download file name dynamically linked incorrectly.
Error:? The following example will download a file name and save download.php id = 1080
wget http://www.centos.bz/download?id=1
Even if the downloaded file is zip format, it still download.php? Id = 1080 command.
Correct: To solve this problem, we can use the -O parameter to specify a file name:
wget -O wordpress.zip http://www.centos.bz/download.php?id=1080
3, download speed using wget -limit -rate
When you execute wget, which by default will take all possible broadband download. But when you're ready to download a large file, but you also need to download other files it is necessary to speed up.
wget –limit-rate=300k http://cn.wordpress.org/wordpress-3.1-zh_CN.zip
4, using wget -c HTTP
Use wget -c to restart interrupted file download:
wget -c http://cn.wordpress.org/wordpress-3.1-zh_CN.zip
For downloading large files when we suddenly interrupted due to network and other very helpful, then we can continue to download instead of re-downloading a file. You can use -c parameters need to continue interrupted downloads.
5, background downloading using wget -b
For download very large files, we can use the parameter -b for background downloading.
wget -b http://cn.wordpress.org/wordpress-3.1-zh_CN.zip
Continuing in background, pid 1840.
Output will be written to `wget-log’.
You can use the following command to check the download progress
tail -f wget-log
6, camouflage agent name download
Some sites can refuse your request to download at the discretion of the proxy name instead of the browser. But you can -user-agent masquerading parameters.
wget –user-agent=”Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16″ 下载链接
7, using wget -spider test download link
When you plan time to download, you should test the download link is valid for a predetermined time. We can increase -spider parameters to be checked.
wget –spider URL
If the correct download link will be displayed
wget –spider URL
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled — not retrieving.
This ensures that the download can be performed at the scheduled time, but when you give a wrong link, the following error will be displayed
wget –spider url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response… 404 Not Found
Remote file does not exist — broken link!!!
You can use the following circumstances spider parameters:
Before downloading a timing check
The break detect whether the site is available
Dead links checking website pages
8, using the number of retries increases wget -tries
If a network problem or download a large file may also fail. wget default retry connection to download the file 20 times. If necessary, you can use -tries increase the number of retries.
wget –tries=40 URL
9, download multiple files using wget -i
First, keep a file download link
cat > filelist.txt
Urll
url2
url3
url4
Then use this file to download and parameter -i
wget -i filelist.txt
10, using wget -mirror mirror sites
The following example is to download the entire site to a local.
wget –mirror -p –convert-links -P ./LOCAL URL
-miror: Account Mirror Download
-p: download all html page to display the normal file
-convert-links: After the download, conversion costs link
-P ./LOCAL: Save all files and directories to a local specified directory
11, using wget -reject filter specified format download
Do you want to download a website, but you do not want to download the image, you can use the following command.
wget –reject=gif url
12, using wget -o to download information into the log file
You do not want to download information directly but in a log file, you can use the following command in the terminal:
wget -o download.log URL
13, using wget -Q limit the total download file size
When the file you want to download more than 5M exit download, you can use the following command:
wget -Q5m -i filelist.txt
Note: This parameter does not work on a single file to download, if only recursive downloads effective.
14, using wget -r -A download the specified format file
You can use this function in the following cases
All the pictures to download a website
All video to download a website
All PDF files downloaded a Web site
wget -r -A.pdf url
15, using wget FTP download
You can use wget to download the complete ftp links.
Anonymous ftp download using wget
wget ftp-url
Use wget user name and password authentication ftp download
wget –ftp-user=USERNAME –ftp-password=PASSWORD url
wget is open source software developed under Linux, the author is Hrvoje Niksic, it was later ported to various platforms including Windows. It has the following functions and features:
(1) supports the transfer function breakpoint; this is the network FlashGet ants and then the biggest selling point, now, Wget can use this feature, those networks are not very good user can rest assured;
(2) supports both FTP and HTTP download method; although most of the software now can download using HTTP, however, in some cases, still you need to use FTP to download software;
(3) support proxy server; for high security strength of the system, your system will generally not directly exposed on the Internet, so the support agent is download the software must be some function;
(4) Easy setup and simple; probably, user habits graphical interface is not too accustomed to the command line, but, in fact, the command line has more advantages on the set, at least, the mouse may be less a lot of times, do not worry about whether or not wrong mouse;
(5) a small program, completely free of charge; small program can be considered not count, because the hard disk is too great; completely free of charge would have to consider, even if there are many so-called free software on the Internet, but the software is not advertising we like;
wget although powerful, but it is quite simple to use, the basic syntax is: wget [parameter list] URL. Here's combined specific example to illustrate the use of wget.
1, download the entire http or ftp site.
wget http://place.your.url/here
This command can be http: //place.your.url/here Home download. Use -x forces establish exactly the same directory on the server, if -nd parameters, then download all the content on the local server will be added to the current directory.
wget -r http://place.your.url/here
This command will follow the recursive method, all the directories and files on the download server, in essence, is to download the entire site. This command must be used with caution, because the download time, all addresses are directed to download sites will also be downloaded, so if this site refer to other sites, then the referenced website will be downloaded! For this reason, this parameter is not used. You can use -l number parameter to specify the download level. For example, download only two, then use the -l 2.
If you want to make mirror sites, you can use the -m parameter, for example: wget -m http: //place.your.url/here
Wget time will automatically determine the appropriate parameters to produce mirror sites. At this point, wget will log on to the server, loaded with a predetermined press robots.txt robots.txt to perform.
2, HTTP.
When the file is particularly large or particularly slow network, it is often a file has not been downloaded, the connection has been cut off, then you need HTTP. The HTTP wget is automatic, -c parameters only need to use, for example:
wget -c http://the.url.of/incomplete/file
Use HTTP server requires support for HTTP. -t parameter indicates the number of retries, for example, need to retry 100 times, then write -t 100, if set to -t 0, it represents the infinite retries until the connection is successful. -T parameter indicates the timeout for, e.g. -T 120, represents the waiting 120 seconds, even if the connection is not timed out.
3, bulk download.
If you have multiple files to download, you can generate a file, the URL of each file write a line, for example, generates a file download.txt, then use the command: wget -i download.txt
This will put each URL download.txt which lists are downloaded. (If the column is a file downloaded file, if the column is a website, then download the home page)
4, selective download.
You can specify that only wget to download a class file, or the file does not download anything. E.g:
wget -m –reject=gif http://target.web.site/subdirectory
For downloads http://target.web.site/subdirectory, but ignored gif file. -accept = LIST acceptable file types, -reject = LIST refuse to accept the type of file.
5, password, and authentication.
wget can only deal with a user name / password way to limit access to the site, you can use two parameters:
-http-user = USER HTTP user settings
-http-passwd = PASS Set HTTP password
The need to do certificate certified sites, you can only use the other download tools, such as curl.
6, the use of a proxy server to download.
If the user's network need to go through a proxy server, you can make wget to download files through a proxy server. At this point you need to create a user .wgetrc file in the current directory. Files can set a proxy server:
http-proxy = 111.111.111.111:8080
ftp-proxy = 111.111.111.111:8080
Respectively, the proxy server and ftp http proxy server. If the proxy server requires a password, use:
-proxy-user = USER set user agent
-proxy-passwd = PASS set the proxy password
These two parameters.
Parameter -proxy = on / off or off using the proxy.
wget There are many useful features, users need to dig.
appendix:
Format:
wget [parameter list] [target software, web URL]
-V, -version Display software version number and then exit;
-h, -help display help information software;
-e, -execute = COMMAND execute a ".wgetrc" command
-o, -output-file = FILE software will save the output information to a file;
-a, -append-output = FILE output software appended to the file;
-d, -debug display output information;
-q, -quiet information not to be displayed;
-i, -input-file = FILE get URL from the file;
-t, -tries = NUMBER whether downloads (0 for infinite times)
-O -output-document = FILE save the downloaded file as a different file name
-nc, -no-clobber do not overwrite existing files
-N, -timestamping only downloads new files than local
-T, -timeout = SECONDS set timeout
-Y, -proxy = on / off closing Agent
-nd, -no-directories do not create directories
-x, -force-directories force a directory
-http-user = USER HTTP user settings
-http-passwd = PASS Set HTTP password
-proxy-user = USER set user agent
-proxy-passwd = PASS set the proxy password
-r, -recursive download the entire Web site, directory (use caution)
-l, -level = NUMBER download level
-A, -accept = LIST acceptable file types
-R, -reject = LIST rejected file types
-D, -domains = LIST acceptable domain name
-exclude-domains = LIST rejected domain
-L, -relative download related link
-follow-ftp download FTP link only
-H, -span-hosts can be downloaded outside of the host
-I, -include-directories = LIST allowed directory
-X, -exclude-directories = LIST rejection directory
Chinese document name will be encoded in ordinary circumstances, but in -cut-dirs time for the normal,
wget -r -np -nH –cut-dirs=3 ftp://host/test/
Test .txt
wget -r -np -nH -nd ftp://host/test/
%B4%FA%B8%D5.txt
wget “ftp://host/test/*”
%B4%FA%B8%D5.txt
Due to unknown reasons, possibly to avoid a special file name, wget automatically gripping portion is treated with a filename encode_string, so that the patch is put encode_string processed into "% 3A" such things, reduction with decode_string to ":", and apply in part of the directory and file name, decode_string is wget built-in functions.
wget -t0 -c -nH -x -np -b -m -P /home/sunny/NOD32view/ http://downloads1.kaspersky-labs.com/bases/ -o wget.log
Use wget to download a single file