what is curl
curl is a data transfer tool. It's available as a development library and a terminal-based cli, both of which have the same engine (the fact is that the cli tool is just a program under the development library).
curl works with every protocol you might use.
what curl can do
Everything related to data transfer (can be done with curl). Everyone has used a browser (say you are using a browser to read this article), all a browser does is request a page, and get a response, it can write and read cookies, and then render (display content, images and execute js script).
Curl can do everything a browser does, except for the final rendering step (since it has nothing to do with data transfer).
In summary, curl can download html pages, fill and submit html forms, download and upload files from FTP/HTTP servers, and read and write cookies .
This makes it an excellent tool for things like scripting, debugging, and forensic analysis.
curl command instance
Let's see what we can do with curl
1. Get a response from the server
Everything returned on the server is a response to a request. So fetching an HTML page is the same as downloading a file.
For example, to get an html response from a URL:
curl http://info.cern.ch/Get a list of articles from the server:
curl https://jsonplaceholder.typicode.com/postsNow that we know how to get a response from the server, we can download the file (such as the google logo):
curl https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.pngThe above command will dump binary image data that you cannot view in the terminal, you need to save them first, and then view them with a photo viewer (FQ is required for domestic access to google).
Note: The position of the options in the command is not in strict order, so if you put the option at the end, and the example puts it at the beginning, you don't need to worry about it.
2. Save the file with the default file name
Every file on the Internet will have a filename, to save the downloaded file with its original name, use the -O option:
curl -O http://www.google.com/robots.txt3. Save the file with a custom name
To save the file with a custom name, use the -o option followed by a (strictly) custom name:
curl -o http://www.google.com/robots.txt googleRobots.txt4. Download multiple files
To download multiple files, separate them with spaces:
curl url1 url2 url3If you want to use the -O option for all urls, you can do this:
curl url1 url2 url3 -O -O -OThe same method can be performed for any option. The first option is for the first url, the second option is for the second url, and so on.
5. Download a series of files
curl has the function of downloading a series of files from the server, see the following example:
curl http://www.google.com/logo/logo[1-9].pngThe above command downloads logo1,png, logo2.png, logo3.png, all the way to logo9.png.
6. Only download the latest version
Download a file only if its modification time is later than a given time:
curl url -z "DD MMM YY MM:HH:SS"7. Continue to download
If a file has been partially transferred, the transfer can be resumed with the -C option. The offset at which the transfer needs to continue should be passed as an argument to the -C flag.
curl -C 1024 http://seeni.linuxhandbook.org/files/largeFile.mpv -O8. Upload files
To upload a file to the server, use the -T option followed by the local file path:
curl -T uploadFile.txt http://upload.linuxhandbook.org/files9. Delete files
To delete a file named deleteFile.txt, use the -X option, which can be used with any HTTP method (such as GET, POST, PUT, DELETE, PATCH). Most FTP servers will be configured with a DELETE method if there is no HTTP method.
curl -X DELETE http://upload.linuxhandbook.org/files/deleteFile.txtYou can also modify the above command to perform corresponding tasks for any HTTP method. For example, if your server allows the TRUNCATE method (a created method, not standard) to delete only the contents of the file, but not the file itself, you can use a command like this:
curl -X TRUNCATE http://upload.linuxhandbook.org/files/mysql.dumpThe above mentioned are the main uses of curl. (During use) you may encounter some difficulties, such as redirection, user identity verification, SSL certificates, etc., we call them add-ons because they are optional, but still for some operations very important. In the following sections, we'll discuss these additions and how to use curl to handle them.
10. Avoid redirects
When you request the address http://www.google.com, you may get www.google.co.in, because of redirection of the address (HTTP packets with status codes in the range 300-399).
You can use the -L option to avoid redirection:
curl -L http://www.google.com11. Authentication
When a server is configured to serve only specific individuals with credentials, they are given a username and password (i.e. require a login to access). You can use the -u option:
curl -u username:password http://seeni.linuxhandbook.org/files/tasks.txt12. Data Transfer Restrictions
If you want to impose a data transfer limit, you can use the --limit-rate option. The following command attempts to limit the download rate to 10k:
curl --limit-rate 10K http://seeni.linuxhandbook.org/files/logoDetails.tgz13. Show/hide transfer status
If the response is performed from the terminal, such as upload, download, then curl will automatically display the status of the transfer (progress bar). If you don't want to see the progress bar, just add the -s option after the command, so that the progress will not be displayed on the terminal.
14. Ignore SSL certificate
Remember when some websites needed to be granted security certificates when they were accessed? If you trust the address and want data transfer to take place, you can use the -k option to ignore SSL certificate verification:
curl -k https://notSoSecure.org/files/logoDetails.tgz15. Get information header
To display headers during data transfer, use the -i option:
curl -i http://www.google.com/robots.txt16. Show header only
If you only want the headers, but not the data, use the -I option:
curl -I http://www.google.com/robots.txt17. Change user agent
Some websites or servers do not allow certain devices to access the system. But how do they know what equipment we use? This is told by the User-Agent HTTP header information. We can use the -A option to change the User Agent:
curl -A "Mozilla FireFox(42.0)" http://notAllowedForCLI.sites.org/randomFile.png18. Send data to server
If the server needs some parameter data, such as token or API key, you can use the -d option to send the data. In the command line, the data to be sent needs to follow the -d option, and & can be used to combine multiple data. This is usually done in the browser with a GET or POST request, which is one of the ways you send form information.
curl -d "token=34343abvfgh&name='seeni'" http://api.restful.org/getcontent19. Write cookies to file
Cookies are small pieces of information that allow sessions to be maintained in the stateless HTTP protocol. If you want to know more about cookies, you can visit Using HTTP cookies (https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies).
To save cookies to a file, use the -c option followed by the cookie filename:
curl -c googleCookie.txt http://www.google.com/files20. Read cookies from file
To read cookies from a file, use the -b option followed by the cookie filename:
curl -b googleCookie.txt http://www.google.com/filesNote that the -b option only reads cookies from the file, if the server resends another cookie, you need to use the -c option to overwrite them.
21. Start a new session
If you want to discard cookies to start a new session, use the -j option. After using this option, even if you read the cookie through -b, it will start a new session.
curl -b googleCookie.txt http://www.google.com/files -j