[linux beginner] curl command: download kaggle dataset with curl command

    I have just come into contact with the Linux server recently, but due to the fact that I cannot download the dataset directly on the server, I have not been able to run my own program on the server for a long time. I checked some ways to use wget, and kaggle's own API, but maybe I'm too lazy to do it. I didn't realize it was very easy until I saw this blog.

The following is reproduced from: https://blog.csdn.net/lujiandong1/article/details/50432789

https://www.kaggle.com/c/avito-context-ad-clicks/data

The data linked above on kaggle cannot be downloaded directly with the wget command.

wget https://www.kaggle.com/c/avito-context-ad-clicks/data command will be rejected.

http://askubuntu.com/questions/161778/how-do-i-use-wget-curl-to-download-from-a-site-i-am-logged-into Various solutions are covered on this blog , in fact, the most important thing is to cookie.txt.

 

Solution Download the file with curl command.

chrome can generate curl commands:

step:

For example, to download: Userinfo.tsv file

Step 1: Go to the download button, right-click to check. Switch to the Network option.

Step 2: Click the download button, and chrome will automatically monitor it for us.

Two UserInfo.tsv.7z appeared

Step 3: Right click on Copyan cURL.(bash)

Then copy the command to ubuntu and execute it, you can download the file

curl 'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/4438/UserInfo.tsv.7z?sv=2012-02-12&se=2015-12-10T16%3A26%3A05Z&sr=b&sp=r&sig=olOI7lwDgKKXFFYmpYXalphxKjFfDa5uQ5w7rIcMQGo%3D'-H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: zh-CN,zh;q=0.8'-H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1;WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73Safari/537.36' -H 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H'Referer: https://www.kaggle.com/c/avito-context-ad-clicks/data' -H'Connection: keep-alive' --compressed

The default is to output to standard output

curl'https://kaggle2.blob.core.windows.net/competitions-data/kaggle/4438/UserInfo.tsv.7z?sv=2012-02-12&se=2015-12-10T16%3A26%3A05Z&sr=b&sp=r&sig=olOI7lwDgKKXFFYmpYXalphxKjFfDa5uQ5w7rIcMQGo%3D'-H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: zh-CN,zh;q=0.8'-H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1;WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73Safari/537.36' -H 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H'Referer: https://www.kaggle.com/c/avito-context-ad-clicks/data' -H'Connection: keep-alive' --compressed -o 1.7z 重定向到1.7z