request to download the gitlab project file

Unfortunately, the company's gitlabAPI permission authorization problem could not be solved for a long time, so I came to simulate the login. In the last article, selenium was used to log in and download the files on gitlab, but it was bloated, so let's use requests again to solve it.

First refer to this article:

https://yq.aliyun.com/articles/252128?spm=a2c4e.11155435.0.0.23ad75b5w4duc6

Later, I found out that it doesn't work on gitlab, and it always reports an error:

Traceback (most recent call last):
  File "D:/Users/EX-FEIXIAOHUI001/PycharmProjects/untitled/requests_gitlab.py", line 47, in <module>
    response = session.post(session_url, headers=session_header, data=login_data)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 549, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 634, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 134, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects, response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

The reason given is that there are too many redirects.

Then try adding this:

'Referer':'http://pagit.paic.com.cn/'

It worked. Referer was originally the word referrer, but now it will be wrong. This thing means that the current request is coming from this address. After reading the information on the Internet, I felt that this thing is not very useful, so I did not study it in depth.

Finally, the source code is attached:

import requests
import re

login_url = 'http://pagit.paic.com.cn/users/sign_in?redirect_to_referer=yes'
username = 'EX-FEIXIAOHUI001'
password = '*****'

user_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'zh-CN,zh;q=0.8',
	'Content-Type': 'application/x-www-form-urlencoded'
}

session_header = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'zh-CN,zh;q=0.8',
	# 'Upgrade-Insecure-Requests':'1',
	'Referer':'http://pagit.paic.com.cn/'
}

session = requests.Session()
response = session.get(login_url,headers=user_headers)
pattern = re.compile(r'<input name="authenticity_token" type="hidden" value="(.*)" />')
# print response.content
authenticity_token = pattern.findall(response.content)[0]
login_data = {
    'utf8': '%E2%9C%93',
    'authenticity_token': authenticity_token,
	'username': username,
    'password': password
}
session_url = "http://pagit.paic.com.cn/users/auth/ldapmain/callback"
response = session.post(session_url, headers=session_header, data=login_data)

zip_url = 'http://pagit.paic.com.cn/EX-FEIXIAOHUI001/project_config/repository/archive.zip'
r = session.get(zip_url)
with open("demo.zip",'wb') as code:
	code.write(r.content)