python3 urllib web download

Build a web download function

Environment: python3, module: python built-in module urllib

import urllib.request
import urllib.error

def get_html(url,user_agent='xxx',num_retries):
    headers = {
    
    'User-agent':user_agent}  #设置默认用户代理
    request = urllib.request.Request(url=url,headers=headers)  #添加请求头参数
    try:
        return urllib.request.urlopen(url=request).read().decode('utf-8')
    except urllib.error.URLError as e:
        print('Error:', e.reason) #抛出异常reason
        html = None
        if num_retries > 0:
        #4xx错误发生在请求存在问题时，5xx错误发生在服务器端存在问题时。只需保证在5xx时重新下载
            if hasattr(e, 'code') and  500 <= e.code < 600:# hasattr() 函数用于判断对象是否包含对应的属性。
                return download(url,user_agent,num_retries-1)
        return html

Note:
url: download URL
user_agent: user agent
num_retries: number of retry downloads
This function downloads web pages and returns HTML, catches exceptions, retry downloads for server-side errors, and sets user agents.