The python reptile of use beautifulsoup

I. Introduction of Beautiful Soup
　　simple terms, Beautiful Soup is a library of python, the most important function is to grab data from a web page. The official explanation: Beautiful Soup provides a simple, python-like function to handle navigation, search, modify tree analysis and other functions. It is a toolkit to provide needed data captured by the user to parse the document, because simple, so do not need much code to write a complete application. Beautiful Soup automatically converted to Unicode encoding input document, the document is converted to an output utf-8 encoded. You do not need to consider encoding, unless the document does not specify a code, then, Beautiful Soup can not automatically identify the encoding. Then, you just need to explain the original coding on it. Beautiful Soup has become and lxml, html6lib as good as the python interpreter, provide users with different analytical strategies or strong rate flexibility.

#安装 Beautiful Soup
pip install beautifulsoup4

#安装解析器
Beautiful Soup支持Python标准库中的HTML解析器,还支持一些第三方的解析器,其中一个是 lxml .根据操作系统不同,可以选择下列方法来安装lxml:

$ apt-get install Python-lxml

$ easy_install lxml

$ pip install lxml

另一个可供选择的解析器是纯Python实现的 html5lib , html5lib的解析方式与浏览器相同,可以选择下列方法来安装html5lib:

$ apt-get install Python-html5lib

$ easy_install html5lib

$ pip install html5lib

Third, the simple use of Beautiful Soup

'''
pip3 install beautifulsoup4  # 安装bs4
pip3 install lxml  # 下载lxml解析器
'''
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="sister"><b>$37</b></p>
<p class="story" id="p">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" >Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

# 从bs4中导入BeautifulSoup
from bs4 import BeautifulSoup

# 调用BeautifulSoup实例化得到一个soup对象
# 参数一: 解析文本
# 参数二:
# 参数二: 解析器（html.parser、lxml...）
soup = BeautifulSoup(html_doc, 'lxml')

print(soup)
print('*' * 100)
print(type(soup))
print('*' * 100)
# 文档美化
html = soup.prettify()
print(html)

Four, Beautiful Soup of traversing the document tree

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="sister"><b>$37</b></p>
<p class="story" id="p">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" >Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'lxml')

'''
遍历文档树：
    1、直接使用
    2、获取标签的名称
    3、获取标签的属性
    4、获取标签的内容
    5、嵌套选择
    6、子节点、子孙节点
    7、父节点、祖先节点
    8、兄弟节点
'''

# 1、直接使用
print(soup.p)  # 查找第一个p标签
print(soup.a)  # 查找第一个a标签

# 2、获取标签的名称
print(soup.head.name)  # 获取head标签的名称

# 3、获取标签的属性
print(soup.a.attrs)  # 获取a标签中的所有属性
print(soup.a.attrs['href'])  # 获取a标签中的href属性

# 4、获取标签的内容
print(soup.p.text)  # $37

# 5、嵌套选择
print(soup.html.head)

# 6、子节点、子孙节点
print(soup.body.children)  # body所有子节点，返回的是迭代器对象
print(list(soup.body.children))  # 强转成列表类型

print(soup.body.descendants)  # 子孙节点
print(list(soup.body.descendants))  # 子孙节点

#  7、父节点、祖先节点
print(soup.p.parent)  # 获取p标签的父亲节点
# 返回的是生成器对象
print(soup.p.parents)  # 获取p标签所有的祖先节点
print(list(soup.p.parents))

# 8、兄弟节点
# 找下一个兄弟
print(soup.p.next_sibling)
# 找下面所有的兄弟，返回的是生成器
print(soup.p.next_siblings)
print(list(soup.p.next_siblings))

# 找上一个兄弟
print(soup.a.previous_sibling)  # 找到第一个a标签的上一个兄弟节点
# 找到a标签上面的所有兄弟节点
print(soup.a.previous_siblings)  # 返回的是生成器
print(list(soup.a.previous_siblings))

Four, Beautiful Soup search of the document tree

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="sister"><b>$37</b></p>
<p class="story" id="p">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" >Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
'''
搜索文档树:
    find()  找一个  
    find_all()  找多个
    
标签查找与属性查找:
    标签:
            name 属性匹配
            attrs 属性查找匹配
            text 文本匹配
            
        - 字符串过滤器   
            字符串全局匹配

        - 正则过滤器
            re模块匹配

        - 列表过滤器
            列表内的数据匹配

        - bool过滤器
            True匹配

        - 方法过滤器
            用于一些要的属性以及不需要的属性查找。

    属性:
        - class_
        - id
'''

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'lxml')

# # 字符串过滤器
# name
p_tag = soup.find(name='p')
print(p_tag)  # 根据文本p查找某个标签
# # 找到所有标签名为p的节点
tag_s1 = soup.find_all(name='p')
print(tag_s1)
#
#
# # attrs
# # 查找第一个class为sister的节点
p = soup.find(attrs={"class": "sister"})
# print(p)
# # 查找所有class为sister的节点
tag_s2 = soup.find_all(attrs={"class": "sister"})
print(tag_s2)


# text
text = soup.find(text="$37")
print(text)
#
#
# # 配合使用:
# # 找到一个id为link2、文本为Lacie的a标签
a_tag = soup.find(name="a", attrs={"id": "link2"}, text="Lacie")
print(a_tag)



# # 正则过滤器
import re
# name
p_tag = soup.find(name=re.compile('p'))
print(p_tag)

# 列表过滤器
import re
# name
tags = soup.find_all(name=['p', 'a', re.compile('html')])
print(tags)

# - bool过滤器
# True匹配
# 找到有id的p标签
p = soup.find(name='p', attrs={"id": True})
print(p)

# 方法过滤器
# 匹配标签名为a、属性有id没有class的标签
def have_id_class(tag):
    if tag.name == 'a' and tag.has_attr('id') and tag.has_attr('class'):
        return tag

tag = soup.find(name=have_id_class)
print(tag)

The python reptile of use beautifulsoup

Guess you like