版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/chengqiuming/article/details/86407377
一 点睛
直接调用节点的名称就可以选择节点元素，再调用string属性就可以得到节点内的文本了，这种选择方式速度非常快。如果单个节点结构层次非常清晰，可以选用这种方式来解析。
二 选择元素
1 代码
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.title)
print(type(soup.title))
print(soup.title.string)
print(soup.head)
print(soup.p)2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
<head><title>The Dormouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>3 说明
这里依然选用刚才的HTML代码，首先打印输出title节点的选择结果，输出结果正是title节点加里面的文字内容。接下来，输出它的类型，是bs4.element.Tag类型，这是Beautiful Soup中一个重要的数据结构。经过选择器选择后，选择结果都是这种Tag类型。Tag具有一些属性，比如string属性，调用该属性，可以得到节点的文本内容，所以接下来的输出结果正是节点的文本内容。
接下来，我们又尝试选择了head节点，结果也是节点加其内部的所有内容。最后，选择了p节点。不过这次情况比较特殊，我们发现结果是第一个p节点的内容，后面的几个p节点并没有选到。也就是说，当有多个节点时，这种选择方式只会选择到第一个匹配的节点，其他的后面节点都会忽略。
三 提取信息
1 获取名称
1.1 代码
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
# name属性获取节点的名称。
# 选取title节点，然后调用name属性就可以得到节点名称
print(soup.title.name)
# 每个节点可能有多个属性，比如p节点有id和class属性
# 调用attrs获取所有属性
# attrs的返回结果是字典形式，它把选择的节点的所有属性和属性值组合成一个字典。
print(soup.p.attrs)
# 要获取name属性，就相当于从字典中获取某个键值，只需要用中括号加属性名就可以了
print(soup.p.attrs['name'])
# 还有一种更简单的获取方式：可以不用写attrs，
# 直接在节点元素后面加中括号，传入属性名就可以获取属性值了。
# 这里需要注意的是，有的返回结果是字符串，有的返回结果是字符串组成的列表。
# 比如，name属性的值是唯一的，返回的结果就是单个字符串。
# 而对于class，一个节点元素可能有多个class，所以返回的是列表。
# 在实际处理过程中，我们要注意判断类型。
print(soup.p['name'])
print(soup.p['class'])
# string属性获取节点元素包含的文本内容，比如要获取第一个p节点的文本
print(soup.p.string)1.2 运行结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
title
{'class': ['title'], 'name': 'dromouse'}
dromouse
dromouse
['title']
The Dormouse's story四 嵌套选择
1 点睛
可以继续调用节点进行下一步的选择。比如，我们获取了head节点元素，我们可以继续调用title来选取head节点下的title元素。
2 代码
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.head.title)
print(type(soup.head.title))
print(soup.head.title.string)3 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story4 说明
第一行结果是调用head之后再次调用title而选择的title节点元素。然后打印输出了它的类型，可以看到，它仍然是bs4.element.Tag类型。也就是说，我们在Tag类型的基础上再次选择得到的依然还是Tag类型，每次返回的结果都相同，所以这样就可以做嵌套选择了。
最后，输出它的string属性，也就是节点里的文本内容。
五 关联选择
1 点睛
在做选择的时候，有时候不能做到一步就选到想要的节点元素，需要先选中某一个节点元素，然后以它为基准再选择它的子节点、父节点、兄弟节点等，这里就来介绍如何选择这些节点元素。
2 子节点实现方法一
2.1 代码
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
# 选取节点元素之后，如果想要获取它的直接子节点，可以调用contents属性
print(soup.p.contents)2.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
['\n Once upon a time there were three little sisters; and their names were\n ', <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>, '\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' \n and\n ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, '\n and they lived at the bottom of a well.\n ']2.3 说明
可以看到，返回结果是列表形式。p节点里既包含文本，又包含节点，最后会将它们以列表形式统一返回。
需要注意的是，列表中的每个元素都是p节点的直接子节点。比如第一个a节点里面包含一层span节点，这相当于孙子节点了，但是返回结果并没有单独把span节点选出来。所以说，contents属性得到的结果是直接子节点的列表。
3 子节点实现方法二
3.1 代码
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
print(i, child)3.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<list_iterator object at 0x0000022D91094E80>
0
Once upon a time there were three little sisters; and their names were
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2
3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4
and
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6
and they lived at the bottom of a well.3.3 说明
还是同样的HTML文本，这里调用了children属性来选择，返回结果是生成器类型。接下来，我们用for循环输出相应的内容。
4 子孙节点实战
4.1 代码
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.descendants)
for i, child in enumerate(soup.p.descendants):
print(i, child)4.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<generator object descendants at 0x00000147EFE1C0A0>
0
Once upon a time there were three little sisters; and their names were
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2
3 <span>Elsie</span>
4 Elsie
5
6
7 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
8 Lacie
9
and
10 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
11 Tillie
12
and they lived at the bottom of a well.4.3 说明
此时返回结果还是生成器。遍历输出一下可以看到，这次的输出结果就包含了span节点。descendants会递归查询所有子节点，得到所有的子孙节点。
5 父节点
5.1 代码
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
# 获取某个节点元素的父节点，可以调用parent属性
print(soup.a.parent)5.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>6 祖父节点
6.1 代码
html = """
<html>
<body>
<p class="story">
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(type(soup.a.parents))
# 想获取所有的祖先节点，可以调用parents属性
print(list(enumerate(soup.a.parents)))6.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
<class 'generator'>
[(0, <p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>), (1, <body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body>), (2, <html>
<body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body></html>), (3, <html>
<body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body></html>)]6.3 说明
可以发现，返回结果是生成器类型。这里用列表输出了它的索引和内容，而列表中的元素就是a节点的祖先节点。
7 兄弟节点
7.1 代码
html = """
<html>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
Hello
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling', soup.a.next_sibling)
print('Prev Sibling', soup.a.previous_sibling)
print('Next Siblings', list(enumerate(soup.a.next_siblings)))
print('Prev Siblings', list(enumerate(soup.a.previous_siblings)))7.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
Next Sibling
Hello
Prev Sibling
Once upon a time there were three little sisters; and their names were
Next Siblings [(0, '\n Hello\n '), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ' \n and\n '), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, '\n and they lived at the bottom of a well.\n ')]
Prev Siblings [(0, '\n Once upon a time there were three little sisters; and their names were\n ')]7.3 说明
这里调用了4个属性，其中next_sibling和previous_sibling分别获取节点的下一个和上一个兄弟元素，next_siblings和previous_siblings则分别返回所有后面和前面的兄弟节点的生成器。
8 提取信息
8.1 代码
html = """
<html>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Bob</a><a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling:')
print(type(soup.a.next_sibling))
print(soup.a.next_sibling)
print(soup.a.next_sibling.string)
print('Parent:')
print(type(soup.a.parents))
print(list(soup.a.parents)[0])
print(list(soup.a.parents)[0].attrs['class'])8.2 结果
E:\WebSpider\venv\Scripts\python.exe E:/WebSpider/4_2.py
Next Sibling:
<class 'bs4.element.Tag'>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
Lacie
Parent:
<class 'generator'>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Bob</a><a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
</p>
['story']8.3 说明
如果返回结果是单个节点，那么可以直接调用string、attrs等属性获得其文本和属性；如果返回结果是多个节点的生成器，则可以转为列表后取出某个元素，然后再调用string、attrs等属性获取其对应节点的文本和属性。