Beautiful Soup借助网页结构和属性等特征来解析网页
Beautiful Soup在解析时实际上依赖解析器，它除了支持Python标准库中的HTML解析器外，还支持一些第三方解析器（比如lxml）。如果使用lxml。那么在初始化Beautiful Soup时，可以把第二个参数改为lxml即可
from bs4 import BeautifulSoup
soup=BeautifulSoup('<p>Hello</p>','lxml')
print(soup.p.string)
输出：Hello
基本用法
from bs4 import BeautifulSoup
html="""
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!--Elsie--></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup=BeautifulSoup(html,'lxml')
print(soup.prettify())
print(soup.title.string)
首先html它是一个HTML字符串，但并不完整，因为body和html没有闭合。
在调用BeautifulSoup时就自动更正了格式
调用prettify()方法可以把要解析的字符串以标准的缩进格式输出。
输出：
<html>
<head>
<title>
The Dormouse's story
</title>
</head>
<body>
<p class="title" name="dromouse">
<b>
The Dormouse's story
</b>
</p>
<p class="story">
Once upon a time there were three were three little sisters;and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<!--Elsie-->
</a>
,
<a class="sister" href="http://example.com/lacie" id="link2">
Lacie
</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">
Tillie
</a>
;
and they lived at the bottom of a well.
</p>
<p class="story">
...
</p>
</body>
</html>
The Dormouse's story
节点选择器
直接调用节点的名称就可以选择节点元素，再调用string属性就可以得到节点内的文本了。
from bs4 import BeautifulSoup
soup=BeautifulSoup(html,'lxml')
print(soup.title)
print(type(soup.title))
print(soup.title.string)
print(soup.head)
print(soup.p)
输出：
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
<head><title>The Dormouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
提取信息
（1）获取名称
利用name属性获取节点的名称
print(soup.title.name)
输出：title
（2）获取属性
可以调用attrs获取所有属性
print(soup.p.attrs)
print(soup.p.attrs['name'])
输出：
{'class': ['title'], 'name': 'dromouse'}
dromouse
也可以不用写attrs，直接在节点元素后面加中括号，传入属性名就可以获取属性值了。
print(soup.p["name"])
print(soup.p["class"])
（3）获取内容
使用string属性获取节点元素包含的文本内容
print(soup.p.string)
输出：The Dormouse’s story
嵌套选择
print(soup.head.title)
print(type(soup.head.title))
print(soup.head.title.string)
输出：
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
关联选择
（1）子节点和子孙节点
选取节点元素之后，如果想要获取它的直接子节点，可以调用contents属性。
html="""
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="story">Once upon a time there were three were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!--Elsie--></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup=BeautifulSoup(html,'lxml')
print(soup.p.contents)
输出：
['Once upon a time there were three were three little sisters;and their names were\n', <a class="sister" href="http://example.com/elsie" id="link1"><!--Elsie--></a>, ',\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' and\n', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, ';\nand they lived at the bottom of a well.']
contents属性得到的结果是直接子节点的列表，可以调用children属性得到相应的结果。
for i,child in enumerate(soup.p.children):
print(i,child)
输出：
0 Once upon a time there were three were three little sisters;and their names were
1 <a class="sister" href="http://example.com/elsie" id="link1"><!--Elsie--></a>
2 ,
3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4 and
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 ;
and they lived at the bottom of a well.
想要得到所有的子孙节点，可以调用descendants属性
（2）父节点和祖先节点
如果要获取某个节点元素的父节点，可以调用parent属性
html="""
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="story">Once upon a time there were three were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>,
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup=BeautifulSoup(html,'lxml')
print(soup.a.parent)
输出：
<p class="story">Once upon a time there were three were three little sisters;and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>,
</p>
如果想获取所有的祖先节点，可以调用parents属性。
print(list(enumerate(soup.a.parents)))
（3）兄弟节点
next_sibling和previous_sibling分别获取节点的下一个和上一个兄弟元素
next_siblings和previous_siblngs则分别返回后面和前面的兄弟节点
print(soup.a.next_sibling)
print(soup.a.previous_sibling)
print(list(enumerate(soup.a.next_siblings)))
print(list(enumerate(soup.a.previous_siblings)))
方法选择器
find_all()
查询所有符合条件的元素，给它传入一些属性或者文本，就可以得到符合条件的元素
（1）name
print(soup.find_all(name="title"))
（2）attrs
print(soup.find_all(attrs={"id":"link1"}))
（3）text
text参数可用来匹配节点的文本，传入的形式可以是字符串，可以是正则表达式
from bs4 import BeautifulSoup
import re
soup=BeautifulSoup(html,'lxml')
print(soup.find_all(text=re.compile("La")))
输出：[‘Lacie’]
find()
返回的是单个元素，也就是第一个元素
print(soup.find(name="a"))
find_parents()和find_parent()：前者返回所有祖先节点，后者返回直接父节点
find_next_siblings()和find_next_sibling()：前者返回后面所有的兄弟节点，后者返回后面第一个兄弟节点
find_previous_siblings()和find_previous_sibling()：前者返回前面所有的兄弟节点，后者返回前面第一个兄弟节点
find_all_next()和find_next()：前者返回节点后所有符合条件的节点，后者返回第一个符合条件的节点
find_all_previous()和find_previous()：前者返回节点前所有符合条件的节点，后者返回第一个符合条件的节点
CSS选择器
使用CSS选择器时，只需要调用select()方法，传入相应的CSS选择器即可
html="""
<div class="panel">
<div class="panel-heading">
<h4>hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</l1>
<li class="element">Bar</l1>
<li class="element">Jay</l1>
</ul>
<ul class="list" id="list-2">
<li class="element">Foo</l1>
<li class="element">Bar</l1>
</ul>
</div>
</div>
"""
from bs4 import BeautifulSoup
soup=BeautifulSoup(html,'lxml')
print(soup.select(".panel-body #list-1 .element"))
输出：
[<li class="element">Foo
</li>, <li class="element">Bar
</li>, <li class="element">Jay
</li>]
嵌套选择
for ul in soup.select("ul"):
print(ul.select("li"))
获取属性
直接传入中括号和属性名，或者通过attrs属性获取属性值
for ul in soup.select("ul"):
print(ul["id"])
print(ul.attrs["id"])
获取文本
通过string或者get_text()获取文本
for li in soup.select("li"):
print(li.get_text()+" "+li.string)