史上最全的BeautifulSoup解析（基本选择器，标准选择器，css选择器）

其他 2018-05-28 23:49:11 阅读次数: 3

#写在前面


基本使用

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器
print(soup.prettify())#将代码的格式标准化
print(soup.title.string)#崎岖title标签内的文本

输出：

<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>

 
 
The Dormouse's story
 
 
 
 
 
 
标签选择器
 
 
 
 
示例代码如下：
 
 
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
 
 
1，选择元素
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器
print(soup.title)#将title标签里的内容包括其便签全部输出
print(type(soup.title))
print(soup.head)
print(soup.p)
 
 
输出结果如下：
 
 
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
 
 
The Dormouse's story
 
 

   2， 
  获取名称 
 
 
 
 
  
 
 
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.title.name)#获取标签的名称
 
 
输出如下：

 

   扫描二维码关注公众号，回复：
  1073746 查看本文章 
   
 
 
 
title
 
 
3，获取属性
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.attrs['name'])#获取标签的属性
print(soup.p['name'])#获取标签的属性
 
 
输出如下：
 
 
dromouse
dromouse
 
 
 
 
4，获取内容
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.string)#获取标签内的内容
 
 
输出如下：
 
 
The Dormouse's story

 
 
5，嵌套循环选择
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.head.title.string)#层层迭代，获取head标签里的title标签里的文本
 
 
输出如下：
 
 
The Dormouse's story
 
 
6，子节点和子孙节点
 
 
#方法一：获取p标签里的所有子节点，以list保存
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.contents)#获取p标签里的所有子节点，以list保存
 
 
输出如下：
 
 
['\n            Once upon a time there were three little sisters; and their names were\n            ', <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>, '\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' \n            and\n            ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, '\n            and they lived at the bottom of a well.\n        ']

 
 
# 方法二：获取p标签里的所有子节点，以迭代输出
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
    print(i, child)
 
 
输出如下：
 
 
<list_iterator object at 0x04BA7550>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4  
            and
            
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 
            and they lived at the bottom of a well.

 
 
 
 
# 方法三：获取p标签里的所有子节点以及子孙节点（子节点对应的子节点）
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.descendants)
for i, child in enumerate(soup.p.descendants):
    print(i, child)
 
 
输出如下：
 
 
<generator object descendants at 0x06030C00>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <span>Elsie</span>
4 Elsie
5 

6 

7 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
8 Lacie
9  
            and
            
10 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
11 Tillie
12 
            and they lived at the bottom of a well.
       
7， 
 父节点和祖先节点 
 
#方法一：获取标签的父节点（前一级的节点）
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.a.parent)
 
 
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
 
 
# #方法二：获取标签所有的祖先节点（父节点的父节点的父节点。。。直到最顶层（把整个文档输出））
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(list(enumerate(soup.a.parents)))
输出如下： 
 
 
 
[(0, <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>), (1, <body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body>), (2, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>), (3, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>)]

8， 
 兄弟节点 
 
 
 
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(list(enumerate(soup.a.next_siblings)))
print(list(enumerate(soup.a.previous_siblings)))
 
 
输出结果如下：
 
 
[(0, '\n'), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ' \n            and\n            '), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, '\n            and they lived at the bottom of a well.\n        ')]
[(0, '\n            Once upon a time there were three little sisters; and their names were\n            ')]

 
 
 
 
 
 
标准选择器
 
 
 
 
示例代码如下：
 
 
html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
 
 
1，find_all方法
 
 
 
 
 
  
 
   
 
    
find_all( name , attrs , recursive , text , **kwargs )
 
   
 
  
 
 
 
 
 
  
 
  
 
   
 
    
可根据标签名、属性、内容查找文档
 
   
 
  
 
 
 
 
name（标签名字选择）
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all('ul'))#查询所有便签为ul的元素
print(type(soup.find_all('ul')[0]))
 
 
输出如下：
 
 
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>
 
 
# 嵌套搜索
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.find_all('ul'):
    print(ul.find_all('li'))
 
 
输出如下：
 
 
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
 
 
attr（标签的属性选择）
 
 
#使用标签的的属性查找
#方法一：使用字典参数查询
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(attrs={'id': 'list-1'}))
print(soup.find_all(attrs={'name': 'elements'}))
 
 
输出如下：
 
 
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
 
 
#方法二：
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(id='list-1'))
#只选择element属性的内容
print(soup.find_all(class_='element'))
 
 
输出如下：
 
 
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
 
 
text（文本选择）
 
 
#只返回text文本，适合内容匹配，不适合文本查询
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text='Foo'))
 
 
输出如下：
 
 
['Foo', 'Foo']
 
 
 
 
 
 
2，find方法
 
 
 
 
 
  
 
   
 
    
find( name , attrs , recursive , text , **kwargs )
 
   
 
  
 
 
 
 
 
  
 
  
 
   
 
    
find返回单个元素，find_all返回所有元素
 
   
 
  
 
 
 
 
 
 
html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find('ul'))
print(type(soup.find('ul')))
print(soup.find('page'))
 
 
输出如下：
 
 
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<class 'bs4.element.Tag'>
None
 
 

 
 
3，其他的一些find方法
 
 
 
  
 
   
 
    
find_parents() find_parent()
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_parents()返回所有祖先节点，find_parent()返回直接父节点。
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_next_siblings() find_next_sibling()
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_next_siblings()返回后面所有兄弟节点，find_next_sibling()返回后面第一个兄弟节点。
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_previous_siblings() find_previous_sibling()
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_previous_siblings()返回前面所有兄弟节点，find_previous_sibling()返回前面第一个兄弟节点。
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_all_next() find_next()
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_all_next()返回节点后所有符合条件的节点, find_next()返回第一个符合条件的节点
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_all_previous() 和 find_previous()
 
   
 
  
 
 
 
 
 
  
 
   
 
    
find_all_previous()返回节点后所有符合条件的节点, find_previous()返回第一个符合条件的节点
 
   
 
  
 
 
 
 

 
 
 
 
 
  
 
   
 
    
CSS选择器
 
   
 
  
 
 
 
 
 
  
 
  
 
   
 
    
通过select()直接传入CSS选择器即可完成选择
 
   
 
  
 
 
 
 
实例代码如下：
 
 
html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
 
 1，基本语法 
 
 
 
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.select('.panel .panel-heading'))#选择class的类型
print(soup.select('ul li'))#直接选择标签
print(soup.select('#list-2 .element'))#选择id的类型
print(type(soup.select('ul')[0]))
 
 
 
 
输出如下：
 
 
[<div class="panel-heading">
<h4>Hello</h4>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<class 'bs4.element.Tag'>
 
 2，层层迭代 
 
 
 
#把每一组ul的li输出
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
    print(ul.select('li'))
 
 
输出如下：
 
 
#把每一组ul的li输出
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
    print(ul.select('li'))
 
 3，获取属性 
 
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
    print(ul['id'])#这两种方法都能获取标签的属性（id或其他）
    print(ul.attrs['id'])
 
 
输出如下：
 
 
list-1
list-1
list-2
list-2
 
 
4，获取内容
 
 
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for li in soup.select('li'):
    print(li.get_text())#输出li里的内容
 
 
输出如下：
 
 
Foo
Bar
Jay
Foo
Bar
 
 
 
 
 
  
 
   
 
    
总结
 
   
 
  
 
 
 
 
 
  
 
  
 
   
 
    
     
     
  • 推荐使用lxml解析库，必要时使用html.parser
    •  
     
  • 标签选择筛选功能弱但是速度快
    •  
     
  • 建议使用find()、find_all() 查询匹配单个结果或者多个结果
    •  
     
  • 如果对CSS选择器熟悉建议使用select()
    •  
     
  • 记住常用的获取属性和文本值的方法
    •  
    
 
   
 
  
 
 
 
 
至此，三种选择器的基本语法都讲解完毕，希望大家有所收获。
 
 

 
 

 


                    
                    
                    
                    
                    
                    
                    
                    
                    

                        
猜你喜欢

                        

                            
                            
                        

                    


                    
转载自blog.csdn.net/qq_41686130/article/details/79856474

                    

                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                        
                    

                    


                

            

            


                

                    

                        
                        
                        
                        
                    


                    

                        

                            

                                今日推荐
                            


                            

                        

                    

                    

                        

                            

                                基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
                            

                        

                        

                            

                                美国拟限制 AI 大模型出口中国和俄罗斯
                            

                        

                        

                            

                                苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
                            

                        

                        

                            

                                openKylin 社区生态委员会第六次会议圆满召开
                            

                        

                        

                            

                                阿里云正式发布通义千问 2.5
                            

                        

                        

                            

                                Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
                            

                        

                        

                            

                                Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
                            

                        

                        

                            

                                Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
                            

                        

                        

                            

                                《2024 年一季度互联网投融资运行情况》研究报告
                            

                        

                        

                            

                                报告：Django 仍然是 74% 开发者的首选
                            

                        

                        

                            

                                15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
                            

                        

                        

                            

                                TIOBE 5 月榜单：Fortran “复活”进入 Top 10
                            

                        

                        
                        
                    

                


                

                    

                        

                            

                                周排行
                            

                            

                        

                    

                    

                        

                            

                                记一下去大梅沙的准备（2018-05-26）
                            

                        

                        

                            

                                Spring 注解 事务
                            

                        

                        

                            

                                基于HTTP协议的客户端缓存
                            

                        

                        

                            

                                阿里云rds 备份和还原
                            

                        

                        

                            

                                [PHP] 几个拖慢 PHP 程序/API 运行速度的点
                            

                        

                        

                            

                                python 代码风格------------PEP8规则
                            

                        

                        

                            

                                js控制json生成菜单——自制菜单（一）
                            

                        

                        

                            

                                将字符串: 'k:1|k1:2|k2:3|k3:4 ' ,处理成 python 字典: {'k':1, 'k1':2, ...}
                            

                        

                        

                            

                                微信小程序转支付宝小程序
                            

                        

                        

                            

                                Qt551.窗口滚动条
                            

                        

                    

                

                

                    

                        

                            

                                每日归档
                            

                            

                                更多
                            

                            

                        

                    

                    

                        

                            2024-05-13(18)
                        

                        

                            2024-05-12(0)
                        

                        

                            2024-05-11(38)
                        

                        

                            2024-05-10(38)
                        

                        

                            2024-05-09(35)
                        

                        

                            2024-05-08(42)
                        

                        

                            2024-05-07(14)
                        

                        

                            2024-05-06(40)
                        

                        

                            2024-05-05(0)
                        

                        

                            2024-05-04(7)
                        

                    

                


            


            
        

    






















    


        

            
代码天地 © 2018 codetd.com

            境外服务器
             
            最新电视剧2022