【Python】爬虫-----数据解析之Xpath解析

前言：Xpath是在xml文档中搜索内容的一门语言，html是xml的子集。

一、Xpath解析xml

要解析的内容：

xml = """
    <bookstore>
    <book category="CHILDREN">
        <title>Harry Potter</title>
        <author>J K. Rowling</author>
        <year>2005</year>
        <price>29.99</price>
    </book>
    <book category="WEB">
        <title>Learning XML</title>
        <author>Erik T. Ray</author>
        <year>2003</year>
        <price>39.95</price>
    </book>
</bookstore>
"""

获取title里面的内容：

from lxml import etree
tree = etree.XML(xml)#加载xml内容
result = tree.xpath("/bookstore/book/title/text()")
print(result)
#['Harry Potter', 'Learning XML']

// 符号： 用于获取父类里的所有内容。

from lxml import etree
tree = etree.XML(xml)#加载xml内容
result = tree.xpath("/bookstore/book//text()")
print(result)
#['\n        ', 'Harry Potter', '\n        ', 'J K. Rowling', '\n        ', '2005', '\n        ', '29.99', '\n    ', '\n        ', 'Learning XML', '\n        ', 'Erik T. Ray', '\n        ', '2003', '\n        ', '39.95', '\n    ']

* 符号：通配符，截取所有节点里的内容。节点就是<title>....</title>这一类东西。

from lxml import etree
tree = etree.XML(xml)#加载xml内容
result = tree.xpath("/bookstore/book/*/text()")
print(result)
#['Harry Potter', 'J K. Rowling', '2005', '29.99', 'Learning XML', 'Erik T. Ray', '2003', '39.95']

二、Xpath解析html

要解析的网页内容(文件名为a.html)：

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8" />
    <title>Title</title>
</head>
<body>
    <ul>
        <li><a href="http://www.baidu.com">百度</a></li>
        <li><a href="http://www.google.com">谷歌</a></li>
        <li><a href="http://www.sogou.com">搜狗</a></li>
    </ul>
    <ol>
        <li><a href="python">Python</a> </li>
        <li><a href="c++">C++</a> </li>
        <li><a href="java">Java</a> </li>
    </ol>
    <div class="langue">语言</div>
    <div class="program">编程</div>
</body>
</html>

通过索引值来获取“百度”这个词：

from lxml import etree
tree = etree.parse("a.html")
#xpath的索引顺序是从1开始的。
result = tree.xpath("/html/body/ul/li[1]/a/text()")[0]
print(result)#百度

通过属性值来获取“百度”这个词：

from lxml import etree
tree = etree.parse("a.html")
result = tree.xpath("/html/body/ul/li/a[@href='http://www.baidu.com']/text()")[0]
print(result)#百度

遍历标签里的内容：

from lxml import etree
tree = etree.parse("a.html")
ul_li = tree.xpath("/html/body/ul/li")
for content in ul_li:
    result = content.xpath("./a/text()")[0]
    print(result)#百度 谷歌 搜狗

拿取标签里的属性值：

from lxml import etree
tree = etree.parse("a.html")
result = tree.xpath("/html/body/ul/li[1]/a/@href")[0]
print(result)#http://www.baidu.com

更快获取xpath路径：

打开网页页面，按f12，选中要选的标签，右键选择复制xpath。

【Python】爬虫-----数据解析之Xpath解析

一、Xpath解析xml

二、Xpath解析html

Guess you like