Python crawler example 1

Web page

http://www.pythonscraping.com/pages/warandpeace.html

This is war and peace, and the green characters in it are names of people.

Web page source code

The shortcut key Ctrl+U can display the source code of the webpage

Observe the source code and find that it is all <span class="red">or<span class="green">

The green font that represents the name of the person is<span class="green">

Python crawling code

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.pythonscraping.com/pages/warandpeace.html")
bsObj = BeautifulSoup(html, "lxml")
nameList = bsObj.findAll("span", {
   
   "class":"green"})

for name in nameList:
    print(name.get_text())

Output:

Anna
Pavlovna Scherer
Empress Marya
Fedorovna
Prince Vasili Kuragin
Anna Pavlovna
St. Petersburg
the prince
Anna Pavlovna
Anna Pavlovna
the prince
the prince
the prince
Prince Vasili
Anna Pavlovna
Anna Pavlovna
the prince
Wintzingerode
King of Prussia
le Vicomte de Mortemart
Montmorencys
Rohans
Abbe Morio
the Emperor
the prince
Prince Vasili
Dowager Empress Marya Fedorovna
the baron
Anna Pavlovna
the Empress
the Empress
Anna Pavlovna's
Her Majesty
Baron
Funke
The prince
Anna
Pavlovna
the Empress
The prince
Anatole
the prince
The prince
Anna
Pavlovna
Anna Pavlovna

The code comes from the book "Python Network Data Acquisition"