Notes python network data collection (III)

Chapter VI to read the document

First, the plain text

Processing HTML when the page, the site will in fact <head> encoding format used part of the display page. Most network station, in particular the English site, will bring this label:

Two, CSV

Directly to read the file into a string, and then packaged into StringIO objects, make Python treat it as a file to process

1 from urllib.request import urlopen
2 from io import StringIO
3 import csv
4 
5 data = urlopen("http://pythonscraping.com/files/MontyPythonAlbums.csv").read().decode('ascii', 'ignore')
6 dataFile = StringIO(data)
7 csvReader = csv.reader(dataFile)
8 for row in csvReader:
9     print(row)

Output:

['Name', 'Year']
["Monty Python's Flying Circus", '1970']
['Another Monty Python Record', '1971']
["Monty Python's Previous Record", '1972']
['The Monty Python Matching Tie and Handkerchief', '1973']
['Monty Python Live at Drury Lane', '1974']
['An Album of the Soundtrack of the Trailer of the Film of Monty Python and the Holy Grail', '1975']
['Monty Python Live at City Center', '1977']
['The Monty Python Instant Record Collection', '1977']
["Monty Python's Life of Brian", '1979']
["Monty Python's Cotractual Obligation Album", '1980']
["Monty Python's The Meaning of Life", '1983']
['The Final Rip Off', '1987']
['Monty Python Sings', '1989']
['The Ultimate Monty Python Rip Off', '1994']
['Monty Python Sings Again', '2014']

One is to make use csv.dictReader

 1 from urllib.request import urlopen
 2 from io import StringIO
 3 import csv
 4 
 5 
 6 data = urlopen("http://pythonscraping.com/files/MontyPythonAlbums.csv").read().decode('ascii', 'ignore')
 7 dataFile = StringIO(data)
 8 dictReader = csv.DictReader(dataFile)
 9 
10 print(dictReader.fieldnames)
11 for row in dictReader:
12     print(row)

Output:

['Name', 'Year']
OrderedDict([('Name', "Monty Python's Flying Circus"), ('Year', '1970')])
OrderedDict([('Name', 'Another Monty Python Record'), ('Year', '1971')])
OrderedDict([('Name', "Monty Python's Previous Record"), ('Year', '1972')])
OrderedDict([('Name', 'The Monty Python Matching Tie and Handkerchief'), ('Year', '1973')])
OrderedDict([('Name', 'Monty Python Live at Drury Lane'), ('Year', '1974')])
OrderedDict([('Name', 'An Album of the Soundtrack of the Trailer of the Film of Monty Python and the Holy Grail'), ('Year', '1975')])
OrderedDict([('Name', 'Monty Python Live at City Center'), ('Year', '1977')])
OrderedDict([('Name', 'The Monty Python Instant Record Collection'), ('Year', '1977')])
OrderedDict([('Name', "Monty Python's Life of Brian"), ('Year', '1979')])
OrderedDict([('Name', "Monty Python's Cotractual Obligation Album"), ('Year', '1980')])
OrderedDict([('Name', "Monty Python's The Meaning of Life"), ('Year', '1983')])
OrderedDict([('Name', 'The Final Rip Off'), ('Year', '1987')])
OrderedDict([('Name', 'Monty Python Sings'), ('Year', '1989')])
OrderedDict([('Name', 'The Ultimate Monty Python Rip Off'), ('Year', '1994')])
OrderedDict([('Name', 'Monty Python Sings Again'), ('Year', '2014')])

Here the output of different books, OrderedDict is an ordered objects .

Three, PDF, word, .docx, MySQL

Skip, there is a need to look at, in particular, need to focus on MySQL to look at.