Python3 crawler (9) relational database MySQL for data storage

 Infi-chu:

http://www.cnblogs.com/Infi-chu/

Relational database
The relational database is a database based on the relational model, and the relational model is stored in a two-dimensional table, so the storage method of the relational database is a table composed of rows and columns.
Each column is a field and each row is a record.
A table can be regarded as a collection of entities, and there are relationships between entities, such as primary keys and foreign keys.
Common relational databases include: Oracle, MySQL, SQLite, SQL Server, DB2, etc.

MySQL
is in Python2, the link library of MySQL is MySQLdb,

In Python3, MySQLdb is not officially supported, and PyMySQL is used in Python3

1. Connect to the database

import pymysql
db = pymysql.connect(host='127.0.0.1',user='root',password='123456',port=3306)
cursor=db.cursor()
cursor.execute('SELETE VERSION()') # Use execute method to execute SQL statement
data = cursor.fetchone() # Use the fetchone() method to get the first piece of data
print('Version is :',data)
cursor.execute('CREATE DATABASE spiders DEFAULT CHARCTER SET uft-8') # Create spiders database, the default encoding is UTF-8
db.close()

2. Create the table

import pymysql
db = pymysql.connect(host='127.0.0.1',user='root',password='123456',port=3306,db='spiders')
cursor = db.cursor()
sql = 'CREATE TABLE IF NOT EXISTS tests (id VARCHAR(255) NOT NULL, name VARCHAR(255) NOT NULL, age INT NOT NULL, PRIMARY KEY (id))'
cursor.execute(sql)
db.close()

[Note]
In the actual crawler project, the database should be designed according to the actual situation

3. Insert data

import pymysql
id = '123'
user = 'Infichu'
age = 23
db = pymysql.connect(host='127.0.0.1',user='root',password='123456',port=3306,db='spiders')
cursor = db.cursor()
sql = 'INSERT INTO tests(id,name,age) values(%s,%s,%s)'
try:
    cursor.execute(sql,(id,user,age))
	db.commit() # commit() method, database commit
except:
    db.rollback() # rollback(), database rollback
db.close()

4 attributes of things
Attributes describe
atomicity A thing is an inseparable unit of work, and many operations included in a thing either do or do not do
consistency (consistency) A thing must make the database change from a consistent state to another A consistent state. Consistency and atomicity are closely related to
isolation (isolation) The execution of a thing cannot be interfered with by other things
(durability) Persistence, once a thing is committed, its changes to the data in the database should be permanent. Subsequent operations or malfunctions will not affect it

# Generic insert method
import pymysql
data = {
'id':'1',
'name':'Infi-chu',
'age':23
}
table = 'tests'
keys = ', '.join(data.keys())
values = ', '.join(['%s'*len(data)])
sql = 'INSERT INTO {table}({keys}) VALUES ({values})'.format(table=table,keys=keys,values=values)
try:
    if cursor.execute(sql,tuple(data.values())):
	print('Successful')
except:
    print('Failed')
	db.rollback()
db.close()

4. Update data

sql = 'UPDATE tests SET age=%s WHERE name=%s'
try:
    cursor.execute(sql,(25,'Infi-chu'))
	db.commit()
except:
    db.rollback()
db.close()

# 通用更新方法
import pymysql
data = {
'id':'1',
'name':'Infi-chu',
'age':23
}
table = 'tests'
keys = ', '.join(data.keys())
values = ', '.join(['%s']*len(data))
# ON DUPLICATE KEY UPDATE 表示如果主键已经存在，就执行更新操作
sql = 'INSERT INTO {table}({keys}) VALUES ({values}) ON DUPLICATE KEY UPDATE'.format(table=table,keys=keys,values=values)
update = ','.join([" {keys}=%s".format(key=key) for key in data])
sql += update
try:
if cursor.execute(sql,tuple(data.values())*2):
print('Successful')
db.commit()
except:
print('Failed')
db.rollback()
db.close()

5. Delete data:

table = 'tests'
condition = 'age>20'
sql = 'DELETE FROM {table} WHERE {condition}'.format(table=table,condition=condition)
try:
    cursor.execute(sql)
	db.commit()
except:
	cursor.rollback()
db.close()

6. Query data:

sql = 'SELECT * FROM tests WHERE age>=20'
try:
    cursor.execute(sql)
	print('Count:',cursor.rowcount)
	one = cursor.fetchone()
	print('One:',one)
	results = cursor.fetchall()
	print('Results:',results)
	print('Results Type:',type(results))
	for row in results:
	    print(row)
except:
    print('Error')