Teach you to use Python Fun MySQL database, big data import no longer a problem!

Data analysis is inseparable from the database, how to use python to connect MySQL database, and perform CRUD operations it?

We also encounter the need to import large quantities of data in the database, how to use Python to import large data efficiently it?

This article will explain one by one, and with code and examples.

I. Background

I was connected experiments in Anaconda notebook, the environment Python3.6, of course, can be operated in Python Shell inside.

The most common and most stable connection for MySQL database python library is PyMySQL.

Therefore, this article discusses the use of connecting PyMySQL MySQL database, CRUD operations, and storing large quantities of data.

Reference PyMySQL official documentation and "python Data Collection" section on stored data.

Welcome everyone to read the original document, I believe will be more thorough understanding.

Second, the basic operation

1, the installation PyMySQL library

The easiest way:
at the command linepip install pymysql

Or:
Download whl file installation, the installation process itself Baidu.

2, install the MySQL database

There are two categories MySQL database: MySQL and MariaDB, I use the latter MariaDB.

Both on most properties are compatible, what use do not feel the difference.

Gives Download: MySQL , MariaDB , the installation process is very simple, all the way to Next Step, but to remember your password.

There is a small episode, MySQL and MariaDB is equivalent to the relationship between the sisters, both created by the same person (Widenius) of. After the acquisition of MySQL by Oracle, Mr. Widenius feel uncomfortable, then staged a MariaDB, can completely replace MySQL. Daniel is wayward.

3, SQL basic syntax

Use the following SQL to create tables, queries, data insertion and other functions, here a brief overview of the basic statements of the SQL language.

• View database:SHOW DATABASES；

• Create a database:CREATE DATEBASE 数据库名称；

• Use database:USE 数据库名称；

• View the data sheet:SHOW TABLES；

• Create a data table:CREATE TABLE 表名称（列名1 （数据类型1），列名2 （数据类型2））；

• Insert data:INSERT INTO 表名称（列名1，列名2） VALUES（数据1，数据2）；

• View data:SELECT * FROM 表名称；

• update data:UPDATE 表名称 SET 列名1=新数据1，列名2=新数据2 WHERE 某列=某数据；

4, connect to the database

After installing the necessary files and libraries too, then started connecting to the database, even though the mystery is not difficult Oh!

#首先导入PyMySQL库
import pymysql
#连接数据库，创建连接对象connection
#连接对象作用是：连接数据库、发送数据库信息、处理回滚操作（查询中断时，数据库回到最初状态）、创建新的光标对象
connection = pymysql.connect(host = 'localhost' #host属性
                             user = 'root' #用户名 
                             password = '******'  #此处填登录数据库的密码
                             db = 'mysql' #数据库名
                             )

Implementation of this code are connected!

5, CRUD operations

First, let's look at what the database:

#创建光标对象，一个连接可以有很多光标，一个光标跟踪一种数据状态。
#光标对象作用是：、创建、删除、写入、查询等等
cur = connection.cursor()
#查看有哪些数据库，通过cur.fetchall()获取查询所有结果
print(cur.fetchall())

Print out all databases:

(('information_schema',),
('law',),
('mysql',),
('performance_schema',),
('test',))

Create a table in the test database:

#使用数据库test
cur.execute('USE test')
#在test数据库里创建表student，有name列和age列
cur.execute('CREATE TABLE student(name VARCHAR(20),age TINYINT(3))')

insert a student data to the data table:

sql = 'INSERT INTO student (name,age) VALUES (%s,%s)'
cur.execute(sql,('XiaoMing',23))

View Data Sheet student content:

cur.execute('SELECT * FROM student')
print(cur.fetchone())

Printed as :( 'XiaoMing', 23)

Bingo! We just inserted a data

Finally, remember to turn off the cursor and connection:

#关闭连接对象，否则会导致连接泄漏，消耗数据库资源
connection.close()
#关闭光标
cur.close()

OK, the whole process much the same.

Of course, there are very basic operation, using the method requires more PyMySQL official document looking to go.

Third, the import of large data files

To csv file, for example, csv file into the database there are two ways:

1, an SQL introduced through an insert method, for small amount of data CSV file is not elaborated here.

2, by a method of introducing load data, fast, suitable for large data files is the focus of this paper.

Sample CSV file as follows:

The overall work is divided into three steps:

1, is connected with the python mysql database;

2, create a table based on a CSV file Form Field;

3, a method using the load data content importing a CSV file.

The introduction sql load data syntax:

LOAD DATA LOCAL INFILE 'csv_file_path' INTO TABLE table_name FIELDS TERMINATED BY ',' LINES TERMINATED BY '\\r\\n' IGNORE 1 LINES

csv_file_pathIt refers to a file absolute path

table_name refers to the name of the table

FIELDS TERMINATED BY ',' refers to the comma-delimited

LINES TERMINATED BY '\\r\\n' finger wrap

IGNORE 1 LINES skipping the first row, the first row of field names as table

All codes are given below:

#导入pymysql方法
import pymysql


#连接数据库
config = {'host':'',
          'port':3306,
          'user':'username',
          'passwd':'password',
          'charset':'utf8mb4',
          'local_infile':1
          }
conn = pymysql.connect(**config)
cur = conn.cursor()


#load_csv函数，参数分别为csv文件路径，表名称，数据库名称
def load_csv(csv_file_path,table_name,database='evdata'):
    #打开csv文件
    file = open(csv_file_path, 'r',encoding='utf-8')
    #读取csv文件第一行字段名，创建表
    reader = file.readline()
    b = reader.split(',')
    colum = ''
    for a in b:
        colum = colum + a + ' varchar(255),'
    colum = colum[:-1]
    #编写sql，create_sql负责创建表，data_sql负责导入数据
    create_sql = 'create table if not exists ' + table_name + ' ' + '(' + colum + ')' + ' DEFAULT CHARSET=utf8'
    data_sql = "LOAD DATA LOCAL INFILE '%s' INTO TABLE %s FIELDS TERMINATED BY ',' LINES TERMINATED BY '\\r\\n' IGNORE 1 LINES" % (csv_filename,table_name)
 
    #使用数据库
    cur.execute('use %s' % database)
    #设置编码格式
    cur.execute('SET NAMES utf8;')
    cur.execute('SET character_set_connection=utf8;')
    #执行create_sql，创建表
    cur.execute(create_sql)
    #执行data_sql，导入数据
    cur.execute(data_sql)
    conn.commit()
    #关闭连接
    conn.close()
    cur.close()