Python uses Pandas to import data directly into Mysql

Preface

The text and pictures in this article are from the Internet and are for learning and communication purposes only, and do not have any commercial use. If you have any questions, please contact us for processing.

PS: If you need Python learning materials, you can click on the link below to get it by yourself

Python free learning materials, codes and exchange answers click to join

---

The txt file data needs to be imported into the mysql database, and some data processing is required in the middle. After related searches, the to_sql() that comes with pandas can directly import the DataFrame into the database.

Although MySQL has other ways to import data, it needs to process the data before importing. These tasks cannot be completed, so you can use python to achieve all the requirements in one step.

Pandas has many advantages in processing tabular data: APIs are more convenient and fast; it can loop each row and process each value; it can also process entire columns, etc.

The following API is used when importing the database:
Pandas.DataFrame.to_sql()

Parameter introduction and precautions

Official document: [ https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html]

DataFrame.to_sql(name, con, schema=None, if_exists='fail', index=True, index_label=None, chunksize=None, dtype=None, method=None)
常用参数：

• name:
  the name of the table when imported into mysql.
  If the table has been created in mysql with CREATE TABLE, then it is the name of the table.
  If mysql has not created the table, then you can choose a suitable table name by yourself
• con:
  database connection, the sqlalchemy library needs to be installed, currently only supports the connection created by the sqlalchemy library, the connection created by the pymysql library does not support

engine = create_engine("mysql+pymysql://root:[email protected]:3306/routeapp?charset=utf8")
2#SQLALCHEMY_DATABASE_URI = '%s+%s://%s:%s@%s:%s/%s' % (DB_TYPE, DB_DRIVER, DB_USER,DB_PASS, DB_HOST, DB_PORT, DB_NAME)

• if_exists: The following three options mean if the table already exists in the database.
  "fail": report an error directly, no longer operate, similar to IF NOT EXISTS when creating a table in mysql, then create the table
  "replace": delete the table first, then Create
  "append" again: add data directly behind the table
• index: bool
  whether to write the index column of the DataFrame into the table
• index_label:
  If you want to write the index column of the DataFrame into the table, you need to give the name of the index column. If not, then the column index name of the DataFrame will be used.
  Note: The
  con parameter must be checked carefully, otherwise the database The connection will fail, you can refer to the example given above to change it according to your actual database location

Case study

First, the computer has been installed: mysql software, sqlalchemy library, pandas library

Now there is train number information between some cities, which needs to be imported into the database

import pandas as pd
data=pd.read_table('./data_pandas.txt')
data.head()

 

If the table has been created in the database, and the data type of each column has been specified, now you only need to import the data into it

CREATE TABLE IF NOT EXISTS train (
    start_city VARCHAR (100) NOT NULL COMMENT '始发城市',
    start_city_id int COMMENT '始发城市id',
    end_city VARCHAR (100) NOT NULL COMMENT '到达城市',
    end_city_id int COMMENT '到达城市id',
    train_code VARCHAR (20) NOT NULL COMMENT '车次',
    arrival_time VARCHAR (20) NOT NULL COMMENT '到达时间',
    departure_time VARCHAR (20) NOT NULL COMMENT '出发时间',
    run_time INT NOT NULL COMMENT '运行时间(分钟)',
    P1 FLOAT COMMENT '硬座票价',
    P2 FLOAT COMMENT '软座票价',
    P3 FLOAT COMMENT '硬卧票价',
    P4 FLOAT COMMENT '软卧票价',
    P5 FLOAT COMMENT '商务座票价',
    P6 FLOAT COMMENT '一等座',
    P7 FLOAT COMMENT '二等座'
) ENGINE = INNODB DEFAULT CHARSET = utf8 COMMENT = '城市之间火车信息';

Import data with the help of sqlalchemy library

from sqlalchemy import create_engine
engine = create_engine("mysql+pymysql://root:[email protected]:3306/routeapp?charset=utf8")

#SQLALCHEMY_DATABASE_URI = '%s+%s://%s:%s@%s:%s/%s' % (DB_TYPE, DB_DRIVER, DB_USER,DB_PASS, DB_HOST, DB_PORT, DB_NAME)

with engine.begin() as conn:    
 data.to_sql(name='routeapp_train_line_tb_new_2',con=conn,if_exists='append',index=False)

Here, the with statement can be used to implement MySQL's roleback function. It is recommended to use with to import data

 

Reference article

• python3 pandas to_sql fills the hole
  [ https://blog.csdn.net/qnloft/article/details/87979937]
• Commit and rollback with pandas.DataFrame.to_sql()
  [https://capelastegui.wordpress.com/2018/05/21/commit-and-rollback-with-pandas-dataframe-to_sql/]