pandas data analysis - data reading sqlite3 database

In many applications, data is rarely taken from a text file, stored in this way because a large number of data files is very inefficient, very extensive relational databases using SQL-based, in addition to some non-SQL (such as NOSQL) database becomes very popular, select the database often depends on the needs of performance scalability, data integrity and application.

Load data from SQL to DataFrame very simple, in addition pandas there are some functions to simplify the process, for example, I'll use an embedded SQLite database (via sqlite3 python built-in drive)

import sqlite3
from pandas import Series,DataFrame
import pandas.io.sql as sql
query = """
CREATE TABLE test
(a VARCHAR(20),b VARCHAR(20),
c REAL,d INTEGER
);"""

con = sqlite3.connect(':memory:')
con.execute(query)
con.commit()

Then insert rows

When data is selected from the table, it will return a list of tuples

date = [('atl','at',1.25,3),
        ('btl','bt',1.35,4),
        ('ctl','ct',1.45,5)]
stmt = "INSERT INTO test VALUES(?,?,?,?)"
con.executemany(stmt,date)
con.commit()
cur = con.execute('select * from test')
rows = cur.fetchall()
print(rows)

You can use this list of tuples passed to the constructor DataFrame

print(DataFrame(rows))

This operation is very multi-structured data, you certainly do not want to check every time the database is overwritten once, pandas can have a simplified read_sql function of the process, only you need to pass the select statement and connection object can be.

print(sql.read_sql('select * from test',con))