Python reads the excel file and inserts it into the database

 1. Demand background

        Recently, a problem was encountered during the project practice: when using Navicat to import data into the PostgreSQL database, it was found that the time value in the field of the time format had changed, resulting in incorrect time for some data, so the data was manually imported into the database and an error was reported. In order to solve this problem, I decided to write a Python code to read the Excel file and insert the data into the target database row by row to ensure the accuracy of the time data.

2. Practice process

1. Make sure you have installed the required Python libraries (like pandasand psycopg2) for working with Excel files and connecting to PostgreSQL databases.

2. Prepare the Excel file to be imported, make sure that the time field is in the correct format and contains the data to be imported.

3. Write a Python script to perform data import operations.

3. Code practice

import psycopg2
import pandas as pd

# 连接数据库
conn = psycopg2.connect(database='jiangyu_01', user='bigdata', password='postgres', host='192.168.22.168', port='1234')
cur = conn.cursor()

# 读取Excel文件并更新数据库中的数据
def synonym():
    data_xls = pd.read_excel('new.xlsx', usecols=[0, 5], Sheetname='news')  # 读取Excel文件中的两列数据，Sheet名为'news'
    print(len(data_xls))
    for data in data_xls.values:
        sql = """UPDATE public.jiaozheng_dataxx SET upload_time='{}' WHERE id={}""".format(data[1], data[0])
        cur.execute(sql)
        conn.commit()

def get_dataset():
    # 读取数据库中的数据示例
    conn = psycopg2.connect(database='sanyu_01', user='tatt', password='postgres', host='192.168.91.13', port='5432')
    cur = conn.cursor()
    cur.execute("SELECT * FROM public.jiaozheng_dingweixx LIMIT 1")
    data = cur.fetchall()
    print(data)

if __name__ == '__main__':
    synonym()  # 执行更新操作

        The above code is an example to connect to the PostgreSQL database through the psycopg2 library, read the data in the Excel file, and update it row by row to the fields jiaozheng_dataxxin the specified table of the database upload_time.

        The function in the code synonym()is used to read 'new.xlsx'the two columns of data in the 'news' Sheet of the Excel file, and use the SQL update statement to update the data row by row into the database. get_dataset()Functions are used to demonstrate how to read data from the database.

        You can modify the database connection parameters, Excel file path, table name and field name and other information according to your own needs to meet your actual database and data update needs. At the same time, you can also add functions such as exception handling and logging according to your needs to improve the code.

        In this way, the method of reading the file and inserting it into the database is realized, temporarily solving the problem that the current manual import of data using the Navicat tool fails.