Import data from postgresql to mysql

Export data from postgres to csv

1 Login to postgresql with postgres user (root)

2 Create a csv file and chmod 777


3 Modify the ---- title in the original data to: (the title with; will be split into two columns when exported to csv)
update article_article set title = ' China's Copycats: Online vs. Offline.' where id=7939;
update article_article set title = 'Web browsers used most in China: how this impacts online campaigns.' where id=7804;
update article_article set title = 'Web browsers used most in China: how this impacts online campaigns.' where id=7926;
update article_article set title = 'Douban.com: China's Amazon/Digg Hybrid Social Media Network.' where id=7973;
update article_article set title ='“China average daily salary is RMB 111.99???”: Chinese netizens challenge government statistics. ' where id=8029;
update article_article set title =  'China Telecom: Becoming the telephone version of Google.' where id=8072;
update article_article set title =  'The Rich Tighten Pockets: Online Shopping is New Obsession' where id=8169;
update article_article set title =  'The Good, The Bad And The Foreigner: Learning To Accept Myself Through The Eyes Of Others.' where id=7541;
update article_article set title =  'Johnson & Johnson: Big girls don’t cry… or let their bones break.' where id=8066;
select * from article_article where title like '%;%';

4----超级用户postgres，  登录
运行sql
COPY (SELECT * FROM article_article limit 50) TO '/home/david/myfile.csv' WITH CSV HEADER;
5--把导出的csv文件的第一行（culomn名删了）
csv导入mysql
1----mysql reads files from the directory /var/lib/mysql/blog (blog is the database name), so put csv in this directory
and run the sql command in mysql:
load data infile 'myfile. csv'
into table blog.article
fields enclosed by ',' optionally by '"' escaped by '"'
lines terminated by '\n';

***The question
in postgres boolean is represented by true and false (in csv it is t and f)
But 1 and 0 (tinyint) are used in mysql. If you import directly, 0 will be displayed in mysql,
and t and f will be changed to 1 and 0.
2----After importing, use py script to update the boolean value

#encoding:utf-8
import psycopg2
import MySQLdb

psycopg2_conn = psycopg2.connect(
        #database="expat",
        database="postgres",
        user="david",
        password="1",
        host="localhost",
        port="5432"
    )

mysql_conn= MySQLdb.connect(
        host='localhost',
        port = 3306,
        user='root',
        passwd='1',
        db ='blog',
    )


def import_mysql_boolean():
    '''
    boolean in postgres is represented by true and false (t and f in csv)
    However, 1 and 0 (tinyint) are used in mysql, and 0 is displayed in mysql if imported directly.
    Directly changing csv to change t and f to 1 and 0 causes some data columns imported into mysql to not match

    First import the original data into mysql, and then update the boolean value through this function
    '''
    cur = psycopg2_conn.cursor()
    cur.execute('''
        select id,is_approved, is_special, is_welcome, is_shf_featured,
            is_shf_sponsored, is_homepage_sponsored, is_home_featured
        from article_article
        --where id in (8029 ,10691,  10099, 8810, 8936, 8939 , 10126 ,10128 ,10222 ,9217 ,9178,7898,7899)
        --limit 1000
        ''')
    rows = cur.fetchall()
    #print rows

    cur.close()
    psycopg2_conn.commit()
    psycopg2_conn.close()

    cur2 = mysql_conn.cursor()

    for row in rows:
        arow = list()
        arow.append(row[0])
        arow.append(1) if row[1] is True else arow.append(0) # is_approved
        arow.append(1) if row[2] is True else arow.append(0) # is_special
        arow.append(1) if row[3] is True else arow.append(0) # is_welcome
        arow.append(1) if row[4] is True else arow.append(0) # is_shf_featured
        arow.append(1) if row[5] is True else arow.append(0) # is_shf_sponsored
        arow.append(1) if row[6] is True else arow.append(0) # is_homepage_sponsored
        arow.append(1) if row[7] is True else arow.append(0) # is_home_featured
        sql = '''
        update article set
            is_approved={is_approved},
            is_special={is_special},
            is_welcome={is_welcome},
            is_shf_featured={is_shf_featured},
            is_shf_sponsored={is_shf_sponsored},
            is_homepage_sponsored={is_homepage_sponsored},
            is_home_featured={is_home_featured}
        where id = {id}
        '''.format(id=arow[0],
                   is_approved=arow[1],
                   is_special=arow[2],
                   is_welcome=arow[3],
                   is_shf_featured=arow[4],
                   is_shf_sponsored=arow[5],
                   is_homepage_sponsored=arow[6],
                   is_home_featured=arow[7],

            )

        print sql
        cur2.execute(sql)
    cur2.close()
    mysql_conn.commit()
    mysql_conn.close()

 

There is a situation that a field of the table is larger than the cell limit of the csv file. At this time, if the csv file is edited and saved, in fact, the excessively large cell will be split into two cells, and then imported into mysql. It is the two fields in the table, which will cause the table fields to be misaligned. If you encounter this problem, don't edit the csv file. First, import the csv exported from postgresql directly into mysql, and then modify it with sql in mysql without error.

 

 

There is also a situation that requires escaping, such as inserting 'App\Article' into a table, which is written as 'App\\\Article' in python, it will be escaped once by python, and will be converted again when inserting into mysql, In the end, it will be represented as 'App\Article' in the mysql table