pgloader is a good tool for synchronizing data in multiple formats with pg, pgloader postrgresql using the copy data synchronization protocol for efficient processing
characteristic
Load the file into the content pg
- Support multiple data source format
csv, fixed column, dbase, sqlte, ibm ixf, postgresql, mysql, sql server - Instant data conversion
data can be adjusted before the data is loaded, clean up - Projection data
can be loaded required field - Load data from an archive
zip, tar, gzip import data - http (s) Support
Support http request to load data - Target pattern discovery
load data when, pgloader will consider existing columns, and guess csv format as much as possible - Stop error && error recovery to continue
in special cases, source data do not cause damage can not be fully migrated, pgloader will ignore the error and continue to migrate
to other lines
in some cases, you may need to complete the data pre&&postsql the Command
can easily before the data is loaded, and after some sql operations, such as creating a management index, table, privilege- Only architecture && only data
for migration may only need to migrate the schema, it is only possible to migrate data - Online alter architecture
for data type conversion when migrating - Materialized attempt, instant rewriting
can view when the migration will be converted into table - Distribute data to citus
- Coding covering
mysql actually does not enforce data in the data is encoded, but sometimes in order to understand the need for metadata covering
pgloader can help do something like - Continued migration
allows us to migrate the database is similar to the same continuous integration, migration can enhance our confidence
Reference material
https://pgloader.readthedocs.io/en/latest/
https://github.com/dimitri/pgloader