In the actual project environment, sometimes it is necessary to migrate a large amount of data (here refers to the amount of data above the million level) from one server to another database server. There are many ways for SQL Server to perform data migration: backup and restore, import/export data, generate scripts (including data), etc. The following is only a test demonstration for the "import and export data" method (other methods can be found in official documents ).
There is an existing database of about 300W , which needs to be migrated from the local to the remote server (from the source database -> the target database, using the export data method).
1. Generate the exported data SSIS package.
First, in the SQL Server database object manager, expand the database -> select the corresponding database and right-click -> Task (T) -> Export Data (X)..., as shown below:
Open the "SQL Server Import and Export Wizard" page, click "Next" to select the data source (that is, the source from which the data will be copied), and set the server name, authentication, and database of the source database, as shown in the figure below:
"Next step" enters to set the target data source (that is, the target database to be copied to), similar to the previous step, set the server name, authentication, and database of the target data source, as shown in the figure below:
After setting up the source and target databases, the next step is to specify table replication and queries. If you want to migrate all table data in the entire database, select the first item "Copy the data of one or more tables or views"; only migrate part of the data (one or more data tables), select the second item "Write a query to specify the Transferred data", write SQL query statements to manipulate or limit the source data of the copy operation, as shown in the following figure:
The current demonstration migration data test is from the local to the remote server. Due to network problems, the remote server is directly connected to export the data. During the execution, the network may be disconnected.
The following uses the SQL batch export method to process, exporting 10W pieces of data each time.
Select the "Write a query to specify the data to be transferred" option, and in the next step, enter the SQL statement to query the data, as shown in the figure below:
Note: The SQL statement entered here only supports the query select statement.
In the next step, you can view the table or view of the source and target. In the previous step, SQL query was used to export data. Here, the source location shows only a [query] result set, and the target location can select the target database to receive data migration corresponding Data table, as shown below:
Note: The target data table structure of the received data migration must have a one-to-one correspondence with the [query] result set structure returned by the SQL query. If you do not select the target data table, a data table named "Query" will be automatically created in the target database to store the migration data.
In the next step, you can "Run Now" (selected by default) the SSIS package to perform the data export operation, or you can save the SSIS package configured to export the data first, and then execute it at a specific time. The current test is limited by network issues, which is handled by using the "save SSIS package" method. Cancel the "Run Now" option, the "Save SSIS Package" option will be automatically selected, and the "File System" method will be selected to save the SSIS package (generate *.dtsx file), which is convenient for migrating the file to a remote server for export, as shown below:
There are two ways to save SSIS package: [SQL Server] and [File System].
[SQL Server] method => stored in the SQL Server system, need to connect to the SQL Server Integration Server service to execute;
[File System] Mode => Save the SSIS package to the specified system directory.
In the next step, specify a name for the SSIS package and the directory where the file exists (the default is to save it to the system disk -> the document directory where the current user name is located), as shown in the figure:
In the next step, click the "Finish" button, and the system will generate the corresponding SSIS package according to the settings in the previous steps, as shown in the figure:
After completion, click the [Close] button to exit the [Export Data] wizard settings.
Because the data is exported in batches (10W pieces of data are exported each time), the above steps need to be repeated to generate SSIS packages for all data.
2. Execute the export data SSIS package
After completing the first part of the operation, copy the SSIS package to the server where the target database is located, double-click the SSIS package (*.dtsx) to automatically run the execution package utility, as shown in the figure below:
Click the "Execute" button, the system will export the data to the target database table according to the settings of the SSIS package, and automatically display the progress of the execution. When the progress shows the completion time, it means that the SSIS package has been successfully executed, as shown in the figure below:
Repeat the previous step to execute the remaining SSIS packages.
At this point, the data migration process has been completed.
3. Summary
- For data migration, you can use the import/export function that comes with SQL Server
- The exported data can be processed by specifying the SQL query method
- When a large amount of data is migrated and the network communication is limited, the batch export method can be used