Unlike synchronization (a one-off process that brings the schema and data of two databases into sync), replication is a continuous (automatic) process of reproducing data between two databases (although schema updates are also possible). Replication can be done asynchronously, so there is no need for a permanent connection between the two databases, or it can be done during off-peak hours when there is little database server traffic, such as late at night.
The primary role of replication is to create a consolidated repository of all user databases and/or to propagate the same level of information across all users. In either case, the results database is a distributed database where users can access data relevant to their tasks without interfering with the work of others. Database replication is known as normalization, in order to remove data ambiguity or inconsistency between users.
In the Database Synchronization Strategies white paper, we explored strategies for synchronizing two databases of the same and different types using the Navicat Premium database management system. In today's article, we'll cover how to automate row database replication using Navicat Premium's new "Autorun" tool.
copy type
There are at least three different approaches to database replication:
- Snapshot replication: Data on one server is replicated to another database on the same or a different server.
- Merge Replication: Data from two or more databases is merged into a single database.
- Transactional replication: Users receive a complete initial copy of the database and then receive periodic updates as data changes.
In a distributed database management system (DDBMS), changes, additions, and deletions performed to data in one location are automatically reflected in data stored in all other locations. Doing this ensures that each user accesses the same datasets as all other users.
Like synchronization, replication can be homogeneous or heterogeneous:
- Isomorphic: Same source and target database, ie Percona to Percona, MariaDB to MariaDB, MySQL to MySQL.
- Heterogeneous: Different source and target databases, i.e. Oracle to Microsoft SQL Server, PostgresSQL to Amazon DynamoDB, MySQL to Amazon Aurora.
Heterogeneous replication is required when a situation arises where one or more external business partners use a different database type than our own. Automatic periodic data replication between the two environments is often an integral part of this arrangement.
Autorun Tool for Navicat Premium
Introduced in version 12, Navicat Premium's new Autorun tool has an easy-to-use and intuitive interface for creating automated batch jobs. Autorun is a process that runs at one or more regular intervals and can start and end at a specific date and time, just like the Windows Task Scheduler. It can be used for a variety of jobs besides replication, including backup, query, and reporting.
Figure 1: Navicat Premium 12 Autorun Tool in Windows
Figure 2: Navicat Premium 12 autorun tool in macOS
user database
We will use the Sakila sample MySQL database as our user database. It was developed by Mike Hillyer, a former member of the MySQL AB documentation team, specifically to provide standard schemas for use in books, tutorials, articles, and more.
Focused on the film industry, this database covers everything from actors and film studios to video rental stores. If you're interested, you can view the full schema on the MySQL Dev site .
For instructions on setting up a Sakila database with Navicat, see the Generating Reports on MySQL Data article on databasejournal.com .
Sakila MySQL database structure in Navicat Premium 12
snapshot copy
As mentioned above, snapshot replication synchronizes two databases by copying data from one database to another on the same or different server. This is the simplest of the three types.
Create a data synchronization job
A data synchronization profile must first be created in order to run it automatically as a replication process. The steps to implement homogeneous and heterogeneous synchronization in Navicat Premium 12 are described in Database Synchronization Strategies white paper. For the purposes of this tutorial, we will use homogeneous synchronization between the sakila and sakila2 databases as the first example.
Tip: After creating the sakila database, you can create the sakila2 database by right-clicking on the connection in the navigation pane and selecting New Database. Then enter the database name ("sakila2") in the popup.
Open the data synchronization wizard:
- Select " Tools " -> " Data Synchronization " from the menu bar .
- The Options tab for Data Sync contains only checkboxes for a few comparison options. We can leave them alone: /li>
- The next step in the Data Synchronization Wizard is to map the tables. The target table can be selected via the drop-down list. In this example, we don't need to provide any mapping directives, since the tables are the same in both databases:
-
After comparing the data, the window displays the number of records that will be inserted, updated or deleted in the target table. If you don't want to include tables with the same data and tables with a different structure (i.e. in other words, tables that won't update), you can untick "Show same tables and others". There are also checkboxes to deselect tables or actions that you don't want applied to the target.
Selecting a table in the list will display data from the source and target tables in the bottom pane. Values that differ between source and target are highlighted. As in the top pane, you can uncheck the records you don't want to apply to the target.
"Different" option
The type of difference to display can be selected from the drop-down list. The following options are available:
- Different: Displays all records that differ between the source and target tables.
- Insert: Only show records that do not exist in the target table.
- UPDATE: Show only records that exist in source and target tables but have different values.
- Delete: Only show records that do not exist in the source table.
- Same: Displays records that exist and have the same value in the source and target tables.
- All Rows: Displays all records in the source and target tables.
In our case, selecting "update" or "same" will show zero rows, since there are no rows to update (insert only) and no identical rows:
As before, clicking the " Deploy " button generates and displays a "Deploy Script":
This screen, like the previous ones, includes a Save Profile button that lets you save your settings for future use. This particular screen also has a button to save a " deployment script ".
You can still " rebase " the two databases, or continue to " run " the deployment script. There is a " Continue on error " checkbox here. When this box is checked, the deployment will not stop when an error is encountered.
While the script is running, you can view its progress in the Info Log. It shows the number of records processed and the percentage complete:
After closing the dialog, we can confirm that the table in the sakila2 database has data:
Don't forget to save the config file as it will be used by the batch job.
Create batch jobs
We will now set up periodic replication between the sakila and sakila2 databases using Navicat's autorun tool.
- First, click the " Autorun " button in the main toolbar .
- Then click New Batch Job in the Object toolbar to open the New Batch Job tab.
- Browse source connections, databases and/or schemas in the Objects pane. This will cause saved jobs for that database to appear in the bottom "Available Jobs" pane.
In the Available Jobs pane, select the Data Sync job type, then double-click or drag a job to move it from the Available Jobs list to the Selected Jobs list above. (You can delete jobs from the "Selected jobs" list in the same way.) - Click the Save button on the Autorun toolbar and provide a descriptive name in the Save dialog.
This will enable the " Set Task Scheduler " and " Remove Task Scheduler " buttons.
"General" tab
On the General tab of the Scheduled Task dialog box, you can provide a description for the task. This tab also provides several run options.
You can also choose to run the task as "hidden" and configure it to run on a specific operating system.
- In the Security Options box, you can configure the user or group account under which the task will run. There is an option to run the task whether the user is logged on or not. If you select this option, you will need to provide the operating system user's password in Windows Task Scheduler when saving the schedule.
trigger task
The Triggers tab lists the schedule for the task. Tasks can be configured to run on various schedules, including once, daily, weekly, monthly, and any permutation of each.
Click the New... button to display the New Trigger dialog:
The same task can run according to multiple schedules. For example, we can schedule our database sync task to run on the first day of every month and every second Sunday:
Set up email notifications
sender
Specify the sender's email address. For example: " [email protected] ".
recipient, copy
Specify the email addresses of each recipient, separated by commas or semicolons (;).
Subject
Specify the subject of the email in a custom format.
main body
Fill in the content of the email.
host (SMTP server)
Enter the Simple Mail Transfer Protocol (SMTP) server for outgoing messages.
port
Enter the port number of the outgoing email (SMTP) server.
use authentication
If your SMTP server requires authentication to send emails, you can check this option and enter the "Username" and "Password".
secure connection
Specifies that the connection uses 'TLS', 'SSL' secure connection, or 'never'.
Send test email
Navicat will send a test email to you showing success or failure.
Once you've configured your autojob, you can test it by clicking the Start button in the Autorun toolbar.
merge copy
As the name implies, merge replication combines data from two or more databases into a single database. As an exercise, we will merge the contents of the sakila and sakla2 databases into a third database named "sakila_merged" which will store the merged dataset.
Required Data Synchronization Work
A merge replication job will require us to create and save two data synchronization profiles: one for each source database. These steps are identical to the "Creating a Data Sync Job" section above, so we won't repeat them here.
Create batch jobs
Batch jobs may be triggered by the source database or the target, as we did in the previous section "Snapshot Copy". However, it is usually easiest to trigger the batch job from the target database since they will all be on the same server. We will do the same.
- Click the " Autorun " button in the main toolbar .
- Then, click New Batch Job in the Object toolbar to open a new batch job tab.
- Browse source connections, databases and/or schemas in the Objects pane. This will cause saved jobs for that database to appear in the bottom "Available Jobs" pane.
- In the Available Jobs pane, select the Data Sync job type, then double-click or drag a job to move it from the Available Jobs list to the Selected Jobs list above.
- Click the Save button on the Autorun toolbar and provide a descriptive name in the Save dialog. This will enable the " Set Task Scheduler " and " Remove Task Scheduler " buttons.
- On the General tab of the Scheduled Task dialog box, you can provide a description for the task. This tab also provides several run options.
- The Triggers tab lists the schedule for the task. Tasks can be configured to run on various schedules, including once, daily, weekly, monthly, and any permutation of each. This time two jobs will be run instead of one.
- Once you've configured your autojob, you can test it by clicking the Start button in the Autorun toolbar .
transactional replication
In Merge Replication, only the merged database contains all the latest data. Instead, each source database contains only baseline data, plus what has been inserted since it was first populated. In transactional replication, users receive a complete initial copy of a database, and then receive periodic updates as data changes so that all databases use the same set of data. Keeping multiple databases in sync makes transactional replication the most complex type of replication.
Required Data Synchronization Work
With transactional replication, the number of data synchronization jobs required increases significantly because data must be replicated across all user databases. For example, suppose we have three databases named sakila, sakila2, and sakila3. We can merge and propagate full datasets across all user databases using a total of six data synchronization jobs: three for merging user databases and three for updating them with the merged dataset.
Here's the autorun wizard with all six jobs:
In order to allow enough time for the data merge to complete, it is better to split the job into two parts, the first part to merge the data, and the second part to update the user database with the complete data set after a specified delay.
The following is an example of an automated run that propagates a merged dataset to a user database:
Running this job two hours after running the first job should allow enough time for the merge to complete. So if the first job is scheduled to run at midnight, we'll set the job to start at 2am:
Summarize
In the follow-up of the database synchronization strategy white paper, we introduced how to use Navicat Premium 12's autorun tool to automatically run database replication. Used in conjunction with its synchronization tools, DBAs can set up predefined schedules to automatically run various types of replication.
From Navicat Blog