When we synchronize SQL Server data to MySQL or apply SQL Server in the field of big data or other scenarios, we need to monitor the data changes of the source database in real time, then the CDC of SQL Server can come in handy. This article is my introduction to SQL Server Some of the operations used by CDC are my notes, which may help you. I will introduce how to apply CDC in the code later.

If your SQL Server uses the Linux version, this solution is also applicable. If you want to use SSMS, you need to use a Windows machine to remotely connect to your SQL server.

1. Turn on the proxy server

1.1. Before opening

1.2. Execute the open command

sp_configure 'show advanced options', 1;

GO

RECONFIGURE;

Go

sp_configure 'Agent XPs',1;

Go

RECONFIGURE;

Go

1.3. After opening

1.4. Manually start the proxy service

-- 查看sqlserver代理的进程是否启用（如果显示stoped(需手动启动)，则需启动SQL Server代理）

EXEC master.dbo.xp_servicecontrol N'QUERYSTATE', N'SQLSERVERAGENT'

You can right-click "SQL Server Agent" and click Start to start the agent manually;

2. Check and enable the CDC service of the database

2.1. Query the CDC opening status of the database

select is_cdc_enabled from sys.databases where name='HM_5001';

If the query result is 0, it means that the CDC service of the database HM_5001 has not been started.

2.2. Enable the CDC function at the database level

-- 开启数据库CDC

ALTER AUTHORIZATION ON DATABASE::[HM_5001] TO [sa];

if exists(select 1 from sys.databases where name='HM_5001' and is_cdc_enabled=0)

begin

    exec sys.sp_cdc_enable_db

end

;

-- 查询数据库是否开启CDC
select is_cdc_enabled from sys.databases where name='HM_5001';

3. Add CDC-specific file groups and files

-- 查询已存在的文件
SELECT name, physical_name FROM sys.master_files WHERE database_id = DB_ID('HM_5001');

-- 添加CDC文件组和文件
ALTER DATABASE HM_5001 ADD FILEGROUP CDC1;

ALTER DATABASE HM_5001
ADD FILE
(
  NAME= 'HM_5001_CDC1',

  FILENAME = 'D:\DATA\HM_5001_CDC1.ndf',

  SIZE = 100MB
)
TO FILEGROUP CDC1;

4. The operation enables table-level CDC (note: there must be a primary key or unique index in the table)

4.1. View the table with CDC enabled

SELECT name,is_tracked_by_cdc FROM sys.tables WHERE  is_tracked_by_cdc = 1;

4.2. Enable table-level CDC (there must be a primary key or unique index in the table)

-- 开启t1表的CDC

IF EXISTS(SELECT 1 FROM sys.tables WHERE name='t1' AND is_tracked_by_cdc = 0)
BEGIN
    EXEC sys.sp_cdc_enable_table
        @source_schema = 'dbo', 
        @source_name = 't1', 
        @capture_instance = NULL, 
        @supports_net_changes = 1,
        @role_name = NULL, 
        @index_name = NULL, 
        @captured_column_list = NULL, 
        @filegroup_name = 'CDC1' 
END;

Remarks: Replace "t1" in sequence with multiple tables and continue execution

After opening, you can see the following picture:

Every time CDC opens a table, a new table of cdc.dbo_table name_CT will be added here, and this table is the table for recording table data table updates.

If there is data insertion, update, and deletion, it will be recorded here.

For __$operation column: 1=delete, 2=insert, 3=update (old value), 4=update (new value)

For the __$start_lsn column: Since the change is the transaction log of the source and database, the start sequence number (LSN) of its transaction log is saved here

However, Microsoft does not recommend querying such tables directly. It is recommended to use cdc.fn_cdc_get_all_changes_<capture instance> and cdc.fn_cdc_get_net_changes_<capture_instance> to query.

5. Verify whether CDC is successfully enabled

-- 查看哪些表开启了CDC
select name, is_tracked_by_cdc from sys.tables where is_tracked_by_cdc = '1';

-- 查看指定表是否开启了CDC
select name, is_tracked_by_cdc from sys.tables where object_id = OBJECT_ID('dbo.t1');

Additional commands (turn off CDC)

-- 关闭表(“dbo.t1”)的CDC
EXEC sys.sp_cdc_disable_table @source_schema = 'dbo', @source_name = 't1', @capture_instance = 'all';

-- 禁用数据库所有实例CDC
EXEC sys.sp_cdc_disable_db;

-- 查看CDC的job
EXEC sys.sp_cdc_help_jobs
GO

Warm reminder (stepping on the pit record):

1. The database installed under Linux reports an error: SQLServerAgent is not currently running so it cannot be notified of this action

Let's look at the official documents (click to enter):

The reason is found to be the hostname problem in Linux. The SqlServer agent of the Linux version of SqlServer 2019 needs to use the Linux hostname, and only uses the first 15 digits of the hostname .

When the agent logs in, since this is a new server, the hostname has exceeded 15 characters, but only 15 characters are intercepted when the agent logs in, resulting in a mismatch in the account name. If you find the problem, then solve the problem

Solution 1 : Permanently modify hostname (requires server restart)

first step:

vim /etc/hostname

#把你那跟油腻大叔大裤衩子似的name改成不超过15个字符的name

Esc
:wq Enter

Step two:

Restart the Linux server, restart the SQL Server

get it done

Solution 2 : Temporary modification (invalid after Linux restart)

Execute the command directly on Linux:

hostname your new name

Use the hostname command to check whether the modification is successful

After modification, restart the Linux SqlServer service according to the official Microsoft SqlServer documentation .

2. If you find the following prompt:

Executed as user XXX. Unable to delete one or more low water meter changes obsolete change table entries for capture instance of database XX. Execute the command 'delete top(@p1) from [dbo.XXX] where _$start_lsn < @p2'. Please rerun the transaction. . .

At this time, you should realize that a deadlock has occurred.

You can try the following solutions:

-- ----------对作业的更改------------------ 

EXEC sys.sp_cdc_change_job
 @job_type = 'capture' 
 ,@maxtrans = 1000      --每个扫描循环可以处理的最多事务数 
 ,@maxscans = 10        --为了从日志中提取所有行而要执行的最大扫描循环次数 
 ,@continuous = 1       --连续运行最多处理(max_trans * max_scans)个事务 
 ,@pollinginterval = 5 
 ;
 
 EXEC sys.sp_cdc_change_job  
 @job_type = 'cleanup' 
 ,@retention = 4320     --更改行将在更改表中保留的分钟数
 ,@threshold = 5000     --清除时可以使用一条语句删除的删除项的最大数量

 -- ----上述对作业的更改，更改后需重启作业----- 
 EXEC sys.sp_cdc_stop_job @job_type = N'capture'; 
 EXEC sys.sp_cdc_start_job @job_type = N'cleanup'; 

-- 执行命令：
EXECUTE sys.sp_cdc_change_job
@job_type = N'cleanup',
@threshold=2000
GO

-- 把threshold设置为2000。

--- 重启作业：

EXEC sys.sp_cdc_stop_job @job_type = N'capture'; 
EXEC sys.sp_cdc_start_job @job_type = N'cleanup'; 

-- 完成

-- View jobs

EXEC sys.sp_cdc_help_jobs
GO

3. If it is SQL Server on Windows, and the computer is restarted

If so: [Microsoft[ODBC Driver 17 for SQL Server[SQL Serverl During a read in file D:DATAHM 5002 CDC1.ndf at offset 0x00000000012000, the operating system has returned error 21 Device not ready. . Other messages in the SQL Server error log and system event log may provide more details. This is a serious system-level error condition that threatens the integrity of the database and must be corrected immediately. Please perform a full database consistency check (DBCC CHECKDB). This error can be caused by many factors, see SQL Server Books Online for more information. (823)

Or look at the execution steps of the job like this:

Description of problem found:

The first case: After turning on the computer, the database cannot be viewed, and the above error is displayed after clicking the table;

The second case: the table can be viewed normally, but the table under CDC prompts the above error;

The reason--" The computer restarted : this is because the database is not suspended (due to the movement of the hard disk, etc.)"

Suspended (waiting, blocking) processes in the operating system can be defined as processes that are temporarily eliminated from the memory. The resources of the machine are limited. In the case of insufficient resources, the operating system makes reasonable arrangements for the programs in the memory. Some of the processes are temporarily transferred out of the memory. When conditions permit, they will be transferred back to the memory by the operating system again, and re-enter the state waiting to be executed, that is, the ready state. The system does not take any action for a certain period of time.

Solution:

method 1:

win+R to open the command line window, enter services.msc and press Enter, open the service, restart the SQL Server service, and then reopen the database software. Include the database service and the proxy service because the proxy service depends on the database service.

Method 2:

1) Click the "Start" menu on the task bar of the operating system, select the "Run" command, enter the "cmd" command in the drop-down list box, and click the "OK" button.

2) In the cmd window, stop and restart the SQL Server service.

Stop SQL Server

NET STOP MSSQLSERVER

Start SQL Server

NET START MSSQLSERVER

Just reconnect to the database.

Operation Guide for Enabling CDC Change Data Capture in SQL Server Database

Foreword: