Debezium Daily Sharing Series: Customize Debezium Signal Sending and Notification
Debezium 2.3 introduces new improvements in the signal and notification functionality. In addition to the predefined signal and notification channels provided by Debezium, you can also set up new signal and notification channels. This capability enables users to customize the system to meet their unique needs and integrate it with existing infrastructure or third-party solutions. It enables effective monitoring and proactive response to data changes by precisely capturing and communicating signal events and triggering notifications through preferred channels.
Debezium Daily Sharing Series: Debezium Signaling and Notifications - Part 1
1. Custom signal and notification channels
In Debezium, signal and notification channels can be customized to meet specific requirements. For example, we can achieve customization by creating HTTP channels for signals and notifications. This HTTP channel receives signals from http endpoints and can send notifications back to the endpoint after the signal has been delivered.
Let's explore an example showing how to create and utilize HTTP signaling and notification channels using the Debezium Postgres connector, a mock server to send signals, and Postbin to receive notifications through an http endpoint.
Set up the HTTP signal channel:
- Configure the Debezium Postgres connector to receive signals when relevant database changes occur.
- Set up a service to send signals to Debezium using the HTTP channel. The service could be a database, a third-party application, or any other system that can send http requests. In this example we will use a mock server to send signals to Debezium. Mock Server is a service that can be used to mock http requests and responses.
- Configure the mock server to send signals through the http endpoint using the appropriate HTTP method (eg POST).
- Customize the HTTP channel settings as needed to define the http endpoint URL, authentication, headers and any other parameters.
Set up the HTTP notification channel:
- Once Debezium receives and processes the signal, it can trigger a notification to the http endpoint. In this example, we will use the HTTP channel to send notifications to the Postbin bin. Postbin is a service that can be used to receive http requests and view request details.
- Customize the HTTP channel settings for notifications, create a bin in Postbin, and define the http endpoint URL, authentication, headers, and any other parameters as required.
- Forward the notification event to the http endpoint, the Postbin bin, using the appropriate HTTP method (eg POST). The notification payload can be customized as needed.
The full source code for this example in the blog post is available in the http-signal-notification directory of the Debezium examples repository.
Create a java project to build HTTP signaling and notification channels. Run the following command to create a new java project using Maven:
mvn archetype:generate
-DgroupId=io.debezium.examples
-DartifactId=http-signaling-notification
Add the following dependencies to the pom.xml file for Debezium versions (2.3 and higher):
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-core</artifactId>
<version>2.3.0.Final</version>
</dependency>
To receive signals with a mock server, create a Docker Compose file that defines the mock server service. The mock server service is configured as follows:
services:
mockServer:
image: mockserver/mockserver:latest
ports:
- 1080:1080
environment:
- MOCKSERVER_WATCH_INITIALIZATION_JSON=true
- MOCKSERVER_INITIALIZATION_JSON_PATH=/config/initializerJson.json
volumes:
- ./initializerJson.json:/config/initializerJson.json
Set the environment variables MOCKSERVER_WATCH_INITIALIZATION_JSON and MOCKSERVER_INITIALIZATION_JSON_PATH to enable the mock server to watch for changes in the initialization JSON file and specify its path. The initializerJson.json file containing the signal's http request and response information is installed into the mock server container.
The initializerJson.json file defines a simulated http request to the path /api/signal with the query string parameter code=10969. When the mock server receives this request, it will respond with a JSON body containing the id, type, and data. The status code of the response is 200, indicating a successful response. The initializerJson.json file is defined as follows:
[
{
"httpRequest" : {
"method" : "GET",
"path" : "/api/signal",
"queryStringParameters" : {
"code" : ["10969"]
}
},
"httpResponse" : {
"body": "{
\"id\":\"924e3ff8-2245-43ca-ba77-2af9af02fa07\",\"type\":\"log\",\"data\":{
\"message\": \"Signal message received from http endpoint.\"}}",
"statusCode": 200
}
}
]
- id : Arbitrary unique string identifying the signal instance.
- type : The type of signal to send. In this example, the type is log, which requests that the connector add an entry to the connector's log file. After processing a signal, the connector prints the specified message in the log.
- data : JSON-formatted parameters passed to the signal event. In this example, the message parameter is passed to the signal event.
Create an HTTP signal channel by implementing the SignalChannelReader interface, as follows:
public class HttpSignalChannel implements SignalChannelReader {
private static final Logger LOGGER = LoggerFactory.getLogger(HttpSignalChannel.class);
public static final String CHANNEL_NAME = "http";
private static final List<SignalRecord> SIGNALS = new ArrayList<>();
public CommonConnectorConfig connectorConfig;
@Override
public String name() {
(1)
return CHANNEL_NAME;
}
@Override
public void init(CommonConnectorConfig connectorConfig) {
(2)
this.connectorConfig = connectorConfig;
}
@Override
public List<SignalRecord> read() {
(3)
try {
String requestUrl = "http://mockServer:1080/api/signal?code=10969";
// send http request to the mock server
HttpClient httpClient = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(requestUrl))
.GET()
.header("Content-Type", "application/json")
.build();
// read the response
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() == 200) {
ObjectMapper mapper = new ObjectMapper();
String responseBody = response.body();
// parse the response body
JsonNode signalJson = mapper.readTree(responseBody);
Map<String, Object> additionalData = signalJson.has("additionalData") ? mapper.convertValue(signalJson.get("additionalData"), new TypeReference<>() {
}) : new HashMap<>();
String id = signalJson.get("id").asText();
String type = signalJson.get("type").asText();
String data = signalJson.get("data").toString();
SignalRecord signal = new SignalRecord(id, type, data, additionalData);
LOGGER.info("Recorded signal event '{}' ", signal);
// process the signal
SIGNALS.add(signal);
} else {
LOGGER.warn("Error while reading signaling events from endpoint: {}", response.statusCode());
}
} catch (IOException | InterruptedException e) {
LOGGER.warn("Exception while preparing to process the signal '{}' from the endpoint", e.getMessage());
e.printStackTrace();
}
return SIGNALS;
}
@Override
public void close() {
(4)
SIGNALS.clear();
}
}
- The name() method returns the name of the signal channel. To enable Debezium to use a channel, specify the name http in the signal.enabled.channels property of the connector.
- The init() method can be used to initialize specific configuration, variables or connections required by the http channel.
- The read() method reads signals from the http endpoint and returns a list of SignalRecord objects to be processed by the Debezium connector.
- The close() method closes all allocated resources.
Create a notification channel by implementing the NotificationChannel interface, as follows:
public class HttpNotificationChannel implements NotificationChannel {
private static final Logger LOGGER = LoggerFactory.getLogger(HttpNotificationChannel.class);
public static final String CHANNEL_NAME = "http";
private static final String NOTIFICATION_PREFIX = "[HTTP NOTIFICATION SERVICE]";
@Override
public String name() {
(1)
return CHANNEL_NAME;
}
@Override
public void init(CommonConnectorConfig config) {
(2)
// custom configuration
}
@Override
public void send(Notification notification) {
(3)
LOGGER.info(String.format("%s Sending notification to http channel", NOTIFICATION_PREFIX));
String binId = createBin();
sendNotification(binId, notification);
}
private static String createBin() {
// Create a bin on the server
try {
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("https://www.toptal.com/developers/postbin/api/bin"))
.POST(HttpRequest.BodyPublishers.ofString(" "))
.build();
HttpClient httpClient = HttpClient.newHttpClient();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() == HTTP_CREATED) {
String binId = response.body().replaceAll(".*\"binId\":\"([^\"]+)\".*", "$1");
LOGGER.info("Bin created: " + response.body());
return binId;
}
} catch (URISyntaxException | InterruptedException | IOException e) {
throw new RuntimeException(e);
}
return null;
}
private static void sendNotification (String binId, Notification notification) {
// Get notification from the bin
try {
ObjectMapper mapper = new ObjectMapper();
String notificationString = mapper.writeValueAsString(notification);
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("https://www.toptal.com/developers/postbin/" + binId))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(notificationString))
.build();
HttpClient httpClient = HttpClient.newHttpClient();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() == HTTP_OK) {
LOGGER.info("Notification received : " + response.body());
}
} catch (URISyntaxException | InterruptedException | IOException e) {
throw new RuntimeException(e);
}
}
@Override
public void close() {
(4)
}
}
- The name() method returns the name of the notification channel. To enable Debezium to use channels, specify http in the notification.enabled.channels property of the connector.
- The init() method can be used to initialize specific configuration, variables or connections required by the channel.
- The send() method sends a notification to the channel. The notification contains a SignalRecord object that is handled by the Debezium connector.
- The close() method closes all allocated resources.
Declare the HTTP signal and notification channels under the io.debezium.pipeline.signal.SignalChannelReader and io.debezium.pipeline.notification.channels.NotificationChannel files in the META-INF/services directory, respectively.
Compile the Java project and export it as a JAR file. This can be done using Maven or your favorite build tool. Copy the JAR file to the directory containing the JAR file for the Debezium connector to use. For example, if you want to use custom signal and notification channels with the Debezium Postgres connector, copy the JAR file to the /kafka/connect/debezium-connector-postgres directory.
This example provides a Docker Compose file that defines the necessary services, including Mock Server, Zookeeper, Kafka Connect, and a Postgres database.
To start the service, run the following command:
export DEBEZIUM_VERSION=2.3
docker-compose up -d
After making sure the service is up and running, and the Postgres database is ready to accept connections, the next step is to register the connector. This involves creating a connector configuration file. Let's create a file called register-postgres.json with the following properties:
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": 1,
"database.hostname": "postgres",
"database.port": 5432,
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "postgres",
"topic.prefix": "dbserver1",
"schema.include.list": "inventory",
"signal.enabled.channels": "http", 1
"notification.enabled.channels": "http" 2
}
}
- The signal.enabled.channels property specifies the signal channels to be used by the connector. In this case, the connector uses the http signaling channel.
- The notification.enabled.channels property specifies the notification channels to be used by the connector. In this case, the connector uses the http notification channel.
Now that we have the connector configuration file ready, we can register the connector with Kafka Connect by executing the following command:
curl -i -X POST -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/ \
-d @register-postgres.json
After the connector is successfully registered, you can view the connector log to observe signal events. These logs provide insight into the processing and progress of the connector, including any signal-related information. You will encounter log messages similar to the following:
Recorded signal event 'SignalRecord{id='924e3ff8-2245-43ca-ba77-2af9af02fa07', type='log', data='{
"message":"Signal message received from http endpoint."}', additionalData={}}' [io.debezium.examples.signal.HttpSignalChannel]
Additionally, you may notice log messages related to notification events sent to the mailbox. For example:
[HTTP NOTIFICATION SERVICE] Sending notification to http channel [io.debezium.examples.notification.HttpNotificationChannel]
Bin created: {
"binId":"1688742588469-1816775151528","now":1688742588470,"expires":1688744388470} [io.debezium.examples.notification.HttpNotificationChannel]
It provides information about notification events such as creation of a bin with a unique identifier (binId) and other related details. To retrieve a notification event from Postbin, get the binId from the log message and use it to request the corresponding notification event from Postbin. To view notification events, you can access Postbin using the following URL: https://www.toptal.com/developers/postbin/b/:binId. Replace :binId in the URL with the actual binId obtained from the connector log.
The notification event sent to Postbin looks like this:
2. Conclusion
In this tutorial, we explored how to create custom signal and notification channels for Debezium connectors. We created a custom signal channel to receive signal events from the HTTP endpoint. We also created a custom notification channel for sending notification events to the HTTP endpoint.
Debezium's comprehensive signaling and notification system seamlessly integrates with third-party solutions to keep users informed of the status and progress of Debezium connectors. The extensibility of the system enables users to customize signal and notification channels to meet their customized needs.