Remember the Pulsar cluster data migration process

Remember the Pulsar cluster data migration process

background

The company's business adjustment requires migrating a batch of Pulsar clusters running on the cloud to the self-built computer room. So toss the process of Pulsar migration, and record the main operations here.
There are two main methods for Pulsar cluster migration:
1. Migration based on geo-replication cross-computer room replication;
2. Replacement of nodes through node expansion and contraction to complete cluster migration.

Migration based on geo-replication

Based on the cross-computer room replication scheme, the basic architecture is to maintain a consumer inside pulsar to send data to the replica cluster. By enabling replication subscription, the consumer consumption information is synchronized to the replica cluster to realize seamless switching of cross-computer room consumption (the default replication interval is 1s, and there may be 1s of repeated consumption during consumption switching). This method is suitable for clusters with fewer topics
. Migration scenarios where only part of the data is migrated.

Operating procedures:

1. Build a new cluster B
2. Create the tenant, namespace, and partitioned-Topic that need to be copied on the new cluster B as needed. (Ordinary topics will be automatically innovated. Partitioned topics need to be manually created and the number of partitions specified. Otherwise, topics for each partition will be automatically created in the form of ordinary topics.)
3. Add new cluster B to old cluster A:

bin/pulsar-admin clusters create --broker-url pulsar://gz-idc-pulsar-metric-split.company.info:6650 --url http://gz-idc-pulsar-metric-split.company.info:8080 pulsar-gz-idc-metric-split-cluster

4. On cluster A, update cluster B to the tenant and namespace that need to be copied

bin/pulsar-admin tenants update neo -c pulsar-gz-txcloud-cluster,pulsar-gz-idc-metric-split-cluster
bin/pulsar-admin namespaces set-clusters neo/metrics -c pulsar-gz-txcloud-cluster,pulsar-gz-idc-metric-split-cluster

5. Update the Topic consumer and start the replication subscription
6. Verify whether the consumption subscription is started, and verify topic data replication

bin/pulsar-admin topics partitioned-stats-internal persistent://neo/metrics/neo-metrics

• Executing in cluster A, the cursor that can see the synchronous data has been created
  

• In cluster A, you can see that the consumer program has enabled subscription replication
  

7. Switch the consumer and producer programs to the serviceURL of cluster B

Migration based on scaling

1. First, add the migrated target node to the cluster, and deploy Broker and bookkeeper services on the new node. For specific operations, please refer to: official document pulsar bare-metal deployment. After deployment, you need to confirm that the Broker and Bookie services have been registered on zk.

2. Offline the Broker service in the old computer room. Broker is compatible with adaptive expansion and contraction. After confirming that there is no problem with the machine load, just go offline one by one. The client connection will automatically reconnect to the remaining available Broker services.

3. Offline Bookie.

   1. Make sure the number of remaining Bookies >= Write Quorum >= Ack Quorum
   2. Enable automatic copy: bookkeeper shell autorecovery -enable; Set bookie to read-only mode: modify conf/bk_server.conf readOnlyModeEnabled=true,forceReadOnlyBookie=true
   3. bookkeeper shell listunderreplicatedMake sure there are no bookie lists to copy
   4. Wait until there is no data inflow and log off the Bookie servicebin/bookkeeper-daemon.sh stop bookie