Test: Oracle 19c RAC Add private network

Recently, a customer needs to add heartbeat network on a 12.2 version of the RAC environment, consider the way the heartbeat network will be changed before the asm special. I am currently RAC test environment is only 19c (19c 12c of the final stable version), directly under the testing process to verify the note.

• 1. Test Environment Overview
• 2. Add the heartbeat network
• 3. Modify the network asm
• 4. Restart has a cluster
• 5. Check the heartbeat status

1. Test Environment Overview

First, my test environment is this:

[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  cluster_interconnect,asm

2. Add the heartbeat network

After the host level to add a new card enp0s9, configure the corresponding IP address, you can add enp0s9 heartbeat network:

[grid@db195 ~]$ oifcfg setif -global enp0s9/10.10.1.0:cluster_interconnect
[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  cluster_interconnect,asm
enp0s9  10.10.1.0  global  cluster_interconnect

3. Modify the network asm

If you need to confirm prior to asm dedicated, you can then modify:

[grid@db195 ~]$ oifcfg setif -global enp0s8/10.10.1.0:asm
[grid@db195 ~]$ oifcfg getif
enp0s3  192.168.1.0  global  public
enp0s8  10.10.1.0  global  asm
enp0s9  10.10.1.0  global  cluster_interconnect

At this point of view the configuration of the other nodes RAC ensure the same.

4. Restart has a cluster

Use ifconfig -a to view the new card information enp0s9, or not HAIP takeover (no corresponding address information segment 169)

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 66  bytes 19094 (18.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1170 (1.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Need to use crsctl stop has shut down each node in the cluster:

[root@db193 ~]# /u01/app/19.3.0/grid/bin/crsctl stop has
[root@db195 ~]# /u01/app/19.3.0/grid/bin/crsctl stop has

[root@db193 ~]# /u01/app/19.3.0/grid/bin/crsctl start has
[root@db195 ~]# /u01/app/19.3.0/grid/bin/crsctl start has

5. Check the heartbeat status

After the restart, again observe ifconfig -a to see the new card enp0s9 have HAIP corresponding to the address, but no enp0s8, because asm enp0s8 just use:

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.195  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::5a1e:e7de:fe1a:a286  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)
        RX packets 38565  bytes 27624748 (26.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 69723  bytes 72018893 (68.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 12510  bytes 7500503 (7.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14723  bytes 17562072 (16.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.3.55  netmask 255.255.224.0  broadcast 169.254.31.255
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)

If the card is not modified before enp0s8 dedicated to asm, which is asm, cluster_interconnect, then enp0s8 card will remain there HAIP address:

--设置enp0s8为asm,cluster_interconnect
[grid@db195 ~]$ oifcfg setif -global enp0s8/10.10.1.0:asm,cluster_interconnect

--重启集群后，ifconfig -a就会发现enp0s8网卡保持有HAIP地址：
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.195  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::5a1e:e7de:fe1a:a286  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)
        RX packets 41817  bytes 28559675 (27.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 89166  bytes 88323795 (84.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.7.148  netmask 255.255.240.0  broadcast 169.254.15.255
        ether 08:00:27:1d:85:d4  txqueuelen 1000  (Ethernet)

enp0s9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.196  netmask 255.255.255.0  broadcast 10.10.1.255
        inet6 fe80::2549:d14d:7dfd:61c4  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)
        RX packets 103509  bytes 77082525 (73.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 237436  bytes 290562629 (277.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s9:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.22.254  netmask 255.255.240.0  broadcast 169.254.31.255
        ether 08:00:27:59:1a:a6  txqueuelen 1000  (Ethernet)

NOTE: If you add enp0s9 heartbeat heartbeat network card and not the same as before, such as 10.10.2.195, but also pay attention and oifcfg setif enp0s9 / 10.10.2.0: cluster_interconnect, network and new network card network consistent with each other match.
The next thing is to make sure the summary physical NICs increased heartbeat and heartbeat, then the database level directly oifcfg increase or modify the need to restart the cluster has entered into force.