ORACLE+RAC principle and practice series: 1 RAC installation

Demand and environment

Install and deploy ORACLE 19c RAC environment. 55.34.4.153, 55.34.4.154; Two vmware hosts are configured with two network cards respectively , and 5 IPs (public IP) on the same network segment are applied for , and two vmware hosts are configured with three shared disks (OCRDG, DATADG, FRADG);

Preliminary preparation

Create grid user (dual node)

useradd grid

groupadd oinstall

groupadd dba

groupadd oper

groupadd asmadmin

groupadd asmoper

groupadd asmdba

groupadd racdba

usermod -g oinstall -G dba,oper,asmadmin,asmoper,asmdba,racdba grid

uid=1779(grid) gid=1778(oinstall) groups=1778(oinstall),1779(dba),1780(oper),504(asmadmin),505(asmoper),506(asmdba),1784(racdba)

Two-node configuration grid/root mutual trust between the two users, including mutual trust to the own node and mutual trust to the other node

Create installation directory (dual node)

mkdir gridhome
mkdir gridbase
mkdir gridInventory
chown grid:oinstall grid*

Shared disk persistent binding

#!/bin/bash

for i in c d e f
do
echo $i
echo "KERNEL==\"sd?\",SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\",RESULT==\"`/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", SYMLINK+=\"asm-disk$i\",OWNER=\"grid\", GROUP=\"asmadmin\",MODE=\"0660\""  >>/etc/udev/rules.d/99-oracle-asmdevices.rules

done

/sbin/udevadm trigger --type=devices --action=change
ls -l /dev/asm*

exit 0

Shared disk format

dd if=/dev/zero of=/dev/sdc bs=512K count=1

dd if=/dev/zero of=/dev/sdd bs=512K count=1

dd if=/dev/zero of=/dev/sde bs=512K count=1

Network Configuration

#########################################RAC begin

#pubilc ip

55.34.4.153 ora19node1

55.34.4.154 ora19node2

#private ip

192.168.124.5 ora19node1-priv

192.168.124.6 ora19node2-priv

#vitrual ip

55.34.4.157 ora19node1-vip

55.34.4.158 ora19node2-vip

#scan ip

55.34.4.159 ora19nodescan

##############################RAC END################################

/etc/hosts configuration (dual node)

Including public IP, public network VIP, scan IP and private network IP;

ifconfig ens224 192.168.124.5 up

ifconfig ens224 192.168.124.6 up

ip address delete 192.168.124.5 dev ens224

The two-node private network cards go online separately

Formal installation (GI)

Solution grid home
Permission modification

chown  -R grid:oinstall   /data/gridhome/

Verify with grid user

./runcluvfy.sh stage -post hwos -n ora19node1,ora19node2 -verbose	Check hardware and OS
./runcluvfy.sh stage -pre crsinst -n ora19node1,ora19node2 -fixup -verbose	Check crs installation

Note: If it fails, an automatic repair script will appear, which can be executed under two nodes as the root user (please refer to the execution result of runclufy.sh). My vmware host has several dependent packages not installed, such as tables.

/tmp/CVU_19.0.0.0.0_grid/runfixup.sh

libaio-devel-0.3.109	yum install libaio-devel
nfs-utils-1.2.3-15
compat-libstdc++-33-3.2.3

Modify shm size (modify /etc/fstab)

#for rac

tmpfs /dev/shm tmpfs defaults,size=8G 0 0

mount -o remount /dev/shm

注意：上面几个步骤都是本人通过runcluvfy.sh 脚本检查出来的，读者在实践过程中，用此脚本多执行几次，逐一解决问题即可。

创建安装响应文件

vim grid_install.rsp

此相应文件在grid_home 有模板，读者请仔细研读模板（如GI安装类型、各种用户配置等），再填写各个参数。

[grid@sgpbnk01kf ~]<20200902 08:37:39>$ cat grid19_install_new.rsp

oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0

####A

INVENTORY_LOCATION=/data/gridInventory

oracle.install.option=CRS_CONFIG

ORACLE_BASE=/data/grid19base

####B

oracle.install.asm.OSDBA=asmdba

oracle.install.asm.OSOPER=asmoper

oracle.install.asm.OSASM=asmadmin

####C

oracle.install.crs.config.scanType=LOCAL_SCAN

oracle.install.crs.config.SCANClientDataFile=

oracle.install.crs.config.gpnp.scanName=sgpbnkkfscan

oracle.install.crs.config.gpnp.scanPort=1530

####D

oracle.install.crs.config.ClusterConfiguration=STANDALONE

oracle.install.crs.config.configureAsExtendedCluster=false

oracle.install.crs.config.memberClusterManifestFile=

oracle.install.crs.config.clusterName=sgpbnkkfclus

oracle.install.crs.config.gpnp.configureGNS=false

oracle.install.crs.config.autoConfigureClusterNodeVIP=false

oracle.install.crs.config.gpnp.gnsOption=

oracle.install.crs.config.gpnp.gnsClientDataFile=

oracle.install.crs.config.gpnp.gnsSubDomain=

oracle.install.crs.config.gpnp.gnsVIPAddress=

oracle.install.crs.config.sites=

oracle.install.crs.config.clusterNodes=sgpbnk01kf:sgpbnk01kf-vip,sgpbnk02kf:sgpbnk02kf-vip

oracle.install.crs.config.networkInterfaceList=eth0:55.14.64.0:1,ens256:192.168.124.0:5

oracle.install.crs.configureGIMR=

oracle.install.asm.configureGIMRDataDG=false

####E

oracle.install.crs.config.storageOption=CLIENT_ASM_STORAGE

oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/dev/asm-diskf

oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/dev/asm-diskf

####F

oracle.install.crs.config.useIPMI=false

oracle.install.crs.config.ipmi.bmcUsername=

oracle.install.crs.config.ipmi.bmcPassword=

####G

oracle.install.asm.SYSASMPassword=Password12345A

oracle.install.asm.diskGroup.name=OCRDG

oracle.install.asm.diskGroup.redundancy=EXTERNAL

oracle.install.asm.diskGroup.AUSize=4

oracle.install.asm.diskGroup.FailureGroups=

oracle.install.asm.diskGroup.disksWithFailureGroupNames=

oracle.install.asm.diskGroup.disks=/dev/asm-diskf

oracle.install.asm.diskGroup.quorumFailureGroupNames=

oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm-disk*

oracle.install.asm.monitorPassword=Password12345A

oracle.install.asm.gimrDG.name=OCRDG

oracle.install.asm.gimrDG.redundancy=

oracle.install.asm.gimrDG.AUSize=4

oracle.install.asm.gimrDG.FailureGroups=

oracle.install.asm.gimrDG.disksWithFailureGroupNames=

oracle.install.asm.gimrDG.disks=/dev/asm-diskf

oracle.install.asm.gimrDG.quorumFailureGroupNames=

oracle.install.asm.configureAFD=

oracle.install.crs.configureRHPS=

##END

./gridSetup.sh -silent -ignorePrereq -responseFile /home/grid/grid_install.rsp

静默安装

[grid@sgpbnk01kf gridhome]<20200902 08:45:30>$ ./gridSetup.sh -silent -ignorePrereq -responseFile /home/grid/grid_install.rsp

Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.

CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /tmp/GridSetupActions2020-09-02_08-45-59AM/gridSetupActions2020-09-02_08-45-59AM.log

ACTION: Identify the list of failed prerequisite checks from the log: /tmp/GridSetupActions2020-09-02_08-45-59AM/gridSetupActions2020-09-02_08-45-59AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

The response file for this session can be found at:

/u01/gridhome/install/response/grid_2020-09-02_08-45-59AM.rsp

You can find the log of this install session at:

/tmp/GridSetupActions2020-09-02_08-45-59AM/gridSetupActions2020-09-02_08-45-59AM.log

As a root user, execute the following script(s):

1. /u01/gridInventory/orainstRoot.sh

2. /u01/gridhome/root.sh

Execute /u01/gridInventory/orainstRoot.sh on the following nodes:

[sgpbnk01kf, sgpbnk02kf]

Execute /u01/gridhome/root.sh on the following nodes:

[sgpbnk01kf, sgpbnk02kf]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes.

Successfully Setup Software with warning(s).

As install user, execute the following command to complete the configuration.

/u01/gridhome/gridSetup.sh -executeConfigTools -responseFile /home/grid/grid19_install_new.rsp [-silent]

一次性成功安装。

双节点依次执行root脚本，注意要依次执行：现在一个节点执行，并观察日志，确保成功再在第二个节点执行

sh /data/gridInventory/orainstRoot.sh
sh  /data/gridhome/root.sh

[root@sgpbnk01kf ~]<20200902 08:51:57># sh /u01/gridInventory/orainstRoot.sh

Changing permissions of /u01/gridInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/gridInventory to oinstall.

The execution of the script is complete.

[root@sgpbnk01kf ~]<20200902 08:53:01># sh /u01/gridhome/root.sh

Check /u01/gridhome/install/root_sgpbnk01kf_2020-09-02_08-53-03-790273956.log for the output of root script

GI安装后的自动化配置

./gridSetup.sh -silent -executeConfigTools -responseFile /home/grid/grid_install.rsp

GI检查

按照上面的流程，GI安装完毕，可以运行相关命令检查GI的状态。

crsctl status resource -t	查看资源状态，按表格显示-t
srvctl status scan	查看scan的状态
crsctl status resourcegroup -t	查看资源组状态
crsctl status type	查看类型状态（不懂）
crsctl status serverpool	查看服务池状态
crsctl status server	查看服务状态
crsctl check crs	检查集群同步服务
crsctl check ctss	检查集群时间同步(Cluster Time Synchronization Service)
crsctl get cpu equivalency	获取CPU 信息
crsctl get node role status -all	获取节点角色
Olsnodes	查看节点数
crsctl query css votedisk	查询vote盘
srvctl status asm	查看ASM状态
crsctl check cluster	查看集群状态

正式安装（DB）

用户

uid=1778(oracle) gid=1778(oinstall) groups=1778(oinstall),1779(dba),1780(oper),504(asmadmin),506(asmdba),1781(backupdba),1782(dgdba),1783(kmdba),1784(racdba)

useradd oracle

groupadd dba

groupadd oper

groupadd asmadmin

groupadd asmdba

groupadd backupdba

groupadd dgdba

groupadd kmdba

groupadd racdba

usermod -g oinstall -G dba,oper,asmadmin,asmdba,backupdba,dgdba,kmdba,racdba oracle

配置oracle 用户互信

磁盘规划

节点1	节点2
/u01/dbhome	/u01/dbhome
/u01/dbbase	/u01/dbbase
/u01/dbInventory	/u01/dbInventory

DB安装检查

DB安装检查(grid_home目录)

./runcluvfy.sh stage -pre dbinst -n "ora19node1,ora19node2" -verbose

编写响应文件

[oracle@sgpbnk01kf ~]<20200902 09:44:13>$ cat db19c_install.rsp

oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0

oracle.install.option=INSTALL_DB_SWONLY

UNIX_GROUP_NAME=oinstall

INVENTORY_LOCATION=/u01/dbInventory

ORACLE_HOME=/u01/dbhome

ORACLE_BASE=/u01/dbbase

oracle.install.db.InstallEdition=EE

oracle.install.db.OSDBA_GROUP=dba

oracle.install.db.OSOPER_GROUP=oper

oracle.install.db.OSBACKUPDBA_GROUP=backupdba

oracle.install.db.OSDGDBA_GROUP=dgdba

oracle.install.db.OSKMDBA_GROUP=kmdba

oracle.install.db.OSRACDBA_GROUP=racdba

oracle.install.db.rootconfig.executeRootScript=true

oracle.install.db.rootconfig.configMethod=ROOT

oracle.install.db.rootconfig.sudoPath=

oracle.install.db.rootconfig.sudoUserName=

oracle.install.db.CLUSTER_NODES=sgpbnk01kf,sgpbnk02kf

oracle.install.db.config.starterdb.type=

oracle.install.db.config.starterdb.globalDBName=

oracle.install.db.config.starterdb.SID=

oracle.install.db.ConfigureAsContainerDB=

oracle.install.db.config.PDBName=

oracle.install.db.config.starterdb.characterSet=

oracle.install.db.config.starterdb.memoryOption=

oracle.install.db.config.starterdb.memoryLimit=

oracle.install.db.config.starterdb.installExampleSchemas=

oracle.install.db.config.starterdb.password.ALL=

oracle.install.db.config.starterdb.password.SYS=

oracle.install.db.config.starterdb.password.SYSTEM=

oracle.install.db.config.starterdb.password.DBSNMP=

oracle.install.db.config.starterdb.password.PDBADMIN=

oracle.install.db.config.starterdb.managementOption=

oracle.install.db.config.starterdb.omsHost=

oracle.install.db.config.starterdb.omsPort=

oracle.install.db.config.starterdb.emAdminUser=

oracle.install.db.config.starterdb.emAdminPassword=

oracle.install.db.config.starterdb.enableRecovery=

oracle.install.db.config.starterdb.storageType=

oracle.install.db.config.starterdb.fileSystemStorage.dataLocation=

oracle.install.db.config.starterdb.fileSystemStorage.recoveryLocation=

oracle.install.db.config.asm.diskGroup=

oracle.install.db.config.asm.ASMSNMPPassword=

[oracle@ora19node1 dbhome]<20210125 15:52:14>$ ./runInstaller -silent -ignorePrereqFailure -waitForCompletion -responseFile /home/oracle/dbinstall.rsp

Launching Oracle Database Setup Wizard...

Enter password for 'root' user:

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.

CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /data/gridInventory/logs/InstallActions2021-01-25_03-52-17PM/installActions2021-01-25_03-52-17PM.log

ACTION: Identify the list of failed prerequisite checks from the log: /data/gridInventory/logs/InstallActions2021-01-25_03-52-17PM/installActions2021-01-25_03-52-17PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

The response file for this session can be found at:

/data/dbhome/install/response/db_2021-01-25_03-52-17PM.rsp

You can find the log of this install session at:

/data/gridInventory/logs/InstallActions2021-01-25_03-52-17PM/installActions2021-01-25_03-52-17PM.log

RAC建库

oracle用户检查

dbca -silent -executePrereqs -databaseConfigType RAC -nodelist ora19node1,ora19node2

检查并创建磁盘组

select GROUP_NUMBER,name,path from v$asm_disk;
select NAME,COMPATIBILITY,DATABASE_COMPATIBILITY from v$asm_diskgroup;
ALTER DISKGROUP OCRDG SET ATTRIBUTE 'compatible.asm' = '19.0';
ALTER DISKGROUP OCRDG SET ATTRIBUTE 'compatible.rdbms' = '19.0';


CREATE DISKGROUP FRADG EXTERNAL redundancy DISK '/dev/asm-diskd' ATTRIBUTE 'compatible.asm' = '19.0', 'compatible.rdbms'='19.0','compatible.advm' = '19.0';
CREATE DISKGROUP DATADG EXTERNAL redundancy DISK '/dev/asm-diske' ATTRIBUTE 'compatible.asm' = '19.0', 'compatible.rdbms'='19.0','compatible.advm' = '19.0';
alter diskgroup FRADG mount;
alter diskgroup DATADG mount;

创建RAC响应文件

[oracle@sgpbnk01kf ~]<20200902 15:15:30>$ cat dbca_rac.rsp

responseFileVersion=/oracle/assistants/rspfmt_dbca_response_schema_v19.0.0

gdbName=ORA19RAC

sid=ORA19RAC

databaseConfigType=RAC

RACOneNodeServiceName=

policyManaged=

createServerPool=

serverPoolName=

cardinality=

force=

pqPoolName=

pqCardinality=

createAsContainerDatabase=TRUE

numberOfPDBs=0

pdbName=

useLocalUndoForPDBs=TRUE

pdbAdminPassword=zzhzj321zg*A

nodelist=sgpbnk01kf,sgpbnk02kf

templateName=General_Purpose.dbc

sysPassword=zzhzj321zg*A

systemPassword=zzhzj321zg*A

oracleHomeUserPassword=zzhzj321zg*A

emConfiguration=

emExpressPort=5500

runCVUChecks=

dbsnmpPassword=

omsHost=

omsPort=

emUser=

emPassword=

dvConfiguration=

dvUserName=

dvUserPassword=

dvAccountManagerName=

dvAccountManagerPassword=

olsConfiguration=

datafileJarLocation=

datafileDestination=+DATADG

recoveryAreaDestination=+FRADG

storageType=ASM

diskGroupName=DATADG

asmsnmpPassword=zzhzj321zg*A

recoveryGroupName=FRADG

characterSet=

nationalCharacterSet=

registerWithDirService=

dirServiceUserName=

dirServicePassword=

walletPassword=

listeners=

variablesFile=

variables=

initParams=

sampleSchema=FALSE

memoryPercentage=

databaseType=

automaticMemoryManagement=

totalMemory=

静默建库

dbca -silent -ignorePrereqFailure -createDatabase -responseFile /home/oracle/dbca.rsp

[oracle@ora19node1 ~]<20210125 17:16:13>$ dbca -silent -ignorePrereqFailure -createDatabase -responseFile /home/oracle/dbca.rsp

Prepare for db operation

8% complete

Copying database files

33% complete

Creating and starting Oracle instance

34% complete

35% complete

39% complete

44% complete

47% complete

50% complete

Creating cluster database views

52% complete

67% complete

Completing Database Creation

71% complete

73% complete

75% complete

Executing Post Configuration Actions

100% complete

Database creation complete. For details check the logfiles at:

/data/dbbase/cfgtoollogs/dbca/ORA19RAC.

Database Information:

Global Database Name:ORA19RAC

System Identifier(SID) Prefix:ORA19RAC

Look at the log file "/data/dbbase/cfgtoollogs/dbca/ORA19RAC/ORA19RAC.log" for further details.

总结

尽管按照上面的操作：GI---> DB--->DBCA建库一整套流程最终操作成功。但是笔者在此要提醒读者：

1. ORACLE RAC 涉及到知识点非常丰富，实验前建议阅读关于RAC 的架构和原理的文献；

2. 尽管看着笔者操作顺利，但中间其实有一些小小的错误，被笔者忽略了，请读者认真思索每一个步骤；

3. 由于笔者安装过太多次，文献总结不完善，本文中的相关目录的设计可能有些错误，请读者自行规划、修正；总体磁盘规划为：

GI ： gridbase、gridhome、gridInventory

DB ： dbbase、dbhome、dbInventory

4. 附图几张：

GI安装

root脚本执行

查看磁盘组：