RAC添加节点
概述
添加节点主要分成4个步骤走：
1, 准备好主机环境
2, GI软件的扩展。（在扩展之前可以做一个安装条件是否满足的检查）
3, Database软件的扩展
4, Instance的扩展
在第2,3,4中用到的命令也很简单，单有一个前提是第一步准备的linux是完全符合要求的
| 检查db3是否满足rac安装条件 su - grid cluvfy stage -pre nodeadd -n db3 -fixup -verbose cluvfy stage -post hwos -n db3
开始扩展集群软件 su - grid cd /u01/app/11.2.0/grid/oui/bin ./addNode.sh -silent "CLUSTER_NEW_NODES={db3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={db3-vip}" 运行提示的root.sh脚本 /u01/grid/oraInventory/orainstRoot.sh #On nodes db3 /u01/grid/crs/root.sh #On nodes db3 验证集群软件扩展成功 cluvfy stage -post nodeadd -n db3 -verbose
为新节点安装数据库软件 su - oracle cd /u01/app/oracle/product/11.2.0/db/oui/bin ./addNode.sh -silent "CLUSTER_NEW_NODES={db3}" 运行提示的root.sh脚本 /u01/oracle/db/root.sh #On nodes db3
添加实例 dbca 或用命令行直接添加 dbca -silent -addInstance -nodeList db3 -gdbName db -instanceName db3 -sysDBAUserName sys -sysDBAPassword "oracle"
grid用户检查状态 crsctl status resource -t |
接下来演示具体的过程
准备Linux环境
首先克隆出一台干净的虚拟机
给虚拟机配置网络
| 192.168.1.161 db1.up.com db1 192.168.1.162 db2.up.com db2 192.168.1.173 db3.up.com db3
10.0.1.161 db1-priv.up.com db1-priv 10.0.1.162 db2-priv.up.com db2-priv 10.0.1.173 db3-priv.up.com db3-priv
192.168.1.163 db1-vip.up.com db1-vip 192.168.1.164 db2-vip.up.com db2-vip 192.168.1.174 db32-vip.up.com db3-vip
192.168.1.165 db-cluster |
按照这种网络规划给db3节点的eth0, eth1分别配置网络
192.168.1.173
10.0.1.173
设置主机名
| [root@odd ~]# cat /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=yes HOSTNAME=db3.up.com [root@odd ~]# hostname db3.up.com |
修改linux版本
| [root@db3 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 4.8 (Tikanga) |
修改hosts文件配置
| [root@db3 ~]# cat /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 odd.up.com odd localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6
192.168.1.161 db1.up.com db1 192.168.1.162 db2.up.com db2 192.168.1.173 db3.up.com db3
10.0.1.161 db1-priv.up.com db1-priv 10.0.1.162 db2-priv.up.com db2-priv 10.0.1.173 db3-priv.up.com db3-priv
192.168.1.163 db1-vip.up.com db1-vip 192.168.1.164 db2-vip.up.com db2-vip 192.168.1.174 db3-vip.up.com db3-vip
192.168.1.165 db-cluster |
配置yum源
| [root@db3 ~]# mkdir /iso [root@db3 ~]# mount -t iso9660 -o loop /mnt/share/LinuxSoftware/OracleLinux-R5-U8-Server-x86_64-dvd.iso /iso [root@db3 ~]# vim /etc/yum.repos.d/oel.repo [root@db3 ~]# cat /etc/yum.repos.d/oel.repo [source] name=Oracle Enterprise Linux $releasever - Source baseurl=file:///iso/Server enabled=1 gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release |
因为在装rac的时候，系统提示一些包的缺失，需要安装，因此，在这里我们先安装上这些包
[root@db3 ~]# yum install libaio-devel sysstat unixODBC unixODBC-devel
移走ntp配置
[root@db3 ~]# mv /etc/ntp.conf /etc/ntp.conf.bak
添加用户
| groupadd -g 1000 oinstall groupadd -g 1001 asmadmin groupadd -g 1002 dba groupadd -g 1003 oper groupadd -g 1004 asmdba groupadd -g 1005 asmoper useradd -u 1000 -g oinstall -G dba,oper,asmdba oracle useradd -u 1001 -g oinstall -G asmadmin,asmdba,asmoper grid passwd oracle passwd grid |
添加文件夹
| mkdir -p /u01/app/11.2.0/grid mkdir -p /u01/app/grid mkdir -p /u01/app/oracle chown -R grid:oinstall /u01/ chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01/ |
配置udev设备
| 这里就直接将节点1的文件拷贝过来就可以了 [root@db3 ~]# cat /etc/udev/rules.d/99-oracelasm.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB0e123a72-41df0e11_", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB840e50f3-375627c3_", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBb8d6b677-65c5fa86_", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBfc5cfb52-b0c7ce00_", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
[root@db3 ~]# ll /dev/asm* ls: /dev/asm*: No such file or directory [root@db3 ~]# start_udev Starting udev: [ OK ] [root@db3 ~]# ll /dev/asm* brw-rw---- 1 grid asmadmin 8, 16 Mar 6 15:15 /dev/asm-diskb brw-rw---- 1 grid asmadmin 8, 32 Mar 6 15:15 /dev/asm-diskc brw-rw---- 1 grid asmadmin 8, 48 Mar 6 15:15 /dev/asm-diskd brw-rw---- 1 grid asmadmin 8, 64 Mar 6 15:15 /dev/asm-dis |
将start_udev添加到rc.local里
| [root@db3 ~]# cat /etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff.
touch /var/lock/subsys/local mount -t vboxsf Share /mnt/share start_ude |
编辑内核文件/etc/sysctl.conf (两个节点执行)
| kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 1048576 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 fs.aio-max-nr = 1048576 |
添加以上内容，然后执行sysctl -p命令加载这些内核参数
修改资源限制(两个节点执行)
/etc/security/limits.conf 添加如下内容，添加如下内容
| grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 |
编辑grid用户的 .bash_profile文件
| export ORACLE_SID=+ASM3 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/app/11.2.0/grid export PATH=$ORACLE_HOME/bin:$PATH |
编辑oracle用户的.bash_profile文件
| export ORACLE_SID=db3 export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db export PATH=$ORACLE_HOME/bin:$PAT |
还有一个很重要的工作，就是手工配置节点2，与1,2，的互通，这个很重要。
| 在节点3上操作 先切到grid用户下。 mkdir .ssh chmod 755 .ssh ssh-keygen -t rsa 这样才/home/grid/.ssh/目录下就有一个id_rsa.pub这个文件。将里面的一行内容拷贝到节点1的/home/grid/.ssh/authorized_keys这个文件里，然后将这个文件分别拷到节点2,3上。 [grid@db1 .ssh]$ scp authorized_keys db3:/home/grid/.ssh [grid@db1 .ssh]$ scp authorized_keys db2:/home/grid/.ssh
这样在每个节点的grid用户下执行 ssh db1 date ssh db2 date ssh db3 date ssh db1-priv date ssh db2-priv date ssh db3-priv date
[grid@db3 ~]$ ssh db1 date Thu Mar 6 16:29:51 CST 2014 [grid@db3 ~]$ ssh db2 date Thu Mar 6 16:29:51 CST 2014 [grid@db3 ~]$ ssh db3 date Thu Mar 6 16:29:51 CST 2014 [grid@db3 ~]$ ssh db1-priv date Thu Mar 6 16:29:51 CST 2014 [grid@db3 ~]$ ssh db2-priv date Thu Mar 6 16:29:52 CST 2014 [grid@db3 ~]$ ssh db3-priv date Thu Mar 6 16:29:52 CST 2014
如果是第一次执行会让你输入密码，直到执行这6个命令都不需要输入密码，则表示grid用户的ssh互通配置完毕。
用同样的方法配置oracle的互通 |
至此准备工作基本做完了。
GI扩展
在操作之前，确保所有节点都是正常状态。
| [grid@db1 ~]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE db1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE db1 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE db1 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE db1 ora.db.db ora....se.type 0/2 0/1 OFFLINE OFFLINE ora....SM1.asm application 0/5 0/0 ONLINE ONLINE db1 ora....B1.lsnr application 0/5 0/0 ONLINE ONLINE db1 ora.db1.gsd application 0/5 0/0 OFFLINE OFFLINE ora.db1.ons application 0/3 0/0 ONLINE ONLINE db1 ora.db1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE db1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE db2 ora....B2.lsnr application 0/5 0/0 ONLINE ONLINE db2 ora.db2.gsd application 0/5 0/0 OFFLINE OFFLINE ora.db2.ons application 0/3 0/0 ONLINE ONLINE db2 ora.db2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE db2 ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE db1 ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE ora....network ora....rk.type 0/5 0/ ONLINE ONLINE db1 ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE db1 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE db1 |
准备工作完成后，就可以根据新环境将新节点增加到RAC系统了，RAC增加节点总共分为三步进行，第一步就是将clusterware软件系统进行扩展，将所有11g的clusterware软件添加到新节点，第二步就是将ORACLE数据库的软件扩展到新节点，最后一步就是使用dbca创建新节点的数据库实例instance。
先去节点3 ，安装cvuqdisk包
| [root@db3 ~]# rpm -ivh /mnt/share/oracle11g/64/linux.x64_11gR2_grid/grid/rpm/cvuqdisk-1.0.7-1.rpm |
检查节点3是否满足需求
| [root@db1 ~]# su - grid [grid@db1 ~]$ cluvfy stage -pre nodeadd -n db3 -fixup -verbose |
| [grid@db1 ~]$ cluvfy stage -pre nodeadd -n db3 -fixup -verbose
Performing pre-checks for node addition
Checking node reachability...
Check: Node reachability from node "db1" Destination Node Reachable? ------------------------------------ ------------------------ db3 yes Result: Node reachability check passed from node "db1"
Checking user equivalence...
Check: User equivalence for user "grid" Node Name Comment ------------------------------------ ------------------------ db3 passed Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file... Node Name Status Comment ------------ ------------------------ ------------------------ db3 passed db2 passed db1 passed
Verification of the hosts config file successful
Interface information for node "db1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.161 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth0 192.168.1.163 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth1 10.0.1.161 10.0.1.0 0.0.0.0 10.0.1.1 08:00:27:25:BF:57 1500
Interface information for node "db2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.162 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.165 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.164 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth1 10.0.1.162 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:9F:B3:8C 1500
Interface information for node "db3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.173 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth1 10.0.1.173 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:FA:39:E3 1500
Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:eth0 db1:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db3:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db3:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db3:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db3:eth0 yes db2:eth0 db3:eth0 yes Result: Node connectivity passed for interface "eth0"
Result: Node connectivity check passed
Checking CRS integrity... The Oracle clusterware is healthy on node "db1" The Oracle clusterware is healthy on node "db2"
CRS integrity check passed
Checking shared resources...
Checking CRS home location... The location "/u01/app/11.2.0/grid" is not shared but is present/creatable on all nodes
Result: Shared resources check for node addition passed
Checking node connectivity...
Checking hosts config file... Node Name Status Comment ------------ ------------------------ ------------------------ db3 passed db2 passed db1 passed
Verification of the hosts config file successful
Interface information for node "db1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.161 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth0 192.168.1.163 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth1 10.0.1.161 10.0.1.0 0.0.0.0 10.0.1.1 08:00:27:25:BF:57 1500
Interface information for node "db2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.162 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.165 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.164 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth1 10.0.1.162 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:9F:B3:8C 1500
Interface information for node "db3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.173 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth1 10.0.1.173 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:FA:39:E3 1500
Check: Node connectivity of subnet "192.168.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:eth0 db1:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db3:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db2:eth0 yes db1:eth0 db3:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db3:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db3:eth0 yes db2:eth0 db3:eth0 yes Result: Node connectivity passed for subnet "192.168.1.0" with node(s) db1,db2,db3
Check: TCP connectivity of subnet "192.168.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:192.168.1.161 db1:192.168.1.163 passed db1:192.168.1.161 db2:192.168.1.162 passed db1:192.168.1.161 db2:192.168.1.165 passed db1:192.168.1.161 db2:192.168.1.164 passed db1:192.168.1.161 db3:192.168.1.173 passed Result: TCP connectivity check passed for subnet "192.168.1.0"
Check: Node connectivity of subnet "10.0.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:eth1 db2:eth1 yes db1:eth1 db3:eth1 yes db2:eth1 db3:eth1 yes Result: Node connectivity passed for subnet "10.0.1.0" with node(s) db1,db2,db3
Check: TCP connectivity of subnet "10.0.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:10.0.1.161 db2:10.0.1.162 passed db1:10.0.1.161 db3:10.0.1.173 passed Result: TCP connectivity check passed for subnet "10.0.1.0"
WARNING: Could not find a suitable set of interfaces for VIPs
WARNING: Could not find a suitable set of interfaces for the private interconnect
Result: Node connectivity check passed
Check: Total memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 1.96GB (2050664.0KB) 1.5GB (1572864.0KB) passed Result: Total memory check passed
Check: Available memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 1.57GB (1651152.0KB) 50MB (51200.0KB) passed Result: Available memory check passed
Check: Swap space Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 4.88GB (5116692.0KB) 2.93GB (3075996.0KB) passed Result: Swap space check passed
Check: Free disk space for "db3:/tmp" Path Node Name Mount point Available Required Comment ---------------- ------------ ------------ ------------ ------------ ------------ /tmp db3 / 87.93GB 1GB passed Result: Free disk space check passed for "db3:/tmp"
Check: User existence for "grid" Node Name Status Comment ------------ ------------------------ ------------------------ db3 exists passed Result: User existence check passed for "grid"
Check: Run level Node Name run level Required Comment ------------ ------------------------ ------------------------ ---------- db3 5 3,5 passed Result: Run level check passed
Check: Hard limits for "maximum open file descriptors" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- db3 hard 65536 65536 passed Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- db3 soft 1024 1024 passed Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- db3 hard 16384 16384 passed Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- db3 soft 2047 2047 passed Result: Soft limits check passed for "maximum user processes"
Check: System architecture Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 x86_64 x86_64 passed Result: System architecture check passed
Check: Kernel version Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 2.6.32-300.10.1.el5uek 2.6.18 passed Result: Kernel version check passed
Check: Kernel parameter for "semmsl" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 250 250 passed Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 32000 32000 passed Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 32 100 failed Result: Kernel parameter check failed for "semopm"
Check: Kernel parameter for "semmni" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 128 128 passed Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 68719476736 536870912 passed Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 4096 4096 passed Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 4294967296 2097152 passed Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 199621 6815744 failed Result: Kernel parameter check failed for "file-max"
Check: Kernel parameter for "ip_local_port_range" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 between 32768 & 61000 between 9000 & 65500 failed (ignorable) Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 126976 262144 failed Result: Kernel parameter check failed for "rmem_default"
Check: Kernel parameter for "rmem_max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 131071 4194304 failed Result: Kernel parameter check failed for "rmem_max"
Check: Kernel parameter for "wmem_default" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 126976 262144 failed Result: Kernel parameter check failed for "wmem_default"
Check: Kernel parameter for "wmem_max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 131071 1048576 failed Result: Kernel parameter check failed for "wmem_max"
Check: Kernel parameter for "aio-max-nr" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- db3 65536 1048576 failed Result: Kernel parameter check failed for "aio-max-nr"
Check: Package existence for "make-3.81" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 make-3.81-3.el5 make-3.81 passed Result: Package existence check passed for "make-3.81"
Check: Package existence for "binutils-2.17.50.0.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 binutils-2.17.50.0.6-20.el5 binutils-2.17.50.0.6 passed Result: Package existence check passed for "binutils-2.17.50.0.6"
Check: Package existence for "gcc-4.1.2" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 gcc-4.1.2-52.el5 gcc-4.1.2 passed Result: Package existence check passed for "gcc-4.1.2"
Check: Package existence for "libaio-0.3.106 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libaio-0.3.106-5 (i386) libaio-0.3.106 (i386) passed Result: Package existence check passed for "libaio-0.3.106 (i386)"
Check: Package existence for "libaio-0.3.106 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libaio-0.3.106-5 (x86_64) libaio-0.3.106 (x86_64) passed Result: Package existence check passed for "libaio-0.3.106 (x86_64)"
Check: Package existence for "glibc-2.5-24 (i686)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-2.5-81 (i686) glibc-2.5-24 (i686) passed Result: Package existence check passed for "glibc-2.5-24 (i686)"
Check: Package existence for "glibc-2.5-24 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-2.5-81 (x86_64) glibc-2.5-24 (x86_64) passed Result: Package existence check passed for "glibc-2.5-24 (x86_64)"
Check: Package existence for "compat-libstdc++-33-3.2.3 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
Check: Package existence for "compat-libstdc++-33-3.2.3 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
Check: Package existence for "elfutils-libelf-0.125 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.125 (x86_64) passed Result: Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
Check: Package existence for "elfutils-libelf-devel-0.125" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed Result: Package existence check passed for "elfutils-libelf-devel-0.125"
Check: Package existence for "glibc-common-2.5" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-common-2.5-81 glibc-common-2.5 passed Result: Package existence check passed for "glibc-common-2.5"
Check: Package existence for "glibc-devel-2.5 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-devel-2.5-81 (i386) glibc-devel-2.5 (i386) passed Result: Package existence check passed for "glibc-devel-2.5 (i386)"
Check: Package existence for "glibc-devel-2.5 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-devel-2.5-81 (x86_64) glibc-devel-2.5 (x86_64) passed Result: Package existence check passed for "glibc-devel-2.5 (x86_64)"
Check: Package existence for "glibc-headers-2.5" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 glibc-headers-2.5-81 glibc-headers-2.5 passed Result: Package existence check passed for "glibc-headers-2.5"
Check: Package existence for "gcc-c++-4.1.2" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 gcc-c++-4.1.2-52.el5 gcc-c++-4.1.2 passed Result: Package existence check passed for "gcc-c++-4.1.2"
Check: Package existence for "libaio-devel-0.3.106 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.106 (i386) passed Result: Package existence check passed for "libaio-devel-0.3.106 (i386)"
Check: Package existence for "libaio-devel-0.3.106 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.106 (x86_64) passed Result: Package existence check passed for "libaio-devel-0.3.106 (x86_64)"
Check: Package existence for "libgcc-4.1.2 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libgcc-4.1.2-52.el5 (i386) libgcc-4.1.2 (i386) passed Result: Package existence check passed for "libgcc-4.1.2 (i386)"
Check: Package existence for "libgcc-4.1.2 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libgcc-4.1.2-52.el5 (x86_64) libgcc-4.1.2 (x86_64) passed Result: Package existence check passed for "libgcc-4.1.2 (x86_64)"
Check: Package existence for "libstdc++-4.1.2 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libstdc++-4.1.2-52.el5 (i386) libstdc++-4.1.2 (i386) passed Result: Package existence check passed for "libstdc++-4.1.2 (i386)"
Check: Package existence for "libstdc++-4.1.2 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libstdc++-4.1.2-52.el5 (x86_64) libstdc++-4.1.2 (x86_64) passed Result: Package existence check passed for "libstdc++-4.1.2 (x86_64)"
Check: Package existence for "libstdc++-devel-4.1.2 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 libstdc++-devel-4.1.2-52.el5 (x86_64) libstdc++-devel-4.1.2 (x86_64) passed Result: Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
Check: Package existence for "sysstat-7.0.2" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 sysstat-7.0.2-11.el5 sysstat-7.0.2 passed Result: Package existence check passed for "sysstat-7.0.2"
Check: Package existence for "unixODBC-2.2.11 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 unixODBC-2.2.11-10.el5 (i386) unixODBC-2.2.11 (i386) passed Result: Package existence check passed for "unixODBC-2.2.11 (i386)"
Check: Package existence for "unixODBC-2.2.11 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 unixODBC-2.2.11-10.el5 (x86_64) unixODBC-2.2.11 (x86_64) passed Result: Package existence check passed for "unixODBC-2.2.11 (x86_64)"
Check: Package existence for "unixODBC-devel-2.2.11 (i386)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 unixODBC-devel-2.2.11-10.el5 (i386) unixODBC-devel-2.2.11 (i386) passed Result: Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
Check: Package existence for "unixODBC-devel-2.2.11 (x86_64)" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 unixODBC-devel-2.2.11-10.el5 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed Result: Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"
Check: Package existence for "ksh-20060214" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- db3 ksh-20100621-5.el5 ksh-20060214 passed Result: Package existence check passed for "ksh-20060214"
Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed
Checking to make sure user "grid" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ db3 does not exist passed Result: User "grid" is not part of "root" group. Check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started... Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Fixup information has been generated for following node(s): db3 Please run the following script on each node as "root" user to execute the fixups: '/tmp/CVU_11.2.0.1.0_grid/runfixup.sh'
Pre-check for node addition was unsuccessful on all the nodes. |
在这个结果中，遇到软件failed的，就用yum的方式给其装上，其他的问题，网络是通的，用户是对等的，其他的failed就基本可以忽略掉。
| [grid@db1 ~]$ cluvfy stage -post hwos -n db3
Performing post-checks for hardware and operating system setup
Checking node reachability... Node reachability check passed from node "db1"
Checking user equivalence... User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Node connectivity passed for subnet "192.168.1.0" with node(s) db3 TCP connectivity check passed for subnet "192.168.1.0"
Node connectivity passed for subnet "10.0.1.0" with node(s) db3 TCP connectivity check passed for subnet "10.0.1.0"
Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: db3 eth0:192.168.1.173
Interfaces found on subnet "10.0.1.0" that are likely candidates for a private interconnect are: db3 eth1:10.0.1.173
Node connectivity check passed
Check for multiple users with UID value 0 passed
Post-check for hardware and operating system setup was successful. |
这样在节点1上做的，对节点3的检查基本通过，可以开始添加操作了。
| [grid@db1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={db3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={db3-vip}"-bash: cd: /u01/app/11.2.0/grid/oui/bin/addNode.sh: Not a directory [grid@db1 ~]$ /u01/app/11.2.0/grid/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={db3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={db3-vip}" /u01/app/11.2.0/grid/oui/bin/addNode.sh: line 4: ./runInstaller: No such file or directory [grid@db1 ~]$ cat /u01/app/11.2.0/grid/oui/bin/addNode.sh #!/bin/sh OHOME=/u01/app/11.2.0/grid INVPTRLOC=$OHOME/oraInst.loc ./runInstaller -addNode -invPtrLoc $INVPTRLOC ORACLE_HOME=$OHOME $* [grid@db1 ~]$ cd /u01/app/11.2.0/grid/oui/bin/ [grid@db1 bin]$./addNode.sh -silent "CLUSTER_NEW_NODES={db3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={db3-vip}" Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4996 MB Passed Oracle Universal Installer, Version 11.2.0.1.0 Production Copyright (C) 1999, 2009, Oracle. All rights reserved.
Performing tests to see whether nodes db2,db3 are available ............................................................... 100% Done.
... ----------------------------------------------------------------------------- Cluster Node Addition Summary Global Settings Source: /u01/app/11.2.0/grid New Nodes Space Requirements New Nodes db3 /: Required 3.87GB : Available 81.88GB Installed Products Product Names Oracle Grid Infrastructure 11.2.0.1.0 Sun JDK 1.5.0.17.0 Installer SDK Component 11.2.0.1.0 Oracle One-Off Patch Installer 11.2.0.0.2 Oracle Universal Installer 11.2.0.1.0 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 Enterprise Manager Common Core Files 10.2.0.4.2 Oracle DBCA Deconfiguration 11.2.0.1.0 Oracle RAC Deconfiguration 11.2.0.1.0 Oracle Quality of Service Management (Server) 11.2.0.1.0 Installation Plugin Files 11.2.0.1.0 Universal Storage Manager Files 11.2.0.1.0 Oracle Text Required Support Files 11.2.0.1.0 Automatic Storage Management Assistant 11.2.0.1.0 Oracle Database 11g Multimedia Files 11.2.0.1.0 Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 Oracle Globalization Support 11.2.0.1.0 Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 Oracle Core Required Support Files 11.2.0.1.0 Bali Share 1.1.18.0.0 Oracle Database Deconfiguration 11.2.0.1.0 Oracle Quality of Service Management (Client) 11.2.0.1.0 Expat libraries 2.0.1.0.1 Oracle Containers for Java 11.2.0.1.0 Perl Modules 5.10.0.0.1 Secure Socket Layer 11.2.0.1.0 Oracle JDBC/OCI Instant Client 11.2.0.1.0 Oracle Multimedia Client Option 11.2.0.1.0 LDAP Required Support Files 11.2.0.1.0 Character Set Migration Utility 11.2.0.1.0 Perl Interpreter 5.10.0.0.1 PL/SQL Embedded Gateway 11.2.0.1.0 OLAP SQL Scripts 11.2.0.1.0 Database SQL Scripts 11.2.0.1.0 Oracle Extended Windowing Toolkit 3.4.47.0.0 SSL Required Support Files for InstantClient 11.2.0.1.0 SQL*Plus Files for Instant Client 11.2.0.1.0 Oracle Net Required Support Files 11.2.0.1.0 Oracle Database User Interface 2.2.13.0.0 RDBMS Required Support Files for Instant Client 11.2.0.1.0 Enterprise Manager Minimal Integration 11.2.0.1.0 XML Parser for Java 11.2.0.1.0 Oracle Security Developer Tools 11.2.0.1.0 Oracle Wallet Manager 11.2.0.1.0 Enterprise Manager plugin Common Files 11.2.0.1.0 Platform Required Support Files 11.2.0.1.0 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 RDBMS Required Support Files 11.2.0.1.0 Oracle Ice Browser 5.2.3.6.0 Oracle Help For Java 4.2.9.0.0 Enterprise Manager Common Files 10.2.0.4.2 Deinstallation Tool 11.2.0.1.0 Oracle Java Client 11.2.0.1.0 Cluster Verification Utility Files 11.2.0.1.0 Oracle Notification Service (eONS) 11.2.0.1.0 Oracle LDAP administration 11.2.0.1.0 Cluster Verification Utility Common Files 11.2.0.1.0 Oracle Clusterware RDBMS Files 11.2.0.1.0 Oracle Locale Builder 11.2.0.1.0 Oracle Globalization Support 11.2.0.1.0 Buildtools Common Files 11.2.0.1.0 Oracle RAC Required Support Files-HAS 11.2.0.1.0 SQL*Plus Required Support Files 11.2.0.1.0 XDK Required Support Files 11.2.0.1.0 Agent Required Support Files 10.2.0.4.2 Parser Generator Required Support Files 11.2.0.1.0 Precompiler Required Support Files 11.2.0.1.0 Installation Common Files 11.2.0.1.0 Required Support Files 11.2.0.1.0 Oracle JDBC/THIN Interfaces 11.2.0.1.0 Oracle Multimedia Locator 11.2.0.1.0 Oracle Multimedia 11.2.0.1.0 HAS Common Files 11.2.0.1.0 Assistant Common Files 11.2.0.1.0 PL/SQL 11.2.0.1.0 HAS Files for DB 11.2.0.1.0 Oracle Recovery Manager 11.2.0.1.0 Oracle Database Utilities 11.2.0.1.0 Oracle Notification Service 11.2.0.0.0 SQL*Plus 11.2.0.1.0 Oracle Netca Client 11.2.0.1.0 Oracle Net 11.2.0.1.0 Oracle JVM 11.2.0.1.0 Oracle Internet Directory Client 11.2.0.1.0 Oracle Net Listener 11.2.0.1.0 Cluster Ready Services Files 11.2.0.1.0 Oracle Database 11g 11.2.0.1.0
Instantiating scripts for add node (Thursday, March 6, 2014 4:49:56 PM CST) . 1% Done. Instantiation of add node scripts complete
Copying to remote nodes (Thursday, March 6, 2014 4:50:16 PM CST) ............................................................................................... 96% Done. Home copied to new nodes
Saving inventory on nodes (Thursday, March 6, 2014 4:55:59 PM CST) . 100% Done. Save inventory complete WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system. To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'db3'. If you do not register the inventory, you may not be able to update or patch the products you installed. The following configuration scripts need to be executed as the "root" user in each cluster node. /u01/app/oraInventory/orainstRoot.sh #On nodes db3 /u01/app/11.2.0/grid/root.sh #On nodes db3 To execute the configuration scripts: 1. Open a terminal window 2. Log in as "root" 3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/11.2.0/grid was successful. Please check '/tmp/silentInstall.log' for more details. |
接下来，就是使用root用户，在需要添加的节点上执行，也就是节点3.
| [root@db3 ~]# /u01/app/oraInventory/orainstRoot.sh Creating the Oracle inventory pointer file (/etc/oraInst.loc) Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@db3 ~]# /u01/app/11.2.0/grid/root.sh Running Oracle 11g root.sh script...
The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2014-03-06 17:05:21: Parsing the host name 2014-03-06 17:05:21: Checking for super user privileges 2014-03-06 17:05:21: User has super user privileges Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting acfsroot: ACFS-9301: ADVM/ACFS installation can not proceed:
acfsroot: ACFS-9302: No installation files found at /u01/app/11.2.0/grid/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.el5uek-x86_64/bin.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node db1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster CRS-2672: Attempting to start 'ora.mdnsd' on 'db3' CRS-2676: Start of 'ora.mdnsd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'db3' CRS-2676: Start of 'ora.gipcd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'db3' CRS-2676: Start of 'ora.gpnpd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'db3' CRS-2676: Start of 'ora.cssdmonitor' on 'db3' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'db3' CRS-2672: Attempting to start 'ora.diskmon' on 'db3' CRS-2676: Start of 'ora.diskmon' on 'db3' succeeded CRS-2676: Start of 'ora.cssd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'db3' CRS-2676: Start of 'ora.ctssd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.asm' on 'db3' CRS-2676: Start of 'ora.asm' on 'db3' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'db3' CRS-2676: Start of 'ora.crsd' on 'db3' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'db3' CRS-2676: Start of 'ora.evmd' on 'db3' succeeded clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful.
db3 2014/03/06 17:09:38 /u01/app/11.2.0/grid/cdata/db3/backup_20140306_170938.olr Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4996 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful. |
此时再节点1上可以看到节点3已经加到集群里了。
| [grid@db1 bin]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE db1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE db1 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE db2 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE db1 ora.db.db ora....se.type 0/2 0/1 OFFLINE OFFLINE ora....SM1.asm application 0/5 0/0 ONLINE ONLINE db1 ora....B1.lsnr application 0/5 0/0 ONLINE ONLINE db1 ora.db1.gsd application 0/5 0/0 OFFLINE OFFLINE ora.db1.ons application 0/3 0/0 ONLINE ONLINE db1 ora.db1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE db1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE db2 ora....B2.lsnr application 0/5 0/0 ONLINE ONLINE db2 ora.db2.gsd application 0/5 0/0 OFFLINE OFFLINE ora.db2.ons application 0/3 0/0 ONLINE ONLINE db2 ora.db2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE db2 ora....SM3.asm application 0/5 0/0 ONLINE ONLINE db3 ora....B3.lsnr application 0/5 0/0 ONLINE ONLINE db3 ora.db3.gsd application 0/5 0/0 OFFLINE OFFLINE ora.db3.ons application 0/3 0/0 ONLINE ONLINE db3 ora.db3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE db3 ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE db1 ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE ora....network ora....rk.type 0/5 0/ ONLINE ONLINE db1 ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE db1 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE db2 |
验证集群软件扩展成功
| [grid@db1 bin]$cluvfy stage -post nodeadd -n db3 -verbose
Performing post-checks for node addition
Checking node reachability...
Check: Node reachability from node "db1" Destination Node Reachable? ------------------------------------ ------------------------ db3 yes Result: Node reachability check passed from node "db1"
Checking user equivalence...
Check: User equivalence for user "grid" Node Name Comment ------------------------------------ ------------------------ db3 passed Result: User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file... Node Name Status Comment ------------ ------------------------ ------------------------ db3 passed db2 passed db1 passed
Verification of the hosts config file successful
Interface information for node "db3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.173 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth0 192.168.1.174 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth1 10.0.1.173 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:FA:39:E3 1500
Interface information for node "db2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.162 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.165 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.164 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth1 10.0.1.162 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:9F:B3:8C 1500
Interface information for node "db1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.161 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth0 192.168.1.163 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth1 10.0.1.161 10.0.1.0 0.0.0.0 10.0.1.1 08:00:27:25:BF:57 1500
Check: Node connectivity for interface "eth0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db3:eth0 db3:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db1:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db1:eth0 db1:eth0 yes Result: Node connectivity passed for interface "eth0"
Result: Node connectivity check passed
Checking cluster integrity...
Node Name ------------------------------------ db1 db2 db3
Cluster integrity check passed
Checking CRS integrity... The Oracle clusterware is healthy on node "db3" The Oracle clusterware is healthy on node "db2" The Oracle clusterware is healthy on node "db1"
CRS integrity check passed
Checking shared resources...
Checking CRS home location... The location "/u01/app/11.2.0/grid" is not shared but is present/creatable on all nodes
Result: Shared resources check for node addition passed
Checking node connectivity...
Checking hosts config file... Node Name Status Comment ------------ ------------------------ ------------------------ db3 passed db2 passed db1 passed
Verification of the hosts config file successful
Interface information for node "db3" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.173 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth0 192.168.1.174 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:CB:D9:A9 1500 eth1 10.0.1.173 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:FA:39:E3 1500
Interface information for node "db2" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.162 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.165 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth0 192.168.1.164 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:1D:22:8D 1500 eth1 10.0.1.162 10.0.1.0 0.0.0.0 192.168.1.1 08:00:27:9F:B3:8C 1500
Interface information for node "db1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.161 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth0 192.168.1.163 192.168.1.0 0.0.0.0 10.0.1.1 08:00:27:6C:37:49 1500 eth1 10.0.1.161 10.0.1.0 0.0.0.0 10.0.1.1 08:00:27:25:BF:57 1500
Check: Node connectivity of subnet "192.168.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db3:eth0 db3:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db2:eth0 yes db3:eth0 db1:eth0 yes db3:eth0 db1:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db2:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db2:eth0 db1:eth0 yes db1:eth0 db1:eth0 yes Result: Node connectivity passed for subnet "192.168.1.0" with node(s) db3,db2,db1
Check: TCP connectivity of subnet "192.168.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:192.168.1.161 db3:192.168.1.173 passed db1:192.168.1.161 db3:192.168.1.174 passed db1:192.168.1.161 db2:192.168.1.162 passed db1:192.168.1.161 db2:192.168.1.165 passed db1:192.168.1.161 db2:192.168.1.164 passed db1:192.168.1.161 db1:192.168.1.163 passed Result: TCP connectivity check passed for subnet "192.168.1.0"
Check: Node connectivity of subnet "10.0.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db3:eth1 db2:eth1 yes db3:eth1 db1:eth1 yes db2:eth1 db1:eth1 yes Result: Node connectivity passed for subnet "10.0.1.0" with node(s) db3,db2,db1
Check: TCP connectivity of subnet "10.0.1.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- db1:10.0.1.161 db3:10.0.1.173 passed db1:10.0.1.161 db2:10.0.1.162 passed Result: TCP connectivity check passed for subnet "10.0.1.0"
WARNING: Could not find a suitable set of interfaces for VIPs
WARNING: Could not find a suitable set of interfaces for the private interconnect
Result: Node connectivity check passed
Checking node application existence...
Checking existence of VIP node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- db3 yes online passed db2 yes online passed db1 yes online passed Result: Check passed.
Checking existence of ONS node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- db3 no online passed db2 no online passed db1 no online passed Result: Check passed.
Checking existence of GSD node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- db3 no does not exist ignored db2 no does not exist ignored db1 no does not exist ignored Result: Check ignored.
Checking existence of EONS node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- db3 no online passed db2 no online passed db1 no online passed Result: Check passed.
Checking existence of NETWORK node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- db3 no online passed db2 no online passed db1 no online passed Result: Check passed.
Checking Single Client Access Name (SCAN)... SCAN VIP name Node Running? ListenerName Port Running? ---------------- ------------ ------------ ------------ ------------ ------------ db-cluster db2 true LISTENER 1521 true
Checking name resolution setup for "db-cluster"...
ERROR: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "db-cluster" SCAN Name IP Address Status Comment ------------ ------------------------ ------------------------ ---------- db-cluster 192.168.1.165 failed NIS Entry
ERROR: PRVF-4657 : Name resolution setup check for "db-cluster" (IP address: 192.168.1.165) failed
ERROR: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "db-cluster"
Verification of SCAN VIP and Listener setup failed
Checking to make sure user "grid" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ db3 does not exist passed Result: User "grid" is not part of "root" group. Check passed
Checking if Clusterware is installed on all nodes... Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes... Check: CTSS Resource running on all nodes Node Name Status ------------------------------------ ------------------------ db3 passed Result: CTSS resource check passed
Querying CTSS for time offset on all nodes... Result: Query of CTSS for time offset passed
Check CTSS state started... Check: CTSS state Node Name State ------------------------------------ ------------------------ db3 Active CTSS is in Active state. Proceeding with check of clock time offsets on all nodes... Reference Time Offset Limit: 1000.0 msecs Check: Reference Time Offset Node Name Time Offset Status ------------ ------------------------ ------------------------ db3 0.0 passed
Time offset is within the specified limits on the following set of nodes: "[db3]" Result: Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed
Post-check for node addition was unsuccessful on all the nodes. |
验证提示unsuccessful，从返回结果来看
PRVF-4664 : Found inconsistent name resolution entries for SCAN name "db-cluster"
是这个导致的
由于并没有使用dns做轮询去解析这个db-cluster域名，所以这个错误，可以忽略。
扩展database软件了
| [root@db1 ~]# su - oracle [oracle@db1 ~]$cd /u01/app/oracle/product/11.2.0/db/oui/bin [oracle@db1 bin]$ pwd /u01/app/oracle/product/11.2.0/db/oui/bin [oracle@db1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={db3}" Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4992 MB Passed Oracle Universal Installer, Version 11.2.0.1.0 Production Copyright (C) 1999, 2009, Oracle. All rights reserved.
Performing tests to see whether nodes db2,db3 are available ............................................................... 100% Done.
...... ----------------------------------------------------------------------------- Cluster Node Addition Summary Global Settings Source: /u01/app/oracle/product/11.2.0/db New Nodes Space Requirements New Nodes db3 /: Required 4.00GB : Available 78.37GB Installed Products Product Names Oracle Database 11g 11.2.0.1.0 Sun JDK 1.5.0.17.0 Installer SDK Component 11.2.0.1.0 Oracle One-Off Patch Installer 11.2.0.0.2 Oracle Universal Installer 11.2.0.1.0 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 Oracle DBCA Deconfiguration 11.2.0.1.0 Oracle RAC Deconfiguration 11.2.0.1.0 Oracle Database Deconfiguration 11.2.0.1.0 Oracle Configuration Manager 10.3.1.1.0 Oracle ODBC Driverfor Instant Client 11.2.0.1.0 LDAP Required Support Files 11.2.0.1.0 SSL Required Support Files for InstantClient 11.2.0.1.0 Bali Share 1.1.18.0.0 Oracle Extended Windowing Toolkit 3.4.47.0.0 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 Oracle Real Application Testing 11.2.0.1.0 Oracle Database Vault J2EE Application 11.2.0.1.0 Oracle Label Security 11.2.0.1.0 Oracle Data Mining RDBMS Files 11.2.0.1.0 Oracle OLAP RDBMS Files 11.2.0.1.0 Oracle OLAP API 11.2.0.1.0 Platform Required Support Files 11.2.0.1.0 Oracle Database Vault option 11.2.0.1.0 Oracle RAC Required Support Files-HAS 11.2.0.1.0 SQL*Plus Required Support Files 11.2.0.1.0 Oracle Display Fonts 9.0.2.0.0 Oracle Ice Browser 5.2.3.6.0 Oracle JDBC Server Support Package 11.2.0.1.0 Oracle SQL Developer 11.2.0.1.0 Oracle Application Express 11.2.0.1.0 XDK Required Support Files 11.2.0.1.0 RDBMS Required Support Files for Instant Client 11.2.0.1.0 SQLJ Runtime 11.2.0.1.0 Database Workspace Manager 11.2.0.1.0 RDBMS Required Support Files 11.2.0.1.0 Oracle Globalization Support 11.2.0.1.0 Exadata Storage Server 11.2.0.1.0 Provisioning Advisor Framework 10.2.0.4.2 Enterprise Manager Database Plugin -- Repository Support 11.2.0.1.0 Enterprise Manager Repository Core Files 10.2.0.4.2 Enterprise Manager Database Plugin -- Agent Support 11.2.0.1.0 Enterprise Manager Grid Control Core Files 10.2.0.4.2 Enterprise Manager Common Core Files 10.2.0.4.2 Enterprise Manager Agent Core Files 10.2.0.4.2 Agent Required Support Files 10.2.0.4.2 regexp 2.1.9.0.0 Parser Generator Required Support Files 11.2.0.1.0 Oracle 11g Warehouse Builder Required Files 11.2.0.1.0 Oracle Notification Service (eONS) 11.2.0.1.0 Oracle Text Required Support Files 11.2.0.1.0 Precompiler Required Support Files 11.2.0.1.0 Oracle Database 11g Multimedia Files 11.2.0.1.0 Oracle Multimedia Java Advanced Imaging 11.2.0.1.0 Oracle Multimedia Annotator 11.2.0.1.0 Oracle JDBC/OCI Instant Client 11.2.0.1.0 Oracle Multimedia Locator RDBMS Files 11.2.0.1.0 Oracle Core Required Support Files 11.2.0.1.0 Oracle Help For Java 4.2.9.0.0 Sample Schema Data 11.2.0.1.0 Oracle Starter Database 11.2.0.1.0 Oracle Message Gateway Common Files 11.2.0.1.0 Oracle XML Query 11.2.0.1.0 XML Parser for Oracle JVM 11.2.0.1.0 Expat libraries 2.0.1.0.1 Installation Plugin Files 11.2.0.1.0 Enterprise Manager Common Files 10.2.0.4.2 Perl Modules 5.10.0.0.1 Deinstallation Tool 11.2.0.1.0 Oracle Quality of Service Management (Client) 11.2.0.1.0 Perl Interpreter 5.10.0.0.1 JAccelerator (COMPANION) 11.2.0.1.0 Oracle Containers for Java 11.2.0.1.0 Oracle Code Editor 1.2.1.0.0I Oracle Net Required Support Files 11.2.0.1.0 Secure Socket Layer 11.2.0.1.0 Oracle Universal Connection Pool 11.2.0.1.0 Oracle JDBC/THIN Interfaces 11.2.0.1.0 Oracle Multimedia Client Option 11.2.0.1.0 Oracle Java Client 11.2.0.1.0 Character Set Migration Utility 11.2.0.1.0 Oracle Locale Builder 11.2.0.1.0 PL/SQL Embedded Gateway 11.2.0.1.0 OLAP SQL Scripts 11.2.0.1.0 Database SQL Scripts 11.2.0.1.0 Oracle Globalization Support 11.2.0.1.0 Required Support Files 11.2.0.1.0 SQL*Plus Files for Instant Client 11.2.0.1.0 Oracle ODBC Driver 11.2.0.1.0 Oracle Database User Interface 2.2.13.0.0 Oracle Notification Service 11.2.0.0.0 Enterprise Manager Minimal Integration 11.2.0.1.0 XML Parser for Java 11.2.0.1.0 Oracle Security Developer Tools 11.2.0.1.0 Oracle Wallet Manager 11.2.0.1.0 Cluster Verification Utility Common Files 11.2.0.1.0 Oracle Clusterware RDBMS Files 11.2.0.1.0 Oracle UIX 2.2.24.5.0 Enterprise Manager plugin Common Files 11.2.0.1.0 HAS Common Files 11.2.0.1.0 Precompiler Common Files 11.2.0.1.0 Installation Common Files 11.2.0.1.0 Oracle Help for the Web 2.0.14.0.0 Oracle LDAP administration 11.2.0.1.0 Buildtools Common Files 11.2.0.1.0 Assistant Common Files 11.2.0.1.0 Oracle Recovery Manager 11.2.0.1.0 PL/SQL 11.2.0.1.0 Generic Connectivity Common Files 11.2.0.1.0 Oracle Database Gateway for ODBC 11.2.0.1.0 Oracle Programmer 11.2.0.1.0 Oracle Database Utilities 11.2.0.1.0 Enterprise Manager Agent 10.2.0.4.2 Oracle Netca Client 11.2.0.1.0 SQL*Plus 11.2.0.1.0 Oracle Call Interface (OCI) 11.2.0.1.0 Oracle Multimedia Locator 11.2.0.1.0 Oracle Multimedia 11.2.0.1.0 Oracle Net 11.2.0.1.0 Database Configuration and Upgrade Assistants 11.2.0.1.0 Oracle XML Development Kit 11.2.0.1.0 Oracle JVM 11.2.0.1.0 Oracle Advanced Security 11.2.0.1.0 Oracle Internet Directory Client 11.2.0.1.0 HAS Files for DB 11.2.0.1.0 Oracle Enterprise Manager Console DB 11.2.0.1.0 Oracle Net Listener 11.2.0.1.0 Oracle Text 11.2.0.1.0 Oracle Net Services 11.2.0.1.0 Oracle Database 11g 11.2.0.1.0 Oracle OLAP 11.2.0.1.0 Oracle Spatial 11.2.0.1.0 Oracle Partitioning 11.2.0.1.0 Enterprise Edition Options 11.2.0.1.0 -----------------------------------------------------------------------------
Instantiating scripts for add node (Thursday, March 6, 2014 5:20:33 PM CST) . 1% Done. Instantiation of add node scripts complete
Copying to remote nodes (Thursday, March 6, 2014 5:20:57 PM CST) ............................................................................................... 96% Done. Home copied to new nodes
Saving inventory on nodes (Thursday, March 6, 2014 5:35:03 PM CST) . 100% Done. Save inventory complete WARNING: The following configuration scripts need to be executed as the "root" user in each cluster node. /u01/app/oracle/product/11.2.0/db/root.sh #On nodes db3 To execute the configuration scripts: 1. Open a terminal window 2. Log in as "root" 3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/db was successful. Please check '/tmp/silentInstall.log' for more details. |
现在就需要在节点3上执行这个脚本
| [root@db3 ~]# /u01/app/oracle/product/11.2.0/db/root.sh Running Oracle 11g root.sh script...
The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/11.2.0/db
Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions. [root@db3 ~]# |
至此，grid，database软件都已经装完了，接下来添加，在节点1上执行
添加实例
[oracle@db1 ~]$ dbca
或用命令行直接添加
[oracle@db1 ~]$ dbca -silent -addInstance -nodeList db3 -gdbName db -instanceName db3 -sysDBAUserName sys -sysDBAPassword "oracle"
这个地方一定要注意原来的实例是running的
| grid@db2 ~]$ crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE db1 ONLINE ONLINE db2 ONLINE ONLINE db3 ora.LISTENER.lsnr ONLINE ONLINE db1 ONLINE ONLINE db2 ONLINE ONLINE db3 ora.asm ONLINE ONLINE db1 Started ONLINE ONLINE db2 Started ONLINE ONLINE db3 Started ora.eons ONLINE ONLINE db1 ONLINE ONLINE db2 ONLINE ONLINE db3 ora.gsd OFFLINE OFFLINE db1 OFFLINE OFFLINE db2 OFFLINE OFFLINE db3 ora.net1.network ONLINE ONLINE db1 ONLINE ONLINE db2 ONLINE ONLINE db3 ora.ons ONLINE ONLINE db1 ONLINE ONLINE db2 ONLINE ONLINE db3 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE db2 ora.db.db 1 ONLINE ONLINE db1 Open 2 ONLINE ONLINE db2 Open 3 ONLINE ONLINE db3 Open ora.db1.vip 1 ONLINE ONLINE db1 ora.db2.vip 1 ONLINE ONLINE db2 ora.db3.vip 1 ONLINE ONLINE db3 ora.oc4j 1 OFFLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE db2 |