版权声明：本文为博主原创文章，转载请标明出处。 https://blog.csdn.net/ctypyb2002/article/details/86627839
安装方式为两节点 rac ,使用 udev 固定硬盘,没有使用 oracle asmlib.
ip 规划
私有网络用了两张网卡,做haip
nodea nodeb
pub 192.168.56.221 192.168.56.222
priv 192.168.156.221 192.168.156.222
192.168.156.231 192.168.156.232
vip 192.168.56.231 192.168.56.232
scan 192.168.56.241
公有网络 192.168.56.x
私有网络 192.168.156.x
多个私有网卡是为了使用到 oracle 的 haip,增加私有网络带宽.
scan vip 只有如下实现方式:
- 使用 /etc/hosts 来完成,只能配置成1个ip.
- 使用 dns 来完成,需要3个ip.
添加共享硬盘
生产环境都是专业存储,直接划LUN块映射,需要注意的是安装多路径软件包.
创建共享磁盘
实验是在 oracle vm virtualbox 进行的,进入指定的共享盘目录后,可以依次执行如下命令:
"C:\Program Files\Oracle\VirtualBox\VBoxManage" createmedium disk --filename share01.vdi --size 10000 --format VDI --variant Fixed
"C:\Program Files\Oracle\VirtualBox\VBoxManage" createmedium disk --filename share02.vdi --size 11000 --format VDI --variant Fixed
"C:\Program Files\Oracle\VirtualBox\VBoxManage" createmedium disk --filename share03.vdi --size 12000 --format VDI --variant Fixed
"C:\Program Files\Oracle\VirtualBox\VBoxManage" createmedium disk --filename share04.vdi --size 13000 --format VDI --variant Fixed
依次输出的 UUID 如下,
Medium created. UUID: bf451e2d-5f56-4978-aadf-cdd9db84112b
Medium created. UUID: e4743317-9dd9-4d3e-a252-63eebd75bdbf
Medium created. UUID: 8a61a886-7854-489a-bf92-7fbb4ab0432e
Medium created. UUID: de49f559-e2f4-4a23-8783-be1e6e7767d0
添加共享磁盘
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac1 56.221" --storagectl SATA --port 1 --device 0 --type hdd --medium bf451e2d-5f56-4978-aadf-cdd9db84112b --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac2 56.222" --storagectl SATA --port 1 --device 0 --type hdd --medium bf451e2d-5f56-4978-aadf-cdd9db84112b --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac1 56.221" --storagectl SATA --port 2 --device 0 --type hdd --medium e4743317-9dd9-4d3e-a252-63eebd75bdbf --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac2 56.222" --storagectl SATA --port 2 --device 0 --type hdd --medium e4743317-9dd9-4d3e-a252-63eebd75bdbf --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac1 56.221" --storagectl SATA --port 3 --device 0 --type hdd --medium 8a61a886-7854-489a-bf92-7fbb4ab0432e --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac2 56.222" --storagectl SATA --port 3 --device 0 --type hdd --medium 8a61a886-7854-489a-bf92-7fbb4ab0432e --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac1 56.221" --storagectl SATA --port 4 --device 0 --type hdd --medium de49f559-e2f4-4a23-8783-be1e6e7767d0 --mtype shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach "centos 7.4 rac2 56.222" --storagectl SATA --port 4 --device 0 --type hdd --medium de49f559-e2f4-4a23-8783-be1e6e7767d0 --mtype shareable
启用共享硬盘
"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifymedium disk bf451e2d-5f56-4978-aadf-cdd9db84112b --type shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifymedium disk e4743317-9dd9-4d3e-a252-63eebd75bdbf --type shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifymedium disk 8a61a886-7854-489a-bf92-7fbb4ab0432e --type shareable
"C:\Program Files\Oracle\VirtualBox\VBoxManage" modifymedium disk de49f559-e2f4-4a23-8783-be1e6e7767d0 --type shareable
修改网卡
两台机器的网络都参考该设置
# vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s3
UUID=6f5fccd7-ded5-4a09-b1bd-634274c27ebc
DEVICE=enp0s3
ONBOOT=yes
IPADDR=192.168.56.221
PREFIX=24
GATEWAY=192.168.56.1
NETMASK=255.255.255.0
重启网络
# systemctl status netowrk;
systemctl restart netowrk;
修改主机名
两台机器,分别为nodea,nodeb
# vi /etc/hostname
nodea
# vi /etc/hosts
192.168.56.221 nodea
192.168.56.222 nodeb
192.168.156.221 nodea-priv
192.168.156.222 nodeb-priv
192.168.156.231 nodea-priv
192.168.156.232 nodeb-priv
192.168.56.231 nodea-vip
192.168.56.232 nodeb-vip
192.168.56.241 node-scan
设置 shm 和 swap
oracle 的内存管理方式为 amm 和 asmm
amm 方式需要使用 /dev/shm,和 linux 的 hugepage 不兼容.
asmm 方式在大内存情况下必须使用 huage,否则 pagetable 过大会导致系统hang.
请参考 <<linux tmpfs /dev/shm 简述>> <<linux hugepage + oracle 11.2.0.4>>
# df -h /dev/shm
# vi /etc/fstab
tmpfs /dev/shm tmpfs defaults,size=1.5g 0 0
# mount -o remount /dev/shm
# free -m
1 GB~2 GB：内存的1.5倍
2 GB~16 GB：与内存相同
超过16 GB：16 GB
注意：如果为Linux服务器启用HugePages，则应在计算交换空间之前从可用内存中扣除分配给HugePages的内存。
关闭selinux
# vi /etc/selinux/config
SELINUX=DISABLED
或者
# vi /etc/sysconfig/selinux
SELINUX=DISABLED
可以看出两个文件是ln -s 的关系：
# ls -l /etc/sysconfig/selinux
/etc/sysconfig/selinux -> ../selinux/config
修改内核参数
# vi /etc/sysctl.conf
#ORACLE SETTING
#kernel.shmall = 18446744073692774399
#kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
fs.aio-max-nr = 1048576
fs.file-max = 68116544
kernel.sem = 250 32000 100 128
kernel.panic_on_oops = 1
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
#大页设置,大内存时必须使用
#vm.nr_hugepages = xxxx
立即生效
# sysctl -p
修改 pam
# vi /etc/pam.d/login
#ORACLE SETTING
session required /lib64/security/pam_limits.so
禁用ipv6,numa,thp,io scheduler
# vi /etc/default/grub
ipv6.disable=1 numa=off transparent_hugepage=never elevator=deadline
执行下面命令,生成新的grub
# grub2-mkconfig -o /boot/grub2/grub.cfg
# lsmod | grep ipv6
下面是另外一种方法禁用 linux 的透明大页
# vi /etc/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
fi
下面可以针对某个硬盘具体设置 io Scheduler
查看
# cat /sys/block/${ASM_DISK}/queue/scheduler
修改udev 配置文件
# vi /etc/udev/rules.d/60-oracle-schedulers.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"
udev 绑定硬盘
参考另外一篇博客:<<rhel 7.x 使用 udev scsi rules 配置裸设备>>https://blog.csdn.net/ctypyb2002/article/details/78668499
有多个disk,可以多次获取RESULT,在 Linux 7下，可以使用如下命令：
# /usr/lib/udev/scsi_id -g -u /dev/sdb
1ATA_VBOX_HARDDISK_VBc15c17be-e6e41f2d
创建 asmasmdevices 规则文件
# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBc15c17be-e6e41f2d", SYMLINK+="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
重新加载分区
# /usr/sbin/partprobe /dev/sdb
partprobe：
将磁盘分区表变化信息通知内核，请求操作系统重新加载分区表。
-d 不更新内核
-s 显示磁盘分区汇总信息
-h 显示帮助信息
-v 显示版本信息
加载udev 配置文件
# udevadm control --reload-rules
# systemctl status systemd-udevd.service;
systemctl enable systemd-udevd.service;
# udevadm test /sys/block/sdb
# udevadm info --query=all --path=/sys/block/sdb
# udevadm info --query=all --name=sdb
检查udev绑定情况
# ls -l /dev/asm* /dev/sd*
lrwxrwxrwx. 1 root root 3 Jan 24 09:16 /dev/asm-diskb -> sdb
lrwxrwxrwx. 1 root root 3 Jan 24 09:16 /dev/asm-diskc -> sdc
lrwxrwxrwx. 1 root root 3 Jan 24 09:16 /dev/asm-diskd -> sdd
lrwxrwxrwx. 1 root root 3 Jan 24 09:16 /dev/asm-diske -> sde
brw-rw----. 1 root disk 8, 0 Jan 24 08:53 /dev/sda
brw-rw----. 1 root disk 8, 1 Jan 24 08:53 /dev/sda1
brw-rw----. 1 root disk 8, 2 Jan 24 08:53 /dev/sda2
brw-rw----. 1 root root 8, 16 Jan 24 09:16 /dev/sdb
brw-rw----. 1 root root 8, 32 Jan 24 09:16 /dev/sdc
brw-rw----. 1 root root 8, 48 Jan 24 09:16 /dev/sdd
brw-rw----. 1 root root 8, 64 Jan 24 09:16 /dev/sde
禁用zeroconf
# vi /etc/sysconfig/network
NOZEROCONF=yes
创建用户组
# groupadd -g 10000 oinstall;
groupadd -g 10001 dba;
groupadd -g 10002 oper;
# groupadd -g 10003 asmadmin;
groupadd -g 10004 asmdba;
groupadd -g 10005 asmoper;
# groupadd -g 10006 backupdba;
groupadd -g 10007 dgdba;
groupadd -g 10008 kmdba;
groupadd -g 10009 racdba;
创建用户
# useradd -u 10000 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,asmdba,racdba oracle
# useradd -u 10001 -g oinstall -G dba,asmadmin,asmdba,asmoper,racdba grid
查看用户
# id oracle
uid=10000(oracle)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10002(oper),
10004(asmdba),
10006(backupdba),
10007(dgdba),
10008(kmdba),
10009(racdba)
# id grid
uid=10001(grid)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10003(asmadmin),
10004(asmdba),
10005(asmoper),
10009(racdba)
修改密码
# passwd oracle
# passwd grid
创建目录
# mkdir -p /u01/app/gridbase/12.2.0/grid_1;
mkdir -p /u01/app/grid/product/12.2.0/grid_1;
mkdir -p /u01/app/oracle/product/12.2.0/db_1;
# chown -R grid:oinstall /u01;
chown -R grid:oinstall /u01/app/gridbase;
chown -R grid:oinstall /u01/app/grid;
# chown -R oracle:oinstall /u01/app/oracle;
chmod -R 775 /u01/;
配置ssh信任
root,grid,oracle 三个用户都做ssh信任
# ssh-keygen -t rsa
# ssh-copy-id root@nodea;
# ssh-copy-id root@nodeb;
# su - grid
$ ssh-keygen -t rsa
$ ssh-copy-id grid@nodea;
$ ssh-copy-id grid@nodeb;
# su - oracle
$ ssh-keygen -t rsa
$ ssh-copy-id oracle@nodea;
$ ssh-copy-id oracle@nodeb;
修改资源限制
# vi /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 65536
grid soft nofile 1024
grid hard nofile 65536
grid soft stack unlimited
grid hard stack unlimited
grid soft core unlimited
grid hard core unlimited
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
grid soft memlock unlimited
grid hard memlock unlimited
oracle soft nproc 2047
oracle hard nproc 65536
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack unlimited
oracle hard stack unlimited
oracle soft core unlimited
oracle hard core unlimited
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
oracle soft memlock unlimited
oracle hard memlock unlimited
grid 用户环境变量
两台机器的 ORACLE_SID 不一样
# vi /home/grid/.bash_profile
PS1="[`whoami`@`hostname`:"'$PWD]$ '
umask 022
export ORACLE_BASE=/u01/app/gridbase/12.2.0/grid_1
export ORACLE_HOME=/u01/app/grid/product/12.2.0/grid_1
export ORACLE_SID=+ASM1
#export ORACLE_SID=+ASM2
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export PATH=$ORACLE_HOME/bin:$PATH
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
oracle 用户环境变量
两台机器的 ORACLE_SID 不一样
基于 policy 和 admin 管理方式的的 SID不一样,需要注意一下.
# vi /home/oracle/.bash_profile
PS1="[`whoami`@`hostname`:"'$PWD]$ '
umask 022
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0/db_1
export ORACLE_SID=orcl01
#export ORACLE_SID=orcl02
#export ORA_NLS10=$ORACLE_HOME/nls/data
#export TNS_ADMIN=$ORACLE_HOME/network/admin
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
配置 epel 源
# wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo
# yum clean all
# yum makecache
安装依赖包
# yum install nscd xclock nmon lrzsz zip unzip \
dos2unix pigz iputils iftop git tree sysstat make dstat wget bzip2 iputils \
binutils compat-libcap1 compat-libstdc++-33 compat-libstdc++-33.i686 elfutils-libelf-devel \
gcc gcc-c++ glibc glibc*.i686 glibc-devel glibc-devel*.i686 ksh libgcc libgcc*.i686 \
libstdc++ libstdc++*.i686 libstdc++-devel libstdc++-devel*.i686 libaio libaio*.i686 libaio-devel libaio-devel*.i686 \
libXext libXext.i686 libXtst libXtst.i686 libX11 libX11.i686 libXau libXau.i686 libxcb libxcb.i686 \
libXi libXi.i686 make sysstat unixODBC unixODBC*.i686 unixODBC-devel unixODBC-devel*.i686 \
readline libtermcap-devel pdksh net-tools nfs-utils smartmontools libXp
调整 /etc/nsswitch.conf
检查 /etc/resolv.conf
确保每个节点的文件内容都一致
如果scan使用的是hosts方式,相关的告警可以忽略.
禁用ntp服务
# systemctl stop chronyd;
systemctl disable chronyd;
systemctl stop ntpd.service;
systemctl disable ntpd.service;
systemctl stop ntpdate.service;
systemctl disable ntpdate.service;
# mv /etc/chrony.conf /etc/chrony.conf.bak;
mv /etc/ntp.conf /etc/ntp.conf.bak;
禁止avahi-daemon服务
avahi-daemon 服务会影响 oracle的多波通信，进而导致节点重启
因此，oracle环境下不能启用 avahi-daemon 服务
# systemctl stop avahi-daemon.service;
systemctl disable avahi-daemon.service;
禁用一些服务
# systemctl disable iptables.service;
systemctl disable firewalld.service;
systemctl disable cups.service;
# systemctl disable ip6tables.service;
systemctl disable firstboot-graphical.service;
systemctl disable postfix.service;
systemctl disable NetworkManager.service;
确保以下服务正常
# systemctl enable nscd.service;
systemctl enable crond.service;
systemctl enable sshd.service;
systemctl enable sysstat.service;
systemctl enable network.service;
/sbin/chkconfig network on;
最后一步,重启os
# reboot
参考:
Linux: How to Check Current Shared Memory, Semaphore Values (Doc ID 226209.1)