版权声明：本文为作者原创，转载请标明出处！ https://blog.csdn.net/kiral07/article/details/87629569
Oracle 12c、18c ASMFD特性研究
在Oracle 12C以上版本，可以使用asmfd来取代udev规则下的asm磁盘设备绑定，同时他也具有过滤非法IO操作的特性。下面的环境基于RHEL 7.4,测试将UDEV存储转移到AFD磁盘路径。
一、afd配置调整
1、root用户下添加grid环境变量
[root@rac1 ~]# export ORACLE_HOME=/u01/app/12.2.0/grid
[root@rac1 ~]# export ORACLE_BASE=/tmp
2、获取当前asm磁盘组发现路径
[root@rac1 ~]# $ORACLE_HOME/bin/asmcmd dsget
parameter:/dev/asm*
profile:/dev/asm*
3、添加AFD发现路径
[root@rac1 ~]# asmcmd dsset '/dev/asm*','AFD:*'
[root@rac1 ~]# $ORACLE_HOME/bin/asmcmd dsget
parameter:/dev/asm*, AFD:*
profile:/dev/asm*,AFD:*
4、查看节点信息
[root@rac1 ~]# olsnodes -a
rac1 Hub
rac2 Hub
以下需要在所有节点运行
5、停止crs
[root@rac1 ~]# crsctl stop crs
6、安装oracle afd
节点1、节点2
加载以及配置AFD
[root@rac1 yum.repos.d]# asmcmd afd_configure
备注：在7.4以上的redhat或者centos下需要升级kmod才可以启用AFD，在前面一篇文章中已有介绍
解决在RHEL/CentOS7.4以上版本无法使用AFD(Oracle ASMFD)特性
https://blog.csdn.net/kiral07/article/details/87629679
加载afd过程：
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
查询afd状态信息
扫描二维码关注公众号，回复：
5435345 查看本文章
[root@rac1 yum.repos.d]# asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1'
7、启动crs
[root@rac1 yum.repos.d]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
8、查看当前存储设备
[root@rac1 yum.repos.d]# ll /dev/mapper/mpath*
lrwxrwxrwx 1 root root 7 Feb 15 17:18 /dev/mapper/mpathc -> ../dm-1
lrwxrwxrwx 1 root root 7 Feb 15 17:18 /dev/mapper/mpathd -> ../dm-0
此处使用多路径设备mpathc、mpathd
[root@rac2 ~]# multipath -ll
mpathd (14f504e46494c45526147693538302d577037452d39596459) dm-1 OPNFILER,VIRTUAL-DISK
size=30G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 34:0:0:1 sdc 8:32 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 35:0:0:1 sde 8:64 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 36:0:0:1 sdg 8:96 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 37:0:0:1 sdi 8:128 active ready running
mpathc (14f504e46494c45524f444c7844412d717a557a2d6b7a6752) dm-0 OPNFILER,VIRTUAL-DISK
size=40G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 34:0:0:0 sdb 8:16 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 35:0:0:0 sdd 8:48 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 36:0:0:0 sdf 8:80 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 37:0:0:0 sdh 8:112 active ready running
9、添加afd发现路径
切换到grid用户
[root@rac2 ~]# su - grid
使用afd_dsset添加存储路径
[grid@rac2:/home/grid]$asmcmd afd_dsset '/dev/mapper/mpath*'
[grid@rac2:/home/grid]$asmcmd afd_dsget
AFD discovery string: /dev/mapper/mpath*
此时未添加afd label所以为空
[grid@rac2:/home/grid]$asmcmd afd_lsdsk
There are no labelled devices.
至此从步骤5，在rac所有节点已运行所有命令
二、转移UDEV设备到AFD路径
1、查看当前ocr以及voting files磁盘组
[root@rac1 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +CRS
[root@rac1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4e20265767f54f49bf12bd72f367217f (/dev/asm_crs) [CRS]
Located 1 voting disk(s).
2、查看crs磁盘组对应的udev存储路径
[root@rac1 ~]# su - grid
[grid@rac1:/home/grid]$asmcmd lsdsk -G crs
Path
/dev/asm_crs
3、停止rac集群
[root@rac1 ~]# crsctl stop cluster -all
4、转移udev设备到afd
使用label添加asmcrs磁盘组，将udev规则下的磁盘路径转移到afd
[grid@rac1:/home/grid]$asmcmd afd_label asmcrs /dev/mapper/mpathc --migrate
crs磁盘组已加载完毕
[grid@rac1:/home/grid]$asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS ENABLED /dev/mapper/mpathc
备注：由于当前磁盘组已被asm使用，必须使用migrate才可以进行转移。
添加另外一块data磁盘组
[grid@rac1:/home/grid]$asmcmd afd_label asmdata /dev/mapper/mpathd --migrate
查看afd磁盘组，已加载完毕
[grid@rac1:/home/grid]$asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS ENABLED /dev/mapper/mpathc
ASMDATA ENABLED /dev/mapper/mpathd
5、在其余节点扫描afd设备
[grid@rac2:/home/grid]$asmcmd afd_scan
[grid@rac2:/home/grid]$asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS ENABLED /dev/mapper/mpathc
ASMDATA ENABLED /dev/mapper/mpathd
6、启动rac集群
[root@rac1 ~]# crsctl start cluster -all
7、在asm实例下查询asm磁盘信息
原udev asm存储路径信息还能查看到
[root@rac1 ~]# asmcmd lsdsk
Path
AFD:ASMCRS
AFD:ASMDATA
SQL> col name for a10
SQL> col label for a10
SQL> col path for a15
SQL> select NAME,LABEL,PATH from V$ASM_DISK;
NAME LABEL PATH
---------- ---------- ---------------
ASMDATA /dev/asm_data ---->之前udev asm路径
ASMCRS /dev/asm_crs ---->之前udev asm路径
CRS_0000 ASMCRS AFD:ASMCRS
DATA_0000 ASMDATA AFD:ASMDATA
8、修改发现路径
[grid@rac1:/home/grid]$asmcmd dsget
parameter:/dev/asm*, AFD:*
profile:/dev/asm*,AFD:*
只保留afd路径
[grid@rac1:/home/grid]$asmcmd dsset 'AFD:*'
[grid@rac1:/home/grid]$asmcmd dsget
parameter:AFD:*
profile:AFD:*
再次查询udev路径下的设备已没有
SQL> select NAME,LABEL,PATH from V$ASM_DISK;
NAME LABEL PATH
---------- ---------- ---------------
CRS_0000 ASMCRS AFD:ASMCRS
DATA_0000 ASMDATA AFD:ASMDATA
9、移除UDEV规则文件
[root@rac1 ~]# ll -hrt /etc/udev/rules.d/
total 12K
-rw-r--r-- 1 root root 297 Nov 3 17:04 99-oracle-asmdevices.rules.old
-rw-r--r-- 1 root root 224 Feb 15 17:13 53-afd.rules
-rw-r--r-- 1 root root 957 Feb 18 08:55 70-persistent-ipoib.rules
99-oracle-asmdevices.rules重命名之后已无法发现磁盘
[root@rac1 ~]# ll /dev/asm*
ls: cannot access /dev/asm*: No such file or directory
使用afd特性的磁盘组未受影响
[grid@rac2:/home/grid]$asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS ENABLED /dev/mapper/mpathc
ASMDATA ENABLED /dev/mapper/mpathd
到此为止已完成afd的配置与加载
三、ASM磁盘组dd格式化测试
Oracle的afd特性可以过滤掉”非规范“的io操作，只要不是用于oracle的io操作都会被过滤掉，下面使用dd格式化asm整个磁盘组做测试
1、增加一个磁盘组”asmtest“用来做dd格式化实验
[root@rac1 ~]# asmcmd afd_label asmtest /dev/mapper/mpathe
[root@rac1 ~]# asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS ENABLED /dev/mapper/mpathc
ASMDATA ENABLED /dev/mapper/mpathd
ASMTEST ENABLED /dev/mapper/mpathe
[root@rac1 ~]# su - grid
[grid@rac1:/home/grid]$sqlplus / as sysasm
创建asmtest磁盘组
SQL> create diskgroup asmtest external redundancy disk 'AFD:asmtest';
Diskgroup created.
2、创建测试表空间asmtest以及测试表
SQL> create tablespace asmtest datafile '+asmtest' size 100m;
SQL> create table afd (id number) tablespace asmtest;
SQL> insert into afd values (1);
1 row created.
SQL> commit;
Commit complete.
SQL> select * from afd;
ID
----------
1
3、dd格式化
格式化整个磁盘组”asmtest“ —>/dev/mapper/mpathe
[root@rac1 ~]# dd if=/dev/zero of=/dev/mapper/mpathe
dd: writing to dev/mapper/mpathe No space left on device
2097153+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 70.6 s, 15.2 MB/s
4、再次做创建表空间操作
SQL> create tablespace asmtest2 datafile '+asmtest' size 100m;
SQL> create table afd2 (id number) tablespace asmtest2;
SQL> insert into afd2 values (2);
1 row created.
SQL> commit;
Commit complete.
SQL> select * from afd;
ID
----------
2
SQL> ALTER system checkpoint;
System altered.
checkpoint之后也没有报错
5、禁用afd Filter
[root@rac1 ~]# asmcmd afd_filter -d
备注（-d是disable、-e是enable）
[root@rac1 ~]# asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ASMCRS DISABLED /dev/mapper/mpathc
ASMDATA DISABLED /dev/mapper/mpathd
ASMTEST DISABLED /dev/mapper/mpathe
6、再次做dd格式化
[root@rac1 ~]# dd if=/dev/zero of=/dev/mapper/mpathe
dd: writing to dev/mapper/mpathe No space left on device
2097153+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 89.2804 s, 12.0 MB/s
7、在数据库中测试
SQL> insert into afd values (3);
1 row created.
SQL> commit;
Commit complete.
SQL> alter system checkpoint;
alter system checkpoint
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 21718
Session ID: 35 Serial number: 63358
数据库已崩溃
数据库重启之后已无法启动
[oracle@rac1:/home/oracle]$sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Mon Feb 18 14:55:41 2019
Copyright © 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-39510: CRS error performing start on instance 'orcl1' on 'orcl'
CRS-2672: Attempting to start 'ora.AFDTEST.dg' on 'rac1'
CRS-2672: Attempting to start 'ora.AFDTEST.dg' on 'rac2'
CRS-2674: Start of 'ora.AFDTEST.dg' on 'rac1' failed
CRS-2674: Start of 'ora.AFDTEST.dg' on 'rac2' failed
CRS-0215: Could not start resource 'ora.orcl.db 1 1'.
clsr_start_resource:260 status:215
clsrapi_start_db:start_asmdbs status:215
四、拓展研究
在配置完afd之后，在dev路径下会有lable过后的磁盘
[root@rac1 ~]# ll /dev/oracleafd/disks/
total 8
-rwxrwx--- 1 grid oinstall 19 Feb 18 13:40 ASMCRS
-rwxrwx--- 1 grid oinstall 19 Feb 18 13:40 ASMDATA
查看此设备内容发现对应为多路径设备
[root@rac1 ~]# cat /dev/oracleafd/disks/ASMCRS
/dev/mapper/mpathc
[root@rac1 ~]# cat /dev/oracleafd/disks/ASMDATA
/dev/mapper/mpathd
udev规则下会有afd的规则文件
[grid@rac1:/home/grid]$ll -hrt /etc/udev/rules.d/
total 12K
-rw-r--r-- 1 root root 297 Nov 3 17:04 99-oracle-asmdevices.rules.old
-rw-r--r-- 1 root root 224 Feb 15 17:13 53-afd.rules
-rw-r--r-- 1 root root 957 Feb 18 08:55 70-persistent-ipoib.rules
[grid@rac1:/home/grid]$cat /etc/udev/rules.d/53-afd.rules
#
# AFD devices
KERNEL=="oracleafd/.*", OWNER="grid", GROUP="asmadmin", MODE="0775"
KERNEL=="oracleafd/*", OWNER="grid", GROUP="asmadmin", MODE="0775"
KERNEL=="oracleafd/disks/*", OWNER="grid", GROUP="asmadmin", MODE="0664"
五、总结
Oracle的afd特性可以将”危险“IO操作进行重定向，具体原理不得而知，其本质还是通过系统内核使用udev规则加载磁盘设备。
六、参考文献
https://www.oracle.com/technetwork/cn/articles/database/asmfd-2398572-zhs.html