Linux 7使用 multipath及udev绑定磁盘
Jun062019
linux 7的multipath配置方法变化不大,但在/etc目录下面默认是没有multipath.conf文件的,默认也没有启用,需要激活。
[root@SL010A-IVDB02 ~]# ls /etc/multipath.conf ls: cannot access /etc/multipath.conf: No such file or directory [root@SL010A-IVDB02 ~]# multipath -ll May 16 13:00:26 | DM multipath kernel driver not loaded May 16 13:00:26 | /etc/multipath.conf does not exist, blacklisting all devices. May 16 13:00:26 | A default multipath.conf file is located at May 16 13:00:26 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf May 16 13:00:26 | You can run /sbin/mpathconf --enable to create May 16 13:00:26 | /etc/multipath.conf. See man mpathconf(8) for more details May 16 13:00:26 | DM multipath kernel driver not loaded
通过multipath –ll命令的输出可用看到,multipath默认没有启用,需要通过/sbin/mpathconf –enable来激活。
Multipath需要安装device-mapper和device-mapper-multipath包。
[root@SL010A-IVDB02 ~]# rpm -qa|grep device-mapper device-mapper-libs-1.02.149-8.el7.x86_64 device-mapper-1.02.149-8.el7.x86_64 device-mapper-multipath-0.4.9-123.el7.x86_64 device-mapper-event-libs-1.02.149-8.el7.x86_64 device-mapper-event-1.02.149-8.el7.x86_64 device-mapper-persistent-data-0.7.3-3.el7.x86_64 device-mapper-multipath-libs-0.4.9-123.el7.x86_64
下面执行/sbin/mpathconf –enable命令来启用multipath。
[root@SL010A-IVDB02 ~]# mpathconf --enable
mpathconf –enable命令会在/etc目录下创建multipath.conf文件,并激活multipath程序。
[root@SL010A-IVDB02 ~]# ls /etc/multipath.conf /etc/multipath.conf [root@SL010A-IVDB02 ~]# systemctl status multipathd.service ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2019-05-16 12:17:54 CST; 44min ago ConditionPathExists=/etc/multipath.conf was not met
启动multipath程序。
[root@SL010A-IVDB02 ~]# systemctl start multipathd.service [root@SL010A-IVDB02 ~]# systemctl status multipathd.service ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-05-16 13:02:25 CST; 3s ago Process: 80054 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS) Process: 80051 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS) Process: 80049 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS) Main PID: 80057 (multipathd) Tasks: 19 CGroup: /system.slice/multipathd.service └─80057 /sbin/multipathd May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathe: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathf: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathg: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathh: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathi: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathj: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathk: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathl: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: mpathm: event checker started May 16 13:02:25 SL010A-IVDB02 multipathd[80057]: path checkers start up Removed symlink /etc/systemd/system/default.target. Created symlink from /etc/systemd/system/default.target to /usr/lib/systemd/system/multi-user.target.
通过multipath –ll命令列出存储设备。
[root@SL010A-IVDB02 ~]# multipath -ll mpathe (36006016086003500c22ae6353b03e911) dm-4 DGC ,RAID 5 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:12 sdac 65:192 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:12 sdo 8:224 active ready running mpathd (36006016086003500ea9890223b03e911) dm-3 DGC ,RAID 5 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:11 sdn 8:208 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:11 sdab 65:176 active ready running mpathc (36006016086003500d2b6fbca3a03e911) dm-2 DGC ,RAID 5 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:10 sdm 8:192 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:10 sdaa 65:160 active ready running mpathb (360060160860035002689ec0a3903e911) dm-1 DGC ,RAID 10 size=2.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:1 sdr 65:16 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:1 sdd 8:48 active ready running mpatha (36006016086003500b49583f23803e911) dm-0 DGC ,RAID 10 size=2.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:0 sdc 8:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:0 sdq 65:0 active ready running mpathm (36006016086003500f0a963ac3a03e911) dm-12 DGC ,RAID 5 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:9 sdl 8:176 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:9 sdz 65:144 active ready running mpathl (360060160860035001819932c3a03e911) dm-11 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:8 sdy 65:128 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:8 sdk 8:160 active ready running mpathk (360060160860035000eed47083a03e911) dm-10 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:7 sdx 65:112 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:7 sdj 8:144 active ready running mpathj (36006016086003500482ebee93903e911) dm-9 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:6 sdw 65:96 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:6 sdi 8:128 active ready running mpathi (36006016086003500ba73dad43903e911) dm-8 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:5 sdh 8:112 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:5 sdv 65:80 active ready running mpathh (3600601608600350086c452a93903e911) dm-7 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:4 sdg 8:96 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:4 sdu 65:64 active ready running mpathg (36006016086003500b46155863903e911) dm-6 DGC ,RAID 10 size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 2:0:1:3 sdf 8:80 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:1:3 sdt 65:48 active ready running mpathf (360060160860035006c1c231e3903e911) dm-5 DGC ,RAID 10 size=2.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:1:2 sds 65:32 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 2:0:1:2 sde 8:64 active ready running
编辑/etc/multipath.conf文件,添加设备的绑定信息,也就是下文的multipaths部分。
[root@SL010A-IVDB02 ~]# vi /etc/multipath.conf # This is a basic configuration file with some examples, for device mapper # multipath. # # For a complete list of the default configuration values, run either # multipath -t # or # multipathd show config # # For a list of configuration options with descriptions, see the multipath.conf # man page ## By default, devices with vendor = "IBM" and product = "S/390.*" are ## blacklisted. To enable mulitpathing on these devies, uncomment the ## following lines. #blacklist_exceptions { # device { # vendor "IBM" # product "S/390.*" # } #} ## Use user friendly names, instead of using WWIDs as names. defaults { user_friendly_names yes find_multipaths yes } ## ## Here is an example of how to configure some standard options. ## # #defaults { # polling_interval 10 # path_selector "round-robin 0" # path_grouping_policy multibus # uid_attribute ID_SERIAL # prio alua # path_checker readsector0 # rr_min_io 100 # max_fds 8192 # rr_weight priorities # failback immediate # no_path_retry fail # user_friendly_names yes "/etc/multipath.conf" 95L, 2415C ## devnode lines are not recommended for blacklisting specific devices. ## #blacklist { # wwid 26353900f02796769 # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" # devnode "^hd[a-z]" #} #multipaths { # multipath { # wwid 3600508b4000156d700012000000b0000 # alias yellow # path_grouping_policy multibus # path_selector "round-robin 0" # failback manual # rr_weight priorities # no_path_retry 5 # } # multipath { # wwid 1DEC_____321816758474 # alias red # } #} #devices { # device { # vendor "COMPAQ " # product "HSV110 (C)COMPAQ" # path_grouping_policy multibus # path_checker readsector0 # path_selector "round-robin 0" # hardware_handler "0" # failback 15 # rr_weight priorities # no_path_retry queue # } # device { # vendor "COMPAQ " # product "MSA1000 " # path_grouping_policy multibus # } #} blacklist { } # Blacklist all devices by default. Remove this to enable multipathing # on the default devices. blacklist { devnode "sda" } ## Use user friendly names, instead of using WWIDs as names. defaults { user_friendly_names yes } multipaths { multipath { wwid 360060160860035006c1c231e3903e911 alias crs1 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 360060160860035002689ec0a3903e911 alias crs2 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500b49583f23803e911 alias crs3 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500c22ae6353b03e911 alias data1 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500ea9890223b03e911 alias data2 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500d2b6fbca3a03e911 alias data3 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500f0a963ac3a03e911 alias data4 path_grouping_policy multibus path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 360060160860035001819932c3a03e911 alias data5 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 360060160860035000eed47083a03e911 alias data6 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500482ebee93903e911 alias data7 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500ba73dad43903e911 alias data8 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 3600601608600350086c452a93903e911 alias data9 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } multipath { wwid 36006016086003500b46155863903e911 alias data10 path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 } } devices { # device { # vendor "COMPAQ " # product "HSV110 (C)COMPAQ" # path_grouping_policy multibus # getuid_callout "/sbin/scsi_id -g -u -s /block/%n" # path_checker readsector0 # path_selector "round-robin 0" # hardware_handler "0" # failback 15 # rr_weight priorities # no_path_retry queue # } device { vendor "EMC" product "VNX5800" path_grouping_policy multibus } }
通过multipath –r命令刷新multipath,使之生效。
[root@SL010A-IVDB02 ~]# multipath -r Jun 06 15:23:08 | /etc/multipath.conf line 98, duplicate keyword: blacklist Jun 06 15:23:08 | /etc/multipath.conf line 103, duplicate keyword: defaults reload: crs3 (36006016086003500b49583f23803e911) undef DGC ,RAID 10 size=2.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=70 status=undef |- 1:0:1:0 sdc 8:32 active ready running `- 4:0:1:0 sdq 65:0 active ready running reload: crs2 (360060160860035002689ec0a3903e911) undef DGC ,RAID 10 size=2.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=70 status=undef |- 1:0:1:1 sdd 8:48 active ready running `- 4:0:1:1 sdr 65:16 active ready running reload: data3 (36006016086003500d2b6fbca3a03e911) undef DGC ,RAID 5 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=70 status=undef |- 1:0:1:10 sdm 8:192 active ready running `- 4:0:1:10 sdaa 65:160 active ready running reload: data2 (36006016086003500ea9890223b03e911) undef DGC ,RAID 5 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=70 status=undef |- 1:0:1:11 sdn 8:208 active ready running `- 4:0:1:11 sdab 65:176 active ready running reload: data1 (36006016086003500c22ae6353b03e911) undef DGC ,RAID 5 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=30 status=undef |- 1:0:1:12 sdo 8:224 active ready running `- 4:0:1:12 sdac 65:192 active ready running reload: crs1 (360060160860035006c1c231e3903e911) undef DGC ,RAID 10 size=2.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=30 status=undef |- 1:0:1:2 sde 8:64 active ready running `- 4:0:1:2 sds 65:32 active ready running reload: data10 (36006016086003500b46155863903e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 1:0:1:3 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 4:0:1:3 sdt 65:48 active ready running reload: data9 (3600601608600350086c452a93903e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 1:0:1:4 sdg 8:96 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 4:0:1:4 sdu 65:64 active ready running reload: data8 (36006016086003500ba73dad43903e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 1:0:1:5 sdh 8:112 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 4:0:1:5 sdv 65:80 active ready running reload: data7 (36006016086003500482ebee93903e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 4:0:1:6 sdw 65:96 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 1:0:1:6 sdi 8:128 active ready running reload: data6 (360060160860035000eed47083a03e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 4:0:1:7 sdx 65:112 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 1:0:1:7 sdj 8:144 active ready running reload: data5 (360060160860035001819932c3a03e911) undef DGC ,RAID 10 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef |-+- policy='round-robin 0' prio=50 status=undef | `- 4:0:1:8 sdy 65:128 active ready running `-+- policy='round-robin 0' prio=10 status=undef `- 1:0:1:8 sdk 8:160 active ready running reload: data4 (36006016086003500f0a963ac3a03e911) undef DGC ,RAID 5 size=1.0T features='1 queue_if_no_path' hwhandler='1 emc' wp=undef `-+- policy='round-robin 0' prio=70 status=undef |- 1:0:1:9 sdl 8:176 active ready running `- 4:0:1:9 sdz 65:144 active ready running
绑定成功后,会在/dev/mapper目录下生成相关的设备链接。
[root@SL010A-IVDB02 ~]# ll /dev/mapper/ total 0 crw------- 1 root root 10, 236 May 17 10:20 control lrwxrwxrwx 1 root root 8 Jun 6 15:24 crs1 -> ../dm-12 lrwxrwxrwx 1 root root 7 Jun 6 15:23 crs2 -> ../dm-1 lrwxrwxrwx 1 root root 7 Jun 6 15:23 crs3 -> ../dm-7 lrwxrwxrwx 1 root root 7 Jun 6 15:25 data1 -> ../dm-0 lrwxrwxrwx 1 root root 8 Jun 6 15:23 data10 -> ../dm-10 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Jun 6 15:25 data3 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data4 -> ../dm-6 lrwxrwxrwx 1 root root 8 Jun 6 15:25 data5 -> ../dm-11 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data6 -> ../dm-9 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data7 -> ../dm-3 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data8 -> ../dm-8 lrwxrwxrwx 1 root root 7 Jun 6 15:23 data9 -> ../dm-2
然后使用UDEV绑定这些设备的权限。
[root@SL0101A-IVDB02 ~]# cd /etc/udev/rules.d/ [root@SL0101A-IVDB02 rules.d]# vi 99-asm.rules KERNEL=="dm-0", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-2", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-3", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-4", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-5", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-6", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-7", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-8", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-9", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-10", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-11", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm-12", OWNER="grid", GROUP="asmadmin", MODE="0660"
通过下面的命令使UDEV配置生效,Linux 7已经没有start_udev命令了。
[root@SL010A-IVDB02 rules.d]# udevadm trigger --type=devices --action=change
或者使用下面的命令重新加载UDEV的配置。
[root@SL010A-IVDB02 rules.d]# udevadm control --reload-rules
再或者,直接重启UDEV服务。
[root@SL010A-IVDB02 rules.d]# systemctl restart systemd-udev-trigger.service [root@SL010A-IVDB02 rules.d]# ll /dev/dm-* brw-rw---- 1 grid asmadmin 252, 0 May 17 10:20 /dev/dm-0 brw-rw---- 1 grid asmadmin 252, 1 May 17 10:20 /dev/dm-1 brw-rw---- 1 grid asmadmin 252, 10 May 17 10:20 /dev/dm-10 brw-rw---- 1 grid asmadmin 252, 11 May 17 10:20 /dev/dm-11 brw-rw---- 1 grid asmadmin 252, 12 May 17 10:20 /dev/dm-12 brw-rw---- 1 grid asmadmin 252, 2 May 17 10:20 /dev/dm-2 brw-rw---- 1 grid asmadmin 252, 3 May 17 10:20 /dev/dm-3 brw-rw---- 1 grid asmadmin 252, 4 May 17 10:20 /dev/dm-4 brw-rw---- 1 grid asmadmin 252, 5 May 17 10:20 /dev/dm-5 brw-rw---- 1 grid asmadmin 252, 6 May 17 10:20 /dev/dm-6 brw-rw---- 1 grid asmadmin 252, 7 May 17 10:20 /dev/dm-7 brw-rw---- 1 grid asmadmin 252, 8 May 17 10:20 /dev/dm-8 brw-rw---- 1 grid asmadmin 252, 9 May 17 10:20 /dev/dm-9
最后,还需要重启服务器进行验证。