サーバ構成やHeartbeatのインストールなどについては前のエントリを参照。
Hertbeatの設定
/etc/ha.d/ha.cf (hastur)
logfile /var/log/ha-log keepalive 2 deadtime 30 warntime 10 initdead 120 ucast eth0 10.0.0.170 auto_failback off node hastur node nyar
/etc/ha.d/ha.cf (nyar)
logfile /var/log/ha-log keepalive 2 deadtime 30 warntime 10 initdead 120 ucast eth0 10.0.0.171 auto_failback off node hastur node nyar
/etc/ha.d/authkeys (0600)
auth 1 1 crc
/etc/ha.d/haresources (hastur/nyar共通)
hastur ENI
/etc/ha.d/resource.d/ENI (0755)
#!/bin/bash ENI_ID=eni-2d33af44 RETVAL=0 case "$1" in start) /usr/sbin/attach-eni -n $ENI_ID ;; stop) # nothing to do ;; status|monitor) /sbin/ifconfig | fgrep -q eth1 if [ $? -eq 0 ]; then echo "$ENI_ID is attached." else echo "$ENI_ID is not attached." RETVAL=1 fi ;; *) echo $"Usage: $0 {start|stop|status|monitor}" RETVAL=2 ;; esac exit $RETVAL
動作確認
まず、hasterでheartbeatを起動。
[ec2-user@hastur ~]$ sudo /etc/init.d/heartbeat start
Aug 19 09:36:17 nyar heartbeat: [5255]: info: Pacemaker support: false
Aug 19 09:36:17 nyar heartbeat: [5255]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Aug 19 09:36:17 nyar heartbeat: [5255]: info: **************************
Aug 19 09:36:17 nyar heartbeat: [5255]: info: Configuration validated. Starting heartbeat 3.0.5
Aug 19 09:36:17 nyar heartbeat: [5256]: info: heartbeat: version 3.0.5
Aug 19 09:36:17 nyar heartbeat: [5256]: info: Heartbeat generation: 1345368678
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: bound send socket to device: eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: bound receive socket to device: eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.171
Aug 19 09:36:17 nyar heartbeat: [5256]: info: Local status now set to: 'up'
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Link hastur:eth0 up.
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Status update for node hastur: status active
harc[5263]: 2012/08/19_09:36:18 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Comm_now_up(): updating status to active
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Local status now set to: 'active'
Aug 19 09:36:19 nyar heartbeat: [5256]: info: remote resource transition completed.
Aug 19 09:36:19 nyar heartbeat: [5256]: info: remote resource transition completed.
Aug 19 09:36:19 nyar heartbeat: [5256]: info: Local Resource acquisition completed. (none)
Aug 19 09:36:19 nyar heartbeat: [5256]: info: Initial resource acquisition complete (T_RESOURCES(them))
起動したタイミングでENIがhasturにアタッチされる。
[ec2-user@hastur ~]$ ip addr show
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.171/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d8/64 scope link
valid_lft forever preferred_lft forever
8: eth1:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever
nyarでもheartbeatを起動。
[ec2-user@nyar ~]$ sudo /etc/init.d/heartbeat start
対向サーバがアクティブであることが認識される。
Aug 19 09:51:00 nyar heartbeat: [5662]: info: Pacemaker support: false
Aug 19 09:51:00 nyar heartbeat: [5662]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Aug 19 09:51:00 nyar heartbeat: [5662]: info: **************************
Aug 19 09:51:00 nyar heartbeat: [5662]: info: Configuration validated. Starting heartbeat 3.0.5
Aug 19 09:51:00 nyar heartbeat: [5663]: info: heartbeat: version 3.0.5
Aug 19 09:51:00 nyar heartbeat: [5663]: info: Heartbeat generation: 1345368679
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: bound send socket to device: eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: bound receive socket to device: eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.171
Aug 19 09:51:00 nyar heartbeat: [5663]: info: Local status now set to: 'up'
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Link hastur:eth0 up.
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Status update for node hastur: status active
harc[5670]: 2012/08/19_09:51:01 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Comm_now_up(): updating status to active
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Local status now set to: 'active'
Aug 19 09:51:02 nyar heartbeat: [5663]: info: remote resource transition completed.
Aug 19 09:51:02 nyar heartbeat: [5663]: info: remote resource transition completed.
Aug 19 09:51:02 nyar heartbeat: [5663]: info: Local Resource acquisition completed. (none)
Aug 19 09:51:02 nyar heartbeat: [5663]: info: Initial resource acquisition complete (T_RESOURCES(them))
hasterでheartbeatを殺す。
[ec2-user@hastur ~]$ sudo killall -9 heartbeat
nyarにフェイルオーバーし、ENIがnyarにアタッチされる。
Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: node hastur: is dead
Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: No STONITH device configured.
Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: Shared disks are not protected.
Aug 19 09:54:13 nyar heartbeat: [5663]: info: Resources being acquired from hastur.
Aug 19 09:54:13 nyar heartbeat: [5663]: info: Link hastur:eth0 dead.
harc[5760]: 2012/08/19_09:54:13 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:54:13 nyar heartbeat: [5761]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys nyar] to acquire.
mach_down[5789]: 2012/08/19_09:54:13 info: Taking over resource group ENI
ResourceManager[5815]: 2012/08/19_09:54:13 info: Acquiring resource group: hastur ENI
ResourceManager[5815]: 2012/08/19_09:54:13 info: Running /etc/ha.d/resource.d/ENI start
mach_down[5789]: 2012/08/19_09:54:19 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[5789]: 2012/08/19_09:54:19 info: mach_down takeover complete for node hastur.
Aug 19 09:54:19 nyar heartbeat: [5663]: info: mach_down takeover complete.
[ec2-user@nyar ~]$ ip addr show
1: lo:mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.170/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d7/64 scope link
valid_lft forever preferred_lft forever
8: eth1:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever
ダウンタイムは19s。
09:54:11 AM 64 bytes from 10.0.0.150: icmp_seq=1102 ttl=64 time=0.349 ms
09:54:12 AM 64 bytes from 10.0.0.150: icmp_seq=1103 ttl=64 time=0.388 ms
09:54:13 AM 64 bytes from 10.0.0.150: icmp_seq=1104 ttl=64 time=0.383 ms
09:54:32 AM 64 bytes from 10.0.0.150: icmp_seq=1123 ttl=64 time=0.303 ms
09:54:33 AM 64 bytes from 10.0.0.150: icmp_seq=1124 ttl=64 time=0.312 ms
09:54:34 AM 64 bytes from 10.0.0.150: icmp_seq=1125 ttl=64 time=0.317 ms