Heartbeatとattach-eniコマンドを使った冗長化

サーバ構成やHeartbeatのインストールなどについては前のエントリを参照。

Hertbeatの設定

/etc/ha.d/ha.cf (hastur)
logfile /var/log/ha-log
keepalive 2
deadtime 30
warntime 10
initdead 120
ucast eth0 10.0.0.170
auto_failback off
node    hastur
node    nyar
/etc/ha.d/ha.cf (nyar)
logfile /var/log/ha-log
keepalive 2
deadtime 30
warntime 10
initdead 120
ucast eth0 10.0.0.171
auto_failback off
node    hastur
node    nyar
/etc/ha.d/authkeys (0600)
auth 1
1 crc
/etc/ha.d/haresources (hastur/nyar共通)
hastur ENI
/etc/ha.d/resource.d/ENI (0755)
#!/bin/bash
ENI_ID=eni-2d33af44
RETVAL=0

case "$1" in
start)
  /usr/sbin/attach-eni -n $ENI_ID
  ;;

stop)
  # nothing to do
  ;;

status|monitor)
  /sbin/ifconfig | fgrep -q eth1

  if [ $? -eq 0 ]; then
    echo "$ENI_ID is attached."
  else
    echo "$ENI_ID is not attached."
    RETVAL=1
  fi
  ;;

*)
  echo $"Usage: $0 {start|stop|status|monitor}"
  RETVAL=2
  ;;

esac

exit $RETVAL

動作確認

まず、hasterでheartbeatを起動。


[ec2-user@hastur ~]$ sudo /etc/init.d/heartbeat start

Aug 19 09:36:17 nyar heartbeat: [5255]: info: Pacemaker support: false
Aug 19 09:36:17 nyar heartbeat: [5255]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Aug 19 09:36:17 nyar heartbeat: [5255]: info: **************************
Aug 19 09:36:17 nyar heartbeat: [5255]: info: Configuration validated. Starting heartbeat 3.0.5
Aug 19 09:36:17 nyar heartbeat: [5256]: info: heartbeat: version 3.0.5
Aug 19 09:36:17 nyar heartbeat: [5256]: info: Heartbeat generation: 1345368678
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: bound send socket to device: eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: bound receive socket to device: eth0
Aug 19 09:36:17 nyar heartbeat: [5256]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.171
Aug 19 09:36:17 nyar heartbeat: [5256]: info: Local status now set to: 'up'
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Link hastur:eth0 up.
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Status update for node hastur: status active
harc[5263]: 2012/08/19_09:36:18 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Comm_now_up(): updating status to active
Aug 19 09:36:18 nyar heartbeat: [5256]: info: Local status now set to: 'active'
Aug 19 09:36:19 nyar heartbeat: [5256]: info: remote resource transition completed.
Aug 19 09:36:19 nyar heartbeat: [5256]: info: remote resource transition completed.
Aug 19 09:36:19 nyar heartbeat: [5256]: info: Local Resource acquisition completed. (none)
Aug 19 09:36:19 nyar heartbeat: [5256]: info: Initial resource acquisition complete (T_RESOURCES(them))


起動したタイミングでENIがhasturにアタッチされる。


[ec2-user@hastur ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.171/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d8/64 scope link
valid_lft forever preferred_lft forever
8: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever


nyarでもheartbeatを起動。


[ec2-user@nyar ~]$ sudo /etc/init.d/heartbeat start

対向サーバがアクティブであることが認識される。


Aug 19 09:51:00 nyar heartbeat: [5662]: info: Pacemaker support: false
Aug 19 09:51:00 nyar heartbeat: [5662]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Aug 19 09:51:00 nyar heartbeat: [5662]: info: **************************
Aug 19 09:51:00 nyar heartbeat: [5662]: info: Configuration validated. Starting heartbeat 3.0.5
Aug 19 09:51:00 nyar heartbeat: [5663]: info: heartbeat: version 3.0.5
Aug 19 09:51:00 nyar heartbeat: [5663]: info: Heartbeat generation: 1345368679
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: bound send socket to device: eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: bound receive socket to device: eth0
Aug 19 09:51:00 nyar heartbeat: [5663]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.171
Aug 19 09:51:00 nyar heartbeat: [5663]: info: Local status now set to: 'up'
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Link hastur:eth0 up.
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Status update for node hastur: status active
harc[5670]: 2012/08/19_09:51:01 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Comm_now_up(): updating status to active
Aug 19 09:51:01 nyar heartbeat: [5663]: info: Local status now set to: 'active'
Aug 19 09:51:02 nyar heartbeat: [5663]: info: remote resource transition completed.
Aug 19 09:51:02 nyar heartbeat: [5663]: info: remote resource transition completed.
Aug 19 09:51:02 nyar heartbeat: [5663]: info: Local Resource acquisition completed. (none)
Aug 19 09:51:02 nyar heartbeat: [5663]: info: Initial resource acquisition complete (T_RESOURCES(them))


hasterでheartbeatを殺す。


[ec2-user@hastur ~]$ sudo killall -9 heartbeat


nyarにフェイルオーバーし、ENIがnyarにアタッチされる。


Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: node hastur: is dead
Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: No STONITH device configured.
Aug 19 09:54:13 nyar heartbeat: [5663]: WARN: Shared disks are not protected.
Aug 19 09:54:13 nyar heartbeat: [5663]: info: Resources being acquired from hastur.
Aug 19 09:54:13 nyar heartbeat: [5663]: info: Link hastur:eth0 dead.
harc[5760]: 2012/08/19_09:54:13 info: Running /etc/ha.d//rc.d/status status
Aug 19 09:54:13 nyar heartbeat: [5761]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys nyar] to acquire.
mach_down[5789]: 2012/08/19_09:54:13 info: Taking over resource group ENI
ResourceManager[5815]: 2012/08/19_09:54:13 info: Acquiring resource group: hastur ENI
ResourceManager[5815]: 2012/08/19_09:54:13 info: Running /etc/ha.d/resource.d/ENI start
mach_down[5789]: 2012/08/19_09:54:19 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[5789]: 2012/08/19_09:54:19 info: mach_down takeover complete for node hastur.
Aug 19 09:54:19 nyar heartbeat: [5663]: info: mach_down takeover complete.

[ec2-user@nyar ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.170/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d7/64 scope link
valid_lft forever preferred_lft forever
8: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever


ダウンタイムは19s。


09:54:11 AM 64 bytes from 10.0.0.150: icmp_seq=1102 ttl=64 time=0.349 ms
09:54:12 AM 64 bytes from 10.0.0.150: icmp_seq=1103 ttl=64 time=0.388 ms
09:54:13 AM 64 bytes from 10.0.0.150: icmp_seq=1104 ttl=64 time=0.383 ms
09:54:32 AM 64 bytes from 10.0.0.150: icmp_seq=1123 ttl=64 time=0.303 ms
09:54:33 AM 64 bytes from 10.0.0.150: icmp_seq=1124 ttl=64 time=0.312 ms
09:54:34 AM 64 bytes from 10.0.0.150: icmp_seq=1125 ttl=64 time=0.317 ms