読者です 読者をやめる 読者になる 読者になる

Heartbeatとattach-eniコマンドを使った冗長化 その2: L7のヘルスチェック

L7のヘルスチェックを追加してみる。

とりあえず、hasterとnyarにmysql-serverをインストール。
以下のヘルスチェック用デーモンを作成。

/usr/local/bin/mysql-pingd
#!/bin/bash
INTERVAL=5

while true; do
  /usr/bin/mysqladmin ping --connect_timeout=3

  if [ $? -ne 0 ]; then
    /usr/lib64/heartbeat/heartbeat -k
  fi

  sleep $INTERVAL
done


ha.cfに以下の行を追加。

/etc/ha.d/ha.cf
...
respawn root /usr/local/bin/mysql-pingd

動作確認

ENIをhasturにアタッチして、hasturとnyarでheartbeatを起動。
ヘルスチェック用デーモンがheartbeatによって起動される。


Aug 19 13:21:35 hastur heartbeat: [24226]: info: Starting child client "/usr/local/bin/mysql-pingd" (0,0)
Aug 19 13:21:35 hastur heartbeat: [24693]: info: Starting "/usr/local/bin/mysql-pingd" as uid 0 gid 0 (pid 24693)

root 24226 0.0 0.4 50020 7064 ? SLs 13:14 0:00 heartbeat: master control process
root 24671 0.0 0.4 49968 7012 ? SL 13:21 0:00 \_ heartbeat: FIFO reader
root 24672 0.0 0.4 49964 7008 ? SL 13:21 0:00 \_ heartbeat: write: ucast eth0
root 24673 0.0 0.4 49964 7008 ? SL 13:21 0:00 \_ heartbeat: read: ucast eth0
root 24693 0.0 0.0 9228 1240 ? S 13:21 0:00 \_ /bin/bash /usr/local/bin/mysql-pingd
root 24711 0.0 0.0 4104 316 ? S 13:21 0:00 \_ sleep 5


初期状態ではcthulhuからhasturに接続される。


[ec2-user@cthulhu ~]$ mysql -h 10.0.0.150 -u scott -ptiger test -e "select * from foo"

                  • +
bar
                  • +
hastur
                  • +


hasterのmysqlを停止してみる。


[ec2-user@hastur ~]$ sudo /etc/init.d/mysqld stop
Stopping mysqld: [ OK ]
[ec2-user@hastur ~]$ tail /var/log/ha-log
Aug 19 13:21:36 hastur heartbeat: [24226]: info: remote resource transition completed.
Aug 19 13:21:36 hastur heartbeat: [24226]: info: Initial resource acquisition complete (T_RESOURCES(us))
Aug 19 13:24:46 hastur heartbeat: [24226]: info: Heartbeat shutdown in progress. (24226)
Aug 19 13:24:46 hastur heartbeat: [24893]: info: Giving up all HA resources.
ResourceManager[24906]: 2012/08/19_13:24:46 info: Releasing resource group: hastur ENI
ResourceManager[24906]: 2012/08/19_13:24:46 info: Running /etc/ha.d/resource.d/ENI stop
Aug 19 13:24:46 hastur heartbeat: [24893]: info: All HA resources relinquished.
Aug 19 13:24:47 hastur heartbeat: [24226]: info: killing /usr/local/bin/mysql-pingd process group 24693 with signal 15
Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBFIFO process 24671 with signal 15
Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBWRITE process 24672 with signal 15
Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBREAD process 24673 with signal 15
Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24673 exited. 3 remaining
Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24671 exited. 2 remaining
Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24672 exited. 1 remaining
Aug 19 13:24:48 hastur heartbeat: [24226]: info: hastur Heartbeat shutdown complete.


nyarにフェイルオーバーしたことを確認。


[ec2-user@nyar ~]$ tail /var/log/ha-log
Aug 19 13:21:35 nyar heartbeat: [22446]: info: Status update for node hastur: status active
harc[22590]: 2012/08/19_13:21:35 info: Running /etc/ha.d//rc.d/status status
Aug 19 13:21:36 nyar heartbeat: [22446]: info: remote resource transition completed.
Aug 19 13:24:46 nyar heartbeat: [22446]: info: Received shutdown notice from 'hastur'.
Aug 19 13:24:46 nyar heartbeat: [22446]: info: Resources being acquired from hastur.
Aug 19 13:24:46 nyar heartbeat: [22759]: info: acquire all HA resources (standby).
Aug 19 13:24:46 nyar heartbeat: [22760]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys nyar] to acquire.
ResourceManager[22785]: 2012/08/19_13:24:46 info: Acquiring resource group: hastur ENI
ResourceManager[22785]: 2012/08/19_13:24:46 info: Running /etc/ha.d/resource.d/ENI start
Aug 19 13:24:56 nyar heartbeat: [22759]: info: all HA resource acquisition completed (standby).
Aug 19 13:24:56 nyar heartbeat: [22446]: info: Standby resource acquisition done [all].
harc[22859]: 2012/08/19_13:24:56 info: Running /etc/ha.d//rc.d/status status
mach_down[22875]: 2012/08/19_13:24:56 info: Taking over resource group ENI
ResourceManager[22901]: 2012/08/19_13:24:56 info: Acquiring resource group: hastur ENI
ResourceManager[22901]: 2012/08/19_13:24:56 info: Running /etc/ha.d/resource.d/ENI start
mach_down[22875]: 2012/08/19_13:24:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[22875]: 2012/08/19_13:24:56 info: mach_down takeover complete for node hastur.
Aug 19 13:24:56 nyar heartbeat: [22446]: info: mach_down takeover complete.
Aug 19 13:25:17 nyar heartbeat: [22446]: WARN: node hastur: is dead
Aug 19 13:25:17 nyar heartbeat: [22446]: info: Dead node hastur gave up resources.
Aug 19 13:25:17 nyar heartbeat: [22446]: info: Link hastur:eth0 dead.
[ec2-user@nyar ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.170/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d7/64 scope link
valid_lft forever preferred_lft forever
8: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever


cthulhuからはnyarに接続される。


[ec2-user@cthulhu ~]$ mysql -h 10.0.0.150 -u scott -ptiger test -e "select * from foo"

              • +
bar
              • +
nyar
              • +