Heartbeatとattach-eniコマンドを使った冗長化 その２: L7のヘルスチェック

L7のヘルスチェックを追加してみる。

とりあえず、hasterとnyarにmysql-serverをインストール。
以下のヘルスチェック用デーモンを作成。

/usr/local/bin/mysql-pingd

#!/bin/bash
INTERVAL=5

while true; do
  /usr/bin/mysqladmin ping --connect_timeout=3

  if [ $? -ne 0 ]; then
    /usr/lib64/heartbeat/heartbeat -k
  fi

  sleep $INTERVAL
done

ha.cfに以下の行を追加。

/etc/ha.d/ha.cf

...
respawn root /usr/local/bin/mysql-pingd

動作確認

ENIをhasturにアタッチして、hasturとnyarでheartbeatを起動。
ヘルスチェック用デーモンがheartbeatによって起動される。

Aug 19 13:21:35 hastur heartbeat: [24226]: info: Starting child client "/usr/local/bin/mysql-pingd" (0,0)

Aug 19 13:21:35 hastur heartbeat: [24693]: info: Starting "/usr/local/bin/mysql-pingd" as uid 0  gid 0 (pid 24693)

root     24226  0.0  0.4  50020  7064 ?        SLs  13:14   0:00 heartbeat: master control process

root     24671  0.0  0.4  49968  7012 ?        SL   13:21   0:00  \_ heartbeat: FIFO reader

root     24672  0.0  0.4  49964  7008 ?        SL   13:21   0:00  \_ heartbeat: write: ucast eth0

root     24673  0.0  0.4  49964  7008 ?        SL   13:21   0:00  \_ heartbeat: read: ucast eth0

root     24693  0.0  0.0   9228  1240 ?        S    13:21   0:00  \_ /bin/bash /usr/local/bin/mysql-pingd

root     24711  0.0  0.0   4104   316 ?        S    13:21   0:00      \_ sleep 5

初期状態ではcthulhuからhasturに接続される。

[ec2-user@cthulhu ~]$ mysql -h 10.0.0.150 -u scott -ptiger test -e "select * from foo"


















+










    

    
 bar    
    


















+










    

    
 hastur 
    


















+

hasterのmysqlを停止してみる。

[ec2-user@hastur ~]$ sudo /etc/init.d/mysqld stop

Stopping mysqld:                                           [  OK  ]

[ec2-user@hastur ~]$ tail /var/log/ha-log

Aug 19 13:21:36 hastur heartbeat: [24226]: info: remote resource transition completed.

Aug 19 13:21:36 hastur heartbeat: [24226]: info: Initial resource acquisition complete (T_RESOURCES(us))

Aug 19 13:24:46 hastur heartbeat: [24226]: info: Heartbeat shutdown in progress. (24226)

Aug 19 13:24:46 hastur heartbeat: [24893]: info: Giving up all HA resources.

ResourceManager[24906]: 2012/08/19_13:24:46 info: Releasing resource group: hastur ENI

ResourceManager[24906]: 2012/08/19_13:24:46 info: Running /etc/ha.d/resource.d/ENI  stop

Aug 19 13:24:46 hastur heartbeat: [24893]: info: All HA resources relinquished.

Aug 19 13:24:47 hastur heartbeat: [24226]: info: killing /usr/local/bin/mysql-pingd process group 24693 with signal 15

Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBFIFO process 24671 with signal 15

Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBWRITE process 24672 with signal 15

Aug 19 13:24:48 hastur heartbeat: [24226]: info: killing HBREAD process 24673 with signal 15

Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24673 exited. 3 remaining

Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24671 exited. 2 remaining

Aug 19 13:24:48 hastur heartbeat: [24226]: info: Core process 24672 exited. 1 remaining

Aug 19 13:24:48 hastur heartbeat: [24226]: info: hastur Heartbeat shutdown complete.

nyarにフェイルオーバーしたことを確認。

[ec2-user@nyar ~]$  tail /var/log/ha-log

Aug 19 13:21:35 nyar heartbeat: [22446]: info: Status update for node hastur: status active

harc[22590]:    2012/08/19_13:21:35 info: Running /etc/ha.d//rc.d/status status

Aug 19 13:21:36 nyar heartbeat: [22446]: info: remote resource transition completed.

Aug 19 13:24:46 nyar heartbeat: [22446]: info: Received shutdown notice from 'hastur'.

Aug 19 13:24:46 nyar heartbeat: [22446]: info: Resources being acquired from hastur.

Aug 19 13:24:46 nyar heartbeat: [22759]: info: acquire all HA resources (standby).

Aug 19 13:24:46 nyar heartbeat: [22760]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys nyar] to acquire.

ResourceManager[22785]: 2012/08/19_13:24:46 info: Acquiring resource group: hastur ENI

ResourceManager[22785]: 2012/08/19_13:24:46 info: Running /etc/ha.d/resource.d/ENI  start

Aug 19 13:24:56 nyar heartbeat: [22759]: info: all HA resource acquisition completed (standby).

Aug 19 13:24:56 nyar heartbeat: [22446]: info: Standby resource acquisition done [all].

harc[22859]:    2012/08/19_13:24:56 info: Running /etc/ha.d//rc.d/status status

mach_down[22875]:       2012/08/19_13:24:56 info: Taking over resource group ENI

ResourceManager[22901]: 2012/08/19_13:24:56 info: Acquiring resource group: hastur ENI

ResourceManager[22901]: 2012/08/19_13:24:56 info: Running /etc/ha.d/resource.d/ENI  start

mach_down[22875]:       2012/08/19_13:24:56 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired

mach_down[22875]:       2012/08/19_13:24:56 info: mach_down takeover complete for node hastur.

Aug 19 13:24:56 nyar heartbeat: [22446]: info: mach_down takeover complete.

Aug 19 13:25:17 nyar heartbeat: [22446]: WARN: node hastur: is dead

Aug 19 13:25:17 nyar heartbeat: [22446]: info: Dead node hastur gave up resources.

Aug 19 13:25:17 nyar heartbeat: [22446]: info: Link hastur:eth0 dead.

[ec2-user@nyar ~]$ ip addr show

1: lo:  mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 02:9d:4e:94:d0:d7 brd ff:ff:ff:ff:ff:ff

    inet 10.0.0.170/24 brd 10.0.0.255 scope global eth0

    inet6 fe80::9d:4eff:fe94:d0d7/64 scope link

       valid_lft forever preferred_lft forever

8: eth1:  mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff

    inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1

    inet6 fe80::9d:4eff:fe93:95bd/64 scope link

       valid_lft forever preferred_lft forever

cthulhuからはnyarに接続される。

[ec2-user@cthulhu ~]$ mysql -h 10.0.0.150 -u scott -ptiger test -e "select * from foo"














+








    

    
 bar  
    














+








    

    
 nyar 
    














+