attach-eniコマンドを使ったフェイルオーバー

https://bitbucket.org/winebarrel/attach-eni

サーバ

  • cthulhu(クライアント): 10.0.0.172
  • hastur: 10.0.0.171
  • nyar: 10.0.0.170
  • ENI: 10.0.0.150

事前準備

  • haster/nyarにattach-eniをインストールしておく
  • 新規に作成したENIとhuster/nyarのeth0で『Source/Dest. Check』を『disabled』にしておく
  • すべてのサーバでrp_fileterを無効化


sudo sysctl -w net.ipv4.conf.default.rp_filter=0
sudo sysctl -w net.ipv4.conf.all.rp_filter=0
sudo sysctl -w net.ipv4.conf.lo.rp_filter=0
sudo sysctl -w net.ipv4.conf.eth0.rp_filter=0

  • hastur/nyarで/etc/sysconfig/network-scripts/ifcfg-eth1を作成
DEVICE=eth1
BOOTPROTO=dhcp
ONBOOT=no
  • 対向サーバへのルートをeth0に固定


hastur> sudo route add -host 10.0.0.170 dev eth0

nyar> sudo route add -host 10.0.0.171 dev eth0

  • default gwのNICをeth0に固定
    • /etc/sysconfig/network
...
GATEWAYDEV=eth0

動作確認

まずhasturにENIをさしておく。


[ec2-user@hastur ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.171/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d8/64 scope link
valid_lft forever preferred_lft forever
6: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever


cthulhuからpingを飛ばしてみる。


[ec2-user@cthulhu ~]$ ping 10.0.0.150 | awk '{print strftime("%X"), $0}'
08:56:29 AM PING 10.0.0.150 (10.0.0.150) 56(84) bytes of data.
08:56:29 AM 64 bytes from 10.0.0.150: icmp_seq=1 ttl=64 time=0.367 ms
08:56:30 AM 64 bytes from 10.0.0.150: icmp_seq=2 ttl=64 time=0.408 ms
08:56:31 AM 64 bytes from 10.0.0.150: icmp_seq=3 ttl=64 time=0.430 ms


nyarにENIを刺してみる。


[ec2-user@nyar ~]$ sudo /usr/sbin/attach-eni -n eni-2d33af44
[ec2-user@nyar ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d7 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.170/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d7/64 scope link
valid_lft forever preferred_lft forever
6: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever


ダウンタイムは10sくらい。


08:57:53 AM 64 bytes from 10.0.0.150: icmp_seq=41 ttl=64 time=0.402 ms
08:57:54 AM 64 bytes from 10.0.0.150: icmp_seq=42 ttl=64 time=0.415 ms
08:57:55 AM 64 bytes from 10.0.0.150: icmp_seq=43 ttl=64 time=0.486 ms
08:58:09 AM 64 bytes from 10.0.0.150: icmp_seq=57 ttl=64 time=0.306 ms
08:58:10 AM 64 bytes from 10.0.0.150: icmp_seq=58 ttl=64 time=0.374 ms
08:58:11 AM 64 bytes from 10.0.0.150: icmp_seq=59 ttl=64 time=0.301 ms


hasturにフェイルバック。


[ec2-user@hastur ~]$ sudo /usr/sbin/attach-eni -n eni-2d33af44
[ec2-user@hastur ~]$ ip addr show
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:94:d0:d8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.171/24 brd 10.0.0.255 scope global eth0
inet6 fe80::9d:4eff:fe94:d0d8/64 scope link
valid_lft forever preferred_lft forever
7: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:9d:4e:93:95:bd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.150/24 brd 10.0.0.255 scope global eth1
inet6 fe80::9d:4eff:fe93:95bd/64 scope link
valid_lft forever preferred_lft forever

09:00:21 AM 64 bytes from 10.0.0.150: icmp_seq=189 ttl=64 time=0.316 ms
09:00:22 AM 64 bytes from 10.0.0.150: icmp_seq=190 ttl=64 time=0.333 ms
09:00:23 AM 64 bytes from 10.0.0.150: icmp_seq=191 ttl=64 time=0.385 ms
09:00:36 AM 64 bytes from 10.0.0.150: icmp_seq=204 ttl=64 time=0.359 ms
09:00:37 AM 64 bytes from 10.0.0.150: icmp_seq=205 ttl=64 time=0.459 ms
09:00:38 AM 64 bytes from 10.0.0.150: icmp_seq=206 ttl=64 time=0.402 ms

  • 同じセグメントにNICを刺すので仮想IPより癖がある
    • Src/Distチェックとrp_filterを無効化しなくてもすむとよいのだけど
  • スプリットブレインがないのはいいと思う