Sharing a public IP prefix between metal servers in the same project

Overview

If servers in your project are participating in a high-availability schema where a virtual IP is passed around between them for failover purposes, all servers plus the VIP need to be on the same IP prefix. This is accomplished by adding a new VLAN tagged subinterface to the first link-aggregate interface, configuring an IP from the shared prefix on it and adjusting the routing table to switch the default route to the new subinterface. The exising native IP address (/30 prefix) can stay configured but will no longer be used if no routes are configured for that subneti, it is abandoned so to speak by not being used anymore.

Linux (Debian/Ubuntu and derivatives using netplan)

Step 1

  • record existing configuration (shown for two servers)

Server A

ip link show

ip route show

[output]

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens4059f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 52:37:af:27:71:cb brd ff:ff:ff:ff:ff:ff permaddr 90:5a:08:be:ef:0a
    altname enp11s0f0np0
3: ens4059f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 52:37:af:27:71:cb brd ff:ff:ff:ff:ff:ff permaddr 90:5a:08:be:ef:0b
    altname enp11s0f1np1
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:37:af:27:71:cb brd ff:ff:ff:ff:ff:ff
    inet 203.0.113.38/30 brd 192.41.2.39 scope global bond0
       valid_lft forever preferred_lft forever
root@ServerA:~# ip route sh
default via 203.0.113.37 dev bond0 proto static
203.0.113.36/30 dev bond0 proto kernel scope link src 203.0.113.38

Server B

ip link show

ip route show

[output]

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens4059f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 52:37:af:72:17:bc brd ff:ff:ff:ff:ff:ff permaddr 90:5a:08:eb:fe:0a
    altname enp11s0f0np0
3: ens4059f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 52:37:af:72:17:bc brd ff:ff:ff:ff:ff:ff permaddr 90:5a:08:eb:fe:0b
    altname enp11s0f1np1
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:37:af:72:17:bc brd ff:ff:ff:ff:ff:ff
    inet 203.0.113.42/30 brd 192.41.2.39 scope global bond0
       valid_lft forever preferred_lft forever
root@ServerB:~# ip route sh
default via 203.0.113.41 dev bond0 proto static
203.0.113.40/30 dev bond0 proto kernel scope link src 203.0.113.42

existing config file /etc/netplan/50-cloud-init.yaml shown for ServerA

network:
  version: 2
  renderer: networkd
  ethernets:
    # Slave interfaces - no IP configuration
    # These interfaces are dedicated to the bond
    ens4059f0np0:
      dhcp4: false
      dhcp6: false
    ens4059f1np1:
      dhcp4: false
      dhcp6: false
  bonds:
    # Bond master interface
    bond0:
      interfaces: [ens4059f0np0, ens4059f1np1]
      parameters:
        mode: 802.3ad
        mii-monitor-interval: 100
        transmit-hash-policy: layer3+4
        lacp-rate: fast
      addresses: [203.0.113.38/30]
      routes:
        - to: default
          via: 203.0.113.37
      nameservers:
        addresses: [1.1.1.1,8.8.8.8]

Step 2

  • add subinterface with VLAN tag and IP to each server

provided settings

  • VLAN tag 1300
  • IP prefix 192.0.2.104/29
  • default router 192.0.2.105
  • customer usable IPs: 192.0.2.106 through 192.0.3.110

Provisioning

  • add configuration stanza for vlans to end of file /etc/netplan/50-cloud-init.yaml on ServerA (no route yet)
  vlans:
    bond0.1300:
      id: 1300
      link: bond0
      addresses:
      - 192.0.2.106/24
  • add configuration stanza for vlans to end of file /etc/netplan/50-cloud-init.yaml on ServerB (no route yet)
  vlans:
    bond0.1300:
      id: 1300
      link: bond0
      addresses:
      - 192.0.2.107/24

Run

ServerA# netplan apply

Test connection between new tagged interfaces

from ServerA ping bond0.1300 IP of ServerB (192.0.2.107) and vice versa (192.0.2.106)

from ServerA and ServerB ping IP address of default gateway (192.0.2.105)

Do not proceed until this works.

Step 3

Adding specific routes or the default route using the new subinterfaces

  • adding a specific route on root shell (for testing) Let's say you are logging on from your corporate office that is on the network 198.51.100.0/24 to the new IP of the server, it would then need a route back to your office using the subinterface's default router

ServerA # ip route add 198.51.100.0/24 via 192.0.2.105

  • ping an IP address on your corprate network

ServerA # ping 198.51.100.10

  • from your corporate network ping ServerA's IP

CorporatePC# ping 192.0.2.106

If this all works, you can now decide to remove the default route pointing to the original /30 prefix default gateway and move it to the new default gateway. Note this should be done from the KVM console or a session that does not depend on the default route for network connectivity (e.g. you are logging onto ServerA from ServerB over the VLAN 1300 IP network).

ServerA# ip route delete default
ServerA# ip route add default via 192.0.2.105
  • make new default route persistent in file /etc/netplan/50-cloud-init.yaml on ServerA

remove these three lines under the bond0 stanza

      routes:
        - to: default
          via: 203.0.113.37

add three lines under the bond0.1300 stanza

      routes:
        - to: default
          via: 192.0.2.105

Run

ServerA# netplan apply

  • You can now also remove the specific route that was added before if it is covered by the default route

ServerA # ip route delete 198.51.100.0/24 via 192.0.2.105

  • Repeat the route test and adding the defaut route on ServerB

Step 4

Virtual IP on one server at a time

  • on the server that is actively handling traffic for the virtual IP (e.g. 192.0.2.110), configure it as a secondary IP address on the loopback interface

ServerA # ip add add 192.0.2.110/32 dev lo

  • test by pinging the VIP address from another address

  • make sure to bind your payload services to that IP address, e.g. for a web server on port tcp/443 you would need to see a socket bound to all local IP addresses or to the specific VIP address, like in one of the following three lines

ServerA # netstat -rn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 192.0.2.110:443         0.0.0.0:*               LISTEN
tcp6       0      0 :::443                  :::*                    LISTEN
[...]
  • to move the VIP from one server to another, delete it from ServerA and add it to ServerB

Stop services on ServerA

ServerA # ip add delete 192.0.2.110/32 dev lo

ServerB # ip add add 192.0.2.110/32 dev lo

Start services bound to the VIP address on

The network will change ARP resolution to follow, the traffic will now be handled at ServerB.

Automatic Virtual IP failover

Bu cavab sizə kömək etdi? 0 istifadəçi bunu faydalı hesab edir (0 səs)