OpenStack Liberty Neutron Deployment (Part 2 Network Infrastructure)

The physical network

A cloud needs an „physical“ network using real „switching routers“ to transport data between the physical nodes.

The physical network has the following layout:

Physical set up

Physical set up

Each floating pool is implemented using one vlan on the physical network infrastructure. Each flat network requires also one vlan on the physical network. VXLAN requires also one vlan on the physical network.

VXLAN adds additional headers (IP and UDP) to packets. You must increase the mtu of the transport VXLAN vlan to a higher value, let’s take 1600. Do not follow the „recommendations“ to reduce the MTU of VMs using the DHCP server. Such a „recommendation“ only leads to big problems in a production environment — do not use a reduced MTU!

The configuration of the physical switching router could be (using a pseudo config language):

vlan 100 name Floating-Pool-1
vlan 101 name Floating-Pool-2
vlan 200 name Flat-Net-1
vlan 201 name Flat-Net-2
# set the mtu to 1600 for vlan 4000
vlan 4000 name vxlan mtu 1600
# do not use vlan 1 for untagged packets
vlan 4090 name native vlan
interface vlan 100
  description Floating-Network-1
  ip address
interface vlan 101
  description Floating-Network-2
  ip address
interface vlan 200
  description Flat-Network-1
  ip address
interface vlan 201
  description Flat-Network-2
  ip address
# a L3 interface for the vxlan vlan may be added
# the ports to the nodes (network and compute)
# They use the same config !
# just one link to each node - multiple links using LACP may also be used
interface port1
  description to-network-node
  mode trunk
  trunk native vlan 4090
  trunk vlan 100,101,200,201,4000,4090
  mtu 1600
interface port2
  description to-compute-node
  mode trunk
  trunk native vlan 4090
  trunk vlan 100,101,200,201,4000,4090
  mtu 1600

The network set up of the nodes

The basic network set up of compute and network nodes is exactly the same. On the nodes, a basic network configuration of the Operating system must be provided by the user. This configuration connects the transport network for VMs on the nodes to the physical network. The network setup of a node is shown in the drawing below:

Openvswitch Bridges on the Nodes

Openvswitch Bridges on the Nodes

The cool part of the network setup is, that only ONE physical interface (or a LACP bundle) is used to transport VM traffic. This interface is connected to br-uplink. If you read the Openstack manuals and believe in everything which is written there, you would end up using FOUR physical interfaces:

  • one physical interface for VXLAN transport. This is not true – VXLAN transport requires an IP address in the default network namespace of the operating system. We choose an Openvswitch internal port on the Openvswitch bridge br-uplink and attach an IP address to it. The default MTU must be set to a higher value. We use 1600.
  • one link to the physical network via br-vlan (or br-eth1 in the Openstack documentation) using a physical interface for the Openstack ML2 „vlan“ type driver. This is not true – we take an Openvswitch patch port as the uplink. This port connects br-vlan to be use by the Openstack ML2 mechanism driver to br-uplink.
  • two interfaces to map the two floating pools to two L3 agents, each using an Openvswitch bridge, which is connected to the physical network using a physical ethernet interface. This is not true – L3 agents are clever enough to avoid the usage of any additional bridges. And one L3 agent may manage multiple floating pools (or external networks if you are talking „Openstack“).

A part of the network configuration must be provided by the admin of the nodes during system startup. This configuration must be placed in the network configuration of the Operating system. The configuration depends on the Linux distribution.

The shell code for the network setup is (the management network is omitted):

# the bridge, which connects the nodes to the transport network
ovs-vsctl add-br br-uplink
# the bridge used by Openstack Neutron to connect vlans and flatdhcp networks
ovs-vsctl add-br br-vlan
# the integration bridge used by Openstack
ovs-vsctl add-br br-int
# add the uplink (with dot1q tags 101,102,...)
# we assume, that eth1 is the uplink interface
ip link set dev eth1 up
# set the mtu of the physical uplink to the switch
ip link set dev eth1 mtu 1600
# disable gro and lro !! on the uplink
ethtool -K eth1 gro off
ethtool -K eth1 lro off
# enable for intel NICs udp port hashing to distribute traffic to different queues
ethtool -N eth1 rx-flow-hash udp4 sdfn
ovs-vsctl add-port br-uplink eth1 -- set port eth1 vlan_mode=trunk trunk=100,101,200,201,4000
# patch ports between br-uplink and br-vlan
ovs-vsctl add-port br-vlan patch-to-uplink -- set Interface patch-to-uplink type=patch options:peer=patch-to-vlan
ovs-vsctl add-port br-uplink patch-to-vlan -- set Interface patch-to-vlan type=patch options:peer=patch-to-uplink
# !! on br-uplink the allowed vlan tags on the patch port from br-vlan must be filtered using Openflow rules
# !! if this is not done, there is a risk that vlans from the infrastructure may get mixed with local vlans
# !! of br-int, if the neutron Openvswitch agent fails to set up the vlan mapping on br-vlan or br-int
# create the Linux IP interface required for VXLAN transport
# this interface is attached to vlan 4000 of br-uplink
# XXX = Last octet of the VXLAN interface ip address of the node
ovs-vsctl add-port br-uplink l3vxlan tag=4000 -- set Interface l3vxlan type=internal
ip addr add 10.255.255.XXX/24 dev l3vxlan
ip link set dev l3vxlan up
# set the mtu of the logical vxlan interface
ip link set dev l3vxlan mtu 1600

Do not enable ip_forwarding on any of the nodes. This is not necessary, even if the Openstack documentation declares this as a prerequisite.

Neutron in Openstack Liberty does not require to use br-ex to get a functional L3 agent providing router services. The default setup in the Openstack documentation is using br-ex, but there are other ways to implement networking, e.g. the one shown in this article series.

Continue reading (part 3) the Openstack Neutron setup

Updated: 17/01/2021 — 14:48