Configure BGP peering

We have configured Calico to distribute routing information over the Border Gateway Protocol (BGP). This scalable protocol powers routing on the global public Internet.

In many on-premise data centers, each server connects to a top-of-rack (ToR) router operating at the IP layer (layer 3). In that situation, we would need to peer each node with its corresponding ToR router, so that the ToR learns routes to the containers. That configuration is beyond the scope of this guide.

Since we are running in an AWS VPC within a single subnet, the hosts have ethernet (layer 2) connectivity with one another, meaning there are no routers between them. Thus, they can peer directly with each other.

On one of the nodes in your cluster where you have calicoctl installed, check the status.

  1. sudo calicoctl node status

Result

  1. Calico process is running.
  3. IPv4 BGP status
  4. +---------------+-------------------+-------+----------+-------------+
  5. | PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
  6. +---------------+-------------------+-------+----------+-------------+
  7. | 172.31.40.217 | node-to-node mesh | up    | 17:38:47 | Established |
  8. | 172.31.40.30  | node-to-node mesh | up    | 17:40:09 | Established |
  9. | 172.31.45.29  | node-to-node mesh | up    | 17:40:20 | Established |
  10. | 172.31.37.123 | node-to-node mesh | up    | 17:40:29 | Established |
  11. +---------------+-------------------+-------+----------+-------------+
  13. IPv6 BGP status
  14. No IPv6 peers found.

Alternatively, you can create a CalicoNodeStatus resource to get BGP session status for the node.

Notice there are four BGP sessions, one to each other node in the cluster. In a small cluster, this works well and is highly resilient. However, the total number of BGP sessions scales as the square of the number of nodes, and in a large cluster this creates a lot of overhead.

In this lab we will configure a fixed number of route reflectors. Route reflectors announce their own routes and the routes they receive from other peers. This means nodes only need to peer with the route reflectors to get all the routes in the cluster. This peering arrangement means that the number of BGP sessions scales linearly with the number of nodes.

Choose and label nodes

We will establish three route reflectors, which means we avoid a single point of failure even if we take down a route reflector node for maintenance. In a five node cluster that means that only one BGP session is not needed, since the two non-reflector nodes don’t need to peer with one another, but it will save lots of overhead in a large cluster.

Choose three nodes and perform the following for each of them.

Save the node YAML.

  1. calicoctl get node <node name> -o yaml --export > node.yaml

Edit the YAML to add

  1. metadata:
  2.   labels:
  3.     calico-route-reflector: ''
  4. spec:
  5.   bgp:
  6.     routeReflectorClusterID: 224.0.0.1

Reapply the YAML

  1. calicoctl apply -f node.yaml

Configure peering

Configure all non-reflector nodes to peer with all route reflectors

  1. calicoctl apply -f - <<EOF
  2. kind: BGPPeer
  3. apiVersion: projectcalico.org/v3
  4. metadata:
  5.   name: peer-to-rrs
  6. spec:
  7.   nodeSelector: "!has(calico-route-reflector)"
  8.   peerSelector: has(calico-route-reflector)
  9. EOF

Configure all route reflectors to peer with each other

  1. calicoctl apply -f - <<EOF
  2. kind: BGPPeer
  3. apiVersion: projectcalico.org/v3
  4. metadata:
  5.   name: rrs-to-rrs
  6. spec:
  7.   nodeSelector: has(calico-route-reflector)
  8.   peerSelector: has(calico-route-reflector)
  9. EOF

Disable the node-to-node mesh

  1. calicoctl create -f - <<EOF
  2.  apiVersion: projectcalico.org/v3
  3.  kind: BGPConfiguration
  4.  metadata:
  5.    name: default
  6.  spec:
  7.    nodeToNodeMeshEnabled: false
  8.    asNumber: 64512
  9. EOF

On a non-reflector node, you should now see only three peerings.

  1. sudo calicoctl node status

Result

  1. Calico process is running.
  3. IPv4 BGP status
  4. +---------------+---------------+-------+----------+-------------+
  5. | PEER ADDRESS  |   PEER TYPE   | STATE |  SINCE   |    INFO     |
  6. +---------------+---------------+-------+----------+-------------+
  7. | 172.31.37.123 | node specific | up    | 21:52:57 | Established |
  8. | 172.31.40.217 | node specific | up    | 21:52:57 | Established |
  9. | 172.31.42.47  | node specific | up    | 21:52:57 | Established |
  10. +---------------+---------------+-------+----------+-------------+
  12. IPv6 BGP status
  13. No IPv6 peers found.

Alternatively, you can create a CalicoNodeStatus resource to get BGP session status for the node.

Next

Test networking