Load-balancing & Service Discovery

This tutorial will guide you to perform load-balancing and service discovery across multiple Kubernetes clusters when using Cilium.

Prerequisites

You need to have a functioning Cluster Mesh setup, please follow the guide Setting up Cluster Mesh to set it up.

Load-balancing with Global Services

Establishing load-balancing between clusters is achieved by defining a Kubernetes service with identical name and namespace in each cluster and adding the annotation io.cilium/global-service: "true" to declare it global. Cilium will automatically perform load-balancing to pods in both clusters.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: rebel-base
  5.   annotations:
  6.     io.cilium/global-service: "true"
  7. spec:
  8.   type: ClusterIP
  9.   ports:
  10.   - port: 80
  11.   selector:
  12.     name: rebel-base

Disabling Global Service Sharing

By default, a Global Service will load-balance across backends in multiple clusters. This implicitly configures io.cilium/shared-service: "true". To prevent service backends from being shared to other clusters, this option should be disabled.

Below example will expose remote endpoint without sharing local endpoints.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: rebel-base
  5.   annotations:
  6.     io.cilium/global-service: "true"
  7.     io.cilium/shared-service: "false"
  8. spec:
  9.   type: ClusterIP
  10.   ports:
  11.   - port: 80
  12.   selector:
  13.     name: rebel-base

Deploying a Simple Example Service

  1. In cluster 1, deploy:

    1. kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/kubernetes/clustermesh/global-service-example/rebel-base-global-shared.yaml
    2. kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml

  2. In cluster 2, deploy:

    1. kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/kubernetes/clustermesh/global-service-example/rebel-base-global-shared.yaml
    2. kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.12/examples/kubernetes/clustermesh/global-service-example/cluster2.yaml

  3. From either cluster, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in both clusters.

  4. In cluster 1, add io.cilium/shared-service="false" to existing global service

    1. kubectl annotate service rebel-base io.cilium/shared-service="false" --overwrite

  5. From cluster 1, access the global service one more time:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will still see replies from pods in both clusters.

  6. From cluster 2, access the global service again:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods only from cluster 2, as the global service in cluster 1 is no longer shared.

  7. In cluster 1, remove io.cilium/shared-service annotation of existing global service

    1. kubectl annotate service rebel-base io.cilium/shared-service-

  8. From either cluster, access the global service:

    1. kubectl exec -ti deployment/x-wing -- curl rebel-base

    You will see replies from pods in both clusters again.