Multi-cluster Ingress

Users can use MultiClusterIngress API provided in Karmada to import external traffic to services in the member clusters.

Note: To use this feature, the Kubernetes version of the member cluster must be v1.21 or later.

Prerequisites

Karmada has been installed

We can install Karmada by referring to Quick Start, or directly run hack/local-up-karmada.sh script which is also used to run our E2E cases.

Cluster Network

Currently, we need to use the Multi-cluster Service feature to import external traffic.

So we need to ensure that the container networks between the host cluster and member clusters are connected. The host cluster indicates the cluster where the Karmada Control Plane is deployed.

  • If you use the hack/local-up-karmada.sh script to deploy Karmada, Karmada will have three member clusters, and the container networks between the host cluster, member1 and member2 are connected.
  • You can use Submariner or other related open source projects to connected networks between clusters.

Note: In order to prevent routing conflicts, Pod and Service CIDRs of clusters need non-overlapping.

Example

Step 1: Deploy ingress-nginx on the host cluster

We use multi-cluster-ingress-nginx as the demo for demonstration. We’ve made some changes based on the latest version(controller-v1.1.1) of ingress-nginx.

Download code

  1. # for HTTPS
  2. git clone https://github.com/karmada-io/multi-cluster-ingress-nginx.git
  3. # for SSH
  4. git clone git@github.com:karmada-io/multi-cluster-ingress-nginx.git

Build and deploy ingress-nginx

Using the existing karmada-host kind cluster to build and deploy the ingress controller.

  1. export KUBECONFIG=~/.kube/karmada.config
  2. export KIND_CLUSTER_NAME=karmada-host
  3. kubectl config use-context karmada-host
  4. cd multi-cluster-ingress-nginx
  5. make dev-env

Apply kubeconfig secret

Create a secret that contains the karmada-apiserver authentication credential:

  1. # get the 'karmada-apiserver' kubeconfig information and direct it to file /tmp/kubeconfig.yaml
  2. kubectl -n karmada-system get secret kubeconfig --template={{.data.kubeconfig}} | base64 -d > /tmp/kubeconfig.yaml
  3. # create secret with name 'kubeconfig' from file /tmp/kubeconfig.yaml
  4. kubectl -n ingress-nginx create secret generic kubeconfig --from-file=kubeconfig=/tmp/kubeconfig.yaml

Edit ingress-nginx-controller deployment

We want nginx-ingress-controller to access karmada-apiserver to listen to changes in resources(such as multiclusteringress, endpointslices, and service). Therefore, we need to mount the authentication credential of karmada-apiserver to the nginx-ingress-controller.

  1. kubectl -n ingress-nginx edit deployment ingress-nginx-controller

Edit as follows:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   ...
  5. spec:
  6.   #...
  7.   template:
  8.     spec:
  9.       containers:
  10.       - args:
  11.         - /nginx-ingress-controller
  12.         - --karmada-kubeconfig=/etc/kubeconfig  # new line
  13.         #...
  14.         volumeMounts:
  15.         #...
  16.         - mountPath: /etc/kubeconfig            # new line
  17.           name: kubeconfig                      # new line
  18.           subPath: kubeconfig                   # new line
  19.       volumes:
  20.       #...
  21.       - name: kubeconfig                        # new line
  22.         secret:                                 # new line
  23.           secretName: kubeconfig                # new line

Step 2: Use the MCS feature to discovery service

Install ServiceExport and ServiceImport CRDs

Refer to here.

Deploy web on member1 cluster

deploy.yaml:

unfold me to see the yaml

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: web
  5. spec:
  6.   replicas: 1
  7.   selector:
  8.     matchLabels:
  9.       app: web
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: web
  14.     spec:
  15.       containers:
  16.       - name: hello-app
  17.         image: gcr.io/google-samples/hello-app:1.0
  18.         ports:
  19.         - containerPort: 8080
  20.           protocol: TCP
  21. ---      
  22. apiVersion: v1
  23. kind: Service
  24. metadata:
  25.   name: web
  26. spec:
  27.   ports:
  28.   - port: 81
  29.     targetPort: 8080
  30.   selector:
  31.     app: web
  32. ---
  33. apiVersion: policy.karmada.io/v1alpha1
  34. kind: PropagationPolicy
  35. metadata:
  36.   name: mci-workload
  37. spec:
  38.   resourceSelectors:
  39.     - apiVersion: apps/v1
  40.       kind: Deployment
  41.       name: web
  42.     - apiVersion: v1
  43.       kind: Service
  44.       name: web
  45.   placement:
  46.     clusterAffinity:
  47.       clusterNames:
  48.         - member1
  1. kubectl --context karmada-apiserver apply -f deploy.yaml

Export web service from member1 cluster

service_export.yaml:

unfold me to see the yaml

  1. apiVersion: multicluster.x-k8s.io/v1alpha1
  2. kind: ServiceExport
  3. metadata:
  4.   name: web
  5. ---
  6. apiVersion: policy.karmada.io/v1alpha1
  7. kind: PropagationPolicy
  8. metadata:
  9.   name: web-export-policy
  10. spec:
  11.   resourceSelectors:
  12.     - apiVersion: multicluster.x-k8s.io/v1alpha1
  13.       kind: ServiceExport
  14.       name: web
  15.   placement:
  16.     clusterAffinity:
  17.       clusterNames:
  18.         - member1
  1. kubectl --context karmada-apiserver apply -f service_export.yaml

Import web service to member2 cluster

service_import.yaml:

unfold me to see the yaml

  1. apiVersion: multicluster.x-k8s.io/v1alpha1
  2. kind: ServiceImport
  3. metadata:
  4.   name: web
  5. spec:
  6.   type: ClusterSetIP
  7.   ports:
  8.   - port: 81
  9.     protocol: TCP
  10. ---
  11. apiVersion: policy.karmada.io/v1alpha1
  12. kind: PropagationPolicy
  13. metadata:
  14.   name: web-import-policy
  15. spec:
  16.   resourceSelectors:
  17.     - apiVersion: multicluster.x-k8s.io/v1alpha1
  18.       kind: ServiceImport
  19.       name: web
  20.   placement:
  21.     clusterAffinity:
  22.       clusterNames:
  23.         - member2
  1. kubectl --context karmada-apiserver apply -f service_import.yaml

Step 3: Deploy multiclusteringress on karmada-controlplane

mci-web.yaml:

unfold me to see the yaml

  1. apiVersion: networking.karmada.io/v1alpha1
  2. kind: MultiClusterIngress
  3. metadata:
  4.   name: demo-localhost
  5.   namespace: default
  6. spec:
  7.   ingressClassName: nginx
  8.   rules:
  9.   - host: demo.localdev.me
  10.     http:
  11.       paths:
  12.       - backend:
  13.           service:
  14.             name: web
  15.             port:
  16.               number: 81
  17.         path: /web
  18.         pathType: Prefix
  1. kubectl --context karmada-apiserver apply -f mci-web.yaml

Step 4: Local testing

Let’s forward a local port to the ingress controller:

  1. kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80

At this point, if you access http://demo.localdev.me:8080/web/, you should see an HTML page telling you:

  1. Hello, world!
  2. Version: 1.0.0
  3. Hostname: web-xxx-xxx