MatrixOne Distributed Cluster Upgrade

This document will introduce how to Rolling upgrade or Reinstall upgrade MatrixOne clusters.

The upgraded environment introduced in this document will be based on the environment of MatrixOne Distributed Cluster Deployment.

Rolling upgrade

Rolling upgrade is an online upgrade method; the MatrixOne cluster completes the software upgrade while ensuring that some or all services are available.

According to the introduction in MatrixOne Distributed Cluster Deployment, the overall installation of MatrixOne is based on Kubernetes and MatrixOne Operator. Therefore, the rolling upgrade process is to realize automatic version updates by dynamically modifying the MatrixOne image version number in MatrixOne Operator.

Steps

  1. Execute the following command on the terminal on the master0 node to enter the interface for dynamically modifying the yaml configuration file the operator uses.

    1. mo_ns="mo-hn" #the namespace of matrixone cluster
    2. mo_cluster_name="mo" # The cluster name of matrixone, generally mo, is specified according to the name in the YAML file of the matrixonecluster object during deployment or can be confirmed by kubectl get matrixonecluster -n${mo_ns}
    3. # mo-hn and mo have been set in the mo.yaml file of the installation and deployment
    4. kubectl edit matrixonecluster ${mo_cluster_name} -n${mo_ns}

  2. After entering edit mode, modify the value of spec.version; the parameters are as below:

    • ${TAG}: corresponds to the image tag of Matrixone on dockerhub, for example: nightly-f0d52530

    • ${REPO_URL}: public mirror repository for Matrixone, the default is matrixorigin/matrixone. If the target version does not exist in the public mirror warehouse of MatrixOne, you also need to modify the URL of the mirror warehouse to the existing warehouse:

    image-20230407094237806

    image-20230407094251938

  3. After completing the modification, press :wq to save. MatrixOne Operator will automatically pull the new version of the image and restart the component services, including Log Service, TN, and CN. You can also observe its running status through the following commands.

    1. watch -e "kubectl get pod -n${mo_ns}"
    1. NAME                                 READY   STATUS    RESTARTS      AGE
    2. matrixone-operator-f8496ff5c-fp6zm   1/1     Running   0             24h
    3. mo-tn-0                              1/1     Running   1 (51s ago)   18h
    4. mo-log-0                             1/1     Running   0             18h
    5. mo-log-1                             1/1     Running   1 (5s ago)    18h
    6. mo-log-2                             1/1     Running   1 (53s ago)   18h
    7. mo-tp-cn-0                           1/1     Running   1 (53s ago)   18h

    If an error, crashbackoff, etc., occurs, you can further troubleshoot the problem by viewing the component log.

    1. #pod_name is the name of the pod, such as mo-tn-0, mo-tp-cn-0
    2. pod_name=mo-tn-0
    3. kubectl logs ${pod_name} -nmo-hn > /tmp/tn.log
    4. vim /tmp/tn.log

  4. After the Restart of the components in the MatrixOne cluster is completed, you can use the MySQL Client to connect to the cluster. The upgrade is successful if the connection is successful and the user data is complete.

    1. # Connect to the MySQL server using the 'mysql' command line tool
    2. # Use 'kubectl get svc/mo-tp-cn -n mo-hn -o jsonpath='{.spec.clusterIP}' ' to get the cluster IP address of the service in the Kubernetes cluster
    3. # The '-h' parameter specifies the hostname or IP address of the MySQL service
    4. # The '-P' parameter specifies the port number of the MySQL service, here is 6001
    5. # '-uroot' means log in with root user
    6. # '-p111' means the initial password is 111
    7. mysql -h $(kubectl get svc/mo-tp-cn -n mo-hn -o jsonpath='{.spec.clusterIP}') -P 6001 -uroot -p111
    8. root@master0 ~]# mysql -h $(kubectl get svc/mo-tp-cn -n mo-hn -o jsonpath='{.spec.clusterIP}') -P 6001 -uroot -p111
    9. Welcome to the MariaDB monitor.  Commands end with ; or \g.
    10. Your MySQL connection id is 1005
    11. Server version: 8.0.30-MatrixOne-v1.1.0 MatrixOne
    13. Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
    15. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    16. MySQL [(none)]> show databases;
    17. +--------------------+
    18. | Database           |
    19. +--------------------+
    20. | mo_task            |
    21. | information_schema |
    22. | mysql              |
    23. | system_metrics     |
    24. | system             |
    25. | test               |
    26. | mo_catalog         |
    27. +--------------------+
    28. 7 rows in set (0.01 sec)

    Info

    The login account in the above code snippet is the initial account; please change the initial password after logging in to MatrixOne; see Password Management.

  5. The rolling update may be suspended due to incorrect configuration (such as specifying a non-existing version when upgrading). At this point, you can re-modify the dynamic configuration of the operator, reset the version number, roll back the changes, and the failed Pods will be re-updated.

  6. You can check the version number of the current MatrixOne deployment with the following command:

    1. [root@master0 matrixone-operator]# kubectl get matrixoneclusters -n mo-hn -o yaml | grep version
    2.         {"apiVersion":"core.matrixorigin.io/v1alpha1","kind":"MatrixOneCluster","metadata":{"annotations":{},"name":"mo","namespace":"mo-hn"},"spec":{"tn":{"cacheVolume":{"size":"5Gi","storageClassName":"local-path"},"config":"[dn.Txn.Storage]\nbackend = \"TAE\"\nlog-backend = \"logservice\"\n[dn.Ckp]\nflush-interval = \"60s\"\nmin-count = 100\nscan-interval = \"5s\"\nincremental-interval = \"60s\"\nglobal-interval = \"100000s\"\n[log]\nlevel = \"error\"\nformat = \"json\"\nmax-size = 512\n","replicas":1,"resources":{"limits":{"cpu":"200m","memory":"1Gi"},"requests":{"cpu":"100m","memory":"500Mi"}}},"imagePullPolicy":"IfNotPresent","imageRepository":"matrixorigin/matrixone","logService":{"config":"[log]\nlevel = \"error\"\nformat = \"json\"\nmax-size = 512\n","pvcRetentionPolicy":"Retain","replicas":3,"resources":{"limits":{"cpu":"200m","memory":"1Gi"},"requests":{"cpu":"100m","memory":"500Mi"}},"sharedStorage":{"s3":{"endpoint":"http://minio.mostorage:9000","path":"minio-mo","secretRef":{"name":"minio"},"type":"minio"}},"volume":{"size":"1Gi"}},"tp":{"cacheVolume":{"size":"5Gi","storageClassName":"local-path"},"config":"[cn.Engine]\ntype = \"distributed-tae\"\n[log]\nlevel = \"debug\"\nformat = \"json\"\nmax-size = 512\n","nodePort":31429,"replicas":1,"resources":{"limits":{"cpu":"200m","memory":"2Gi"},"requests":{"cpu":"100m","memory":"500Mi"}},"serviceType":"NodePort"},"version":"nightly-54b5e8c"}}
    3.     version: nightly-54b5e8c

Reinstall and upgrade

Reinstalling and upgrading mean all MatrixOne clusters will be deleted and the data discarded and reinstalled.

Applicable scene:

  • no need for old data
  • The versions before and after the upgrade are not compatible with each other due to special reasons

Note

Before the operation, please ensure the data has been backed up (see modump backup tool) and the business knows that the database has stopped.

Steps

1. Delete the old version cluster

In master0, the old version cluster can be deleted in any of the following ways:

  1. # Method 1: Delete through the YAML file of the mo cluster during deployment, for example:
  2. kubectl delete -f /root/deploy/mo.yaml
  3. # Method 2: By deleting the matrixonecluster object, where mo is the name
  4. kubectl delete matrixonecluster.core.matrixorigin.io mo -nmo-hn

Confirm that mo-related resources have been deleted by checking the pod status:

  1. kubectl get pod -nmo-hn

In addition, if the PVC used by mo has not been deleted, use the following command to delete it manually:

  1. kubectl get pvc -nmo-hn
  2. # For example, the PVC used by the log service has not been deleted, delete it manually
  3. kubectl delete pvc mo-data-mo-log-0 -nmo-hn
  4. kubectl delete pvc mo-data-mo-log-1 -nmo-hn
  5. kubectl delete pvc mo-data-mo-log-2 -nmo-hn

2. Empty bucket data

On the MinIO control page, delete the data in the bucket used by MinIO used by MO, including subdirectories such as mo-data, etc.

image-minio-delete-bucket

Or operate through the MinIO client mc:

  1. mc rb --force minio/minio-mo/data/
  2. mc rb --force minio/minio-mo/etl

In addition, if you do not want to delete old data, you can create a new MinIO bucket and specify a unique bucket name in the YAML file for deploying the MatrixOne cluster.

image-minio-new-bucket

3. Deploy the new version cluster

Edit the yaml file that defines the MO cluster, refer to the Rolling Upgrade chapter, modify the .spec.version field to the latest version, and redeploy the MatrixOne cluster:

  1. vi mo.yaml
  2. # content omitted
  3. ...
  4. kubectl apply -f mo.yaml

4. Check if the upgrade is successful

Please check if MatrixOne has started successfully with the following command.

As shown in the following code example, when the Log Service, TN, and CN are all running normally, the MatrixOne cluster starts successfully. Connecting through the MySQL Client can also check if the database functions correctly.

  1. [root@master0 ~]# kubectl get pods -n mo-hn      
  2. NAME                                  READY   STATUS    RESTARTS     AGE
  3. matrixone-operator-6c9c49fbd7-lw2h2   1/1     Running   2 (8h ago)   9h
  4. mo-tn-0                               1/1     Running   0            2m13s
  5. mo-log-0                              1/1     Running   0            2m47s
  6. mo-log-1                              1/1     Running   0            2m47s
  7. mo-log-2                              1/1     Running   0            2m47s
  8. mo-tp-cn-0                            1/1     Running   0            111s