Configure shared memory for MatrixOne

In a MatrixOne cluster, it is necessary to configure external shared memory to ensure that each node in a distributed environment can use the same storage for database operations. This document describes configuring different types of shared memory for a MatrixOne cluster on Kubernetes. The currently configured storage services that MatrixOne can support are AWS S3, MinIO, and NFS.

Before you start

Configure shared memory

Overview

The shared memory used by the cluster is configured via the .spec.logset.sharedStorage field of the MatrixOneCluster object. This field cannot be modified after the cluster is created.

Configuration Steps

You can choose one of AWS S3, MinIO, or NFS to configure MatrixOne’s shared memory according to your environment.

Option 1: Configure AWS S3

If you are using the AWS public cloud, it is recommended to configure AWS S3 as the shared memory of the MatrixOne cluster. The configuration steps are as follows:

  1. Use an existing S3 bucket or refer to the Create Bucket official document to create a new S3 bucket.

  2. Refer to the Managing Access Keys for IAM Users official document to create a set of access keys.

  3. Create a k8s Secret in the target Namespace, including the access key generated in the second step, which authorizes the MatrixOne cluster to access the corresponding S3 bucket. The code example is as follows:

    1. kubectl -n <YOUR_NAMESPACE> create secret generic aws --from-literal=AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID> --from-literal=AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>

  4. Modify the YAML definition of the cluster, and configure the sharedStorage field by the following commands. You can use path to specify the specific path where the cluster stores data in the target S3 bucket. If only the first-level directory (ie ${bucketName}) is configured, the data will be stored in the root directory of the target bucket:

    1. pec:
    2.   logService:
    3.     sharedStorage:
    4.       s3:
    5.         path: ${bucketName}/${path}
    6.         # bucket 所在的 region
    7.         region: us-west-2
    8.         secretRef:
    9.           name: aws

  5. Create a MatrixOne cluster and verify that the cluster starts correctly. For the steps to create a MatrixOne cluster, please refer to Using MatrixOne Operator to Deploy a MatrixOne Cluster.

Configure other authentication

The above example uses a static access key to grant the MatrixOne cluster access to the target S3 bucket. Kubernetes can also be based on IAM roles for service accounts provided by AWS. The mechanism authorizes the MatrixOne cluster, which avoids storing static key files in the system and enhances security. The configuration process is as follows:

  1. Refer to AWS Official Document, create a k8s ServiceAccount associated with AWS Role, and permit the corresponding AWS Role to access the target S3 bucket.

  2. Configure the MatrixOne cluster to use the ServiceAccount created in the first step through the .spec.serviceAccountName field, and delete the secretRef configuration of S3. The code example is as follows:

    1. pec:
    2. ogService:
    3.     sharedStorage:
    4.       s3:
    5.         path: ${bucketName}/${path}
    6.   serviceAccountName: ${serviceAccountName}

Option 2: Using MinIO

  1. Refer to the official MinIO documentation, Installing MinIO Storage in a Linux environment.
  2. Log into MinIO Console to create a new bucket on MinIO.

  3. Refer to the MinIO User Manager official document to create an access password with sufficient permissions key.

  4. Create a k8s Secret in the target Namespace, including the access key generated in the step 2, to authorize the MatrixOne cluster to access the corresponding MinIO storage bucket:

    1. kubectl -n <YOUR_NAMESPACE> create secret generic minio --from-literal=AWS_ACCESS_KEY_ID=<YOUR_MINIO_ACCESS_KEY_ID> --from-literal=AWS_SECRET_ACCESS_KEY=<YOUR_MINIO_SECRET_ACCESS_KEY>

  5. Modify the YAML definition of the cluster, and configure the sharedStorage field by the following commands. You can use path to specify the specific path where the cluster stores data in the target bucket. If only the first-level directory (ie ${bucketName}) is configured, the data will be stored in the root directory of the target bucket:

    1. pec:
    2.   logSere:
    3.     sharedStorage:
    4.       s3:
    5.         type: minio
    6.         path: ${bucketName}/${path}
    7.         endpoint: ${minioURL}
    8.         secretRef:
    9.           name: minio

Option 3: Using NFS

Coming soon…

Configure shared memory Cache

To improve performance, matrixone-operator will automatically configure shared memory cache for MatrixOne. The rules are as follows:

  • If the component does not have a memory resource request (.esources.requests.memory), then the cache will not be enabled by default.
  • If the component is configured with a memory resource application, it will default start the memory-based shared memory cache. The cache size is 50% of the memory resource application.
  • You can always explicitly enable memory cache and directly specify the cache size by setting .spec.cn.shreStorageCache.memoryCacheSize and .spec.dn.sharedStorageCache.memoryCacheSize, the setting example is as follows:
  1. pec:
  2.   dn:
  3.     replica 1
  4. + sharedStorageCache:
  5. + emoryCacheSize: 1Gi

Note

A huge memory cache will cause Pod out of memory.