Set up a GlusterFS Server

As an open-source distributed file system, GlusterFS allows you to mount glusterfs volumes to your Pods. If a glusterfs volume is pre-populated with data, they can be shared among your Pods in a Kubernetes cluster.

This tutorial demonstrates how to configure GlusterFS on three server machines and install Heketi to manage your GlusterFS cluster.

Once you have GlusterFS and Heketi set up, you can install GlusterFS on your client machine and use KubeKey to create a KubeSphere cluster with GlusterFS as a storage class.

Prepare GlusterFS Nodes

There are three server machines of Ubuntu 16.04 in this example with each having one attached disk.

HostnameIP AddressOperating SystemDevice
server1192.168.0.2Ubuntu 16.04, 4 Cores, 4 GB Memory/dev/vdd 300 GB
server2192.168.0.3Ubuntu 16.04, 4 Cores, 4 GB Memory/dev/vdd 300 GB
server3192.168.0.4Ubuntu 16.04, 4 Cores, 4 GB Memory/dev/vdd 300 GB

Note

  • Heketi will be installed on server1, which provides a RESTful management interface to manage the lifecycle of GlusterFS volumes. You can install it on a separate machine as well.

  • Attach more block storage disks to your server machine if you need more storage space.

  • Data will be saved to /dev/vdd (block device), which must be original without partitioning or formatting.

Set up Passwordless SSH Login

Configure root login

  1. Log in to server1 and switch to the root user.

    1. sudo -i

  2. Change the root user password:

    1. passwd

    Note

    Make sure password authentication is enabled in the file /etc/ssh/sshd_config (the value of PasswordAuthentication should be yes).

  3. Change the root user password of server2 and server3 as well.

Add hosts file entries

  1. Configure your DNS or edit the /etc/hosts file on all server machines to add their hostnames and IP addresses:

    1. vi /etc/hosts
    1. # hostname loopback address
    2. 192.168.0.2  server1
    3. 192.168.0.3  server2
    4. 192.168.0.4  server3

  2. Make sure you add the above entries to the hosts file of all server machines.

Configure passwordless SSH login

  1. On server1, create a key by running the following command. Press Enter directly for all the prompts.

    1. ssh-keygen

  2. Copy the key to all GlusterFS nodes.

    1. ssh-copy-id [email protected]
    1. ssh-copy-id [email protected]
    1. ssh-copy-id [email protected]

  3. Verify that you can access all server machines from server1 through passwordless login.

    1. ssh [email protected]
    1. ssh [email protected]
    1. ssh [email protected]

Install GlusterFS on All Server Machines

  1. On server1, run the following command to install software-properties-common.

    1. apt-get install software-properties-common

  2. Add the community GlusterFS PPA.

    1. add-apt-repository ppa:gluster/glusterfs-7

  3. Make sure you are using the latest package.

    1. apt-get update

  4. Install the GlusterFS server.

    1. apt-get install glusterfs-server -y

  5. Make sure you run the above commands on server2 and server3 as well and verify the version on all machines.

    1. glusterfs -V

Note

The above commands may be slightly different if you do no install GlusterFS on Ubuntu. For more information, see the Gluster documentation.

Load Kernel Modules

  1. Run the following commands to load three necessary kernel modules on server1.

    1. echo dm_thin_pool | sudo tee -a /etc/modules
    1. echo dm_snapshot | sudo tee -a /etc/modules
    1. echo dm_mirror | sudo tee -a /etc/modules

  2. Intall thin-provisioning-tools.

    1. apt-get -y install thin-provisioning-tools

  3. Make sure you run the above commands on server2 and server3 as well.

Create a GlusterFS Cluster

  1. Run the following command on server1 to add other nodes and create a cluster.

    1. gluster peer probe server2
    1. gluster peer probe server3

  2. Verify that all nodes in the cluster are connected successfully.

    1. gluster peer status

  3. Expected output:

    1. Number of Peers: 2
    3. Hostname: server2
    4. Uuid: e1192d6a-b65e-4ce8-804c-72d9425211a6
    5. State: Peer in Cluster (Connected)
    7. Hostname: server3
    8. Uuid: 9bd733e4-96d4-49d5-8958-6c947a2b4fa6
    9. State: Peer in Cluster (Connected)

Install Heketi

As GlusterFS itself does not provide a way for API calls, you can install Heketi to manage the lifecycle of GlusterFS volumes with a RESTful API for Kubernetes calls. In this way, your Kubernetes cluster can dynamically provision GlusterFS volumes. Heketi v7.0.0 will be installed in this example. For more information about available Heketi versions, see its Release Page.

  1. Download Heketi on server1.

    1. wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-v7.0.0.linux.amd64.tar.gz

    Note

    You can also install Heketi on a separate machine.

  2. Unzip the file.

    1. tar -xf heketi-v7.0.0.linux.amd64.tar.gz
    1. cd heketi
    1. cp heketi /usr/bin
    1. cp heketi-cli /usr/bin

  3. Create a Heketi service file.

    1. vi /lib/systemd/system/heketi.service
    1. [Unit]
    2. Description=Heketi Server
    3. [Service]
    4. Type=simple
    5. WorkingDirectory=/var/lib/heketi
    6. ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
    7. Restart=on-failure
    8. StandardOutput=syslog
    9. StandardError=syslog
    10. [Install]
    11. WantedBy=multi-user.target

  4. Create Heketi folders.

    1. mkdir -p /var/lib/heketi
    1. mkdir -p /etc/heketi

  5. Create a JSON file for Heketi configurations.

    1. vi /etc/heketi/heketi.json

    An example file:

    ``` { “_port_comment”: “Heketi Server Port Number”, “port”: “8080”,

    “_use_auth”: “Enable JWT authorization. Please enable for deployment”, “use_auth”: false,

    “_jwt”: “Private keys for access”, “jwt”: {

    1. "_admin": "Admin has access to all APIs",
    2. "admin": {
    3.   "key": "123456"
    4. },
    5. "_user": "User only has access to /volumes endpoint",
    6. "user": {
    7.   "key": "123456"
    8. }

    },

    “_glusterfs_comment”: “GlusterFS Configuration”, “glusterfs”: {

    1. "_executor_comment": [
    2.   "Execute plugin. Possible choices: mock, ssh",
    3.   "mock: This setting is used for testing and development.",
    4.   "      It will not send commands to any node.",
    5.   "ssh:  This setting will notify Heketi to ssh to the nodes.",
    6.   "      It will need the values in sshexec to be configured.",
    7.   "kubernetes: Communicate with GlusterFS containers over",
    8.   "            Kubernetes exec api."
    9. ],
    10. "executor": "ssh",
    12. "_sshexec_comment": "SSH username and private key file information",
    13. "sshexec": {
    14.   "keyfile": "/root/.ssh/id_rsa",
    15.   "user": "root"
    16. },
    18. "_kubeexec_comment": "Kubernetes configuration",
    19. "kubeexec": {
    20.   "host" :"https://kubernetes.host:8443",
    21.   "cert" : "/path/to/crt.file",
    22.   "insecure": false,
    23.   "user": "kubernetes username",
    24.   "password": "password for kubernetes user",
    25.   "namespace": "Kubernetes namespace",
    26.   "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
    27. },
    29. "_db_comment": "Database file name",
    30. "db": "/var/lib/heketi/heketi.db",
    31. "brick_max_size_gb" : 1024,

    “brick_min_size_gb” : 1, “max_bricks_per_volume” : 33,

  1.     "_loglevel_comment": [
  2.       "Set log level. Choices are:",
  3.       "  none, critical, error, warning, info, debug",
  4.       "Default is warning"
  5.     ],
  6.     "loglevel" : "debug"
  7.   }
  8. }
  9. ```
  11. Note
  13. The account `admin` and its `key` value must be provided when you install GlusterFS as a storage class of your KubeSphere cluster.

  1. Start Heketi.

    1. systemctl start heketi

  2. Check the status of Heketi.

    1. systemctl status heketi

    If you can see active (running), it means the installation is successful. Expected output:

    1. ● heketi.service - Heketi Server
    2.    Loaded: loaded (/lib/systemd/system/heketi.service; disabled; vendor preset: enabled)
    3.    Active: active (running) since Tue 2021-03-09 13:04:30 CST; 4s ago
    4.  Main PID: 9282 (heketi)
    5.     Tasks: 8
    6.    Memory: 6.5M
    7.       CPU: 62ms
    8.    CGroup: /system.slice/heketi.service
    9.            └─9282 /usr/bin/heketi --config=/etc/heketi/heketi.json
    11. Mar 09 13:04:30 server1 systemd[1]: Started Heketi Server.
    12. Mar 09 13:04:30 server1 heketi[9282]: Heketi v7.0.0
    13. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Loaded ssh executor
    14. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max bricks per volume set to 33
    15. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Max brick size 1024 GB
    16. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Adv: Min brick size 1 GB
    17. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 GlusterFS Application Loaded
    18. Mar 09 13:04:30 server1 heketi[9282]: [heketi] INFO 2021/03/09 13:04:30 Started Node Health Cache Monitor
    19. Mar 09 13:04:30 server1 heketi[9282]: Listening on port 8080

  3. Enable Heketi.

    1. systemctl enable heketi

    Expected output:

    1. Created symlink from /etc/systemd/system/multi-user.target.wants/heketi.service to /lib/systemd/system/heketi.service.

  4. Create a topology configuration file for Heketi. It contains the information of clusters, nodes, and disks added to Heketi.

    1. vi /etc/heketi/topology.json

    An example file:

    1.    {
    2.     "clusters": [
    3.        {
    4.          "nodes": [
    5.            {
    6.              "node": {
    7.                "hostnames": {
    8.                  "manage": [
    9.                    "192.168.0.2" 
    10.                 ],
    11.                 "storage": [
    12.                   "192.168.0.2" 
    13.                 ]
    14.               },
    15.               "zone": 1
    16.             },
    17.             "devices": [
    18.               "/dev/vdd" 
    19.             ]
    20.           },
    21.           {
    22.             "node": {
    23.               "hostnames": {
    24.                 "manage": [
    25.                   "192.168.0.3" 
    26.                 ],
    27.                 "storage": [
    28.                   "192.168.0.3"
    29.                 ]
    30.               },
    31.               "zone": 1
    32.             },
    33.             "devices": [
    34.               "/dev/vdd" 
    35.             ]
    36.           },
    37.           {
    38.              "node": {
    39.                "hostnames": {
    40.                  "manage": [
    41.                    "192.168.0.4"
    42.                 ],
    43.                 "storage": [
    44.                   "192.168.0.4"  
    45.                 ]
    46.               },
    47.               "zone": 1
    48.             },
    49.             "devices": [
    50.               "/dev/vdd"
    51.             ]
    52.           }
    53.         ]
    54.       }
    55.     ]
    56.   }

    Note

    • Replace the IP addresses above with your own.
    • Add your own disk name for devices.

  5. Load the Heketi JSON file.

    1. export HEKETI_CLI_SERVER=http://localhost:8080
    1. heketi-cli topology load --json=/etc/heketi/topology.json

    Expected output:

    1. Creating cluster ... ID: 2d9e11adede04fe6d07cb81c5a1a7ea4
    2.    Allowing file volumes on cluster.
    3.    Allowing block volumes on cluster.
    4.    Creating node 192.168.0.2 ... ID: 0a9f240ab6fd96ea014948c5605be675
    5.        Adding device /dev/vdd ... OK
    6.    Creating node 192.168.0.3 ... ID: 2468086cadfee8ef9f48bc15db81c88a
    7.        Adding device /dev/vdd ... OK
    8.    Creating node 192.168.0.4 ... ID: 4c21b33d5c32029f5b7dc6406977ec34
    9.        Adding device /dev/vdd ... OK

  6. The above output displays both your cluster ID and node ID. Run the following command to view your cluster information.

    1. heketi-cli cluster info 2d9e11adede04fe6d07cb81c5a1a7ea4 # Use your own cluster ID.

    Expected output:

    1. Cluster id: 2d9e11adede04fe6d07cb81c5a1a7ea4
    2. Nodes:
    3. 0a9f240ab6fd96ea014948c5605be675
    4. 2468086cadfee8ef9f48bc15db81c88a
    5. 4c21b33d5c32029f5b7dc6406977ec34
    6. Volumes:
    8. Block: true
    10. File: true