Knowledge Base

Summary

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC

How to scale up Jiva replica?

How to install OpenEBS in OpenShift 4.1?

How to enable Admission-Controller in OpenShift environment?

How to setup default PodSecurityPolicy to allow the OpenEBS pods to work with all permissions?

How to prevent container logs from exhausting disk space?

How do I reuse an existing PV - after re-creating Kubernetes StatefulSet and its PVC

There are some cases where it had to delete the StatefulSet and re-install a new StatefulSet. In the process you may have to delete the PVCs used by the StatefulSet and retain PV policy by ensuring the Retain as the “Reclaim Policy”. In this case, following are the procedures for re-using an existing PV in your StatefulSet application.

  1. Get the PV name by following command and use it in Step 2.

    1. kubectl get pv

    Following is an example output

    1. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
    2. pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            Delete           Bound     default/mongo-persistent-storage-mongo-0   mongo-pv-az              9m

  2. Patch corresponding PV’s reclaim policy from “Delete” to “Retain”. So that PV will retain even its PVC is deleted.This can be done by using the steps mentioned here.

    Example Output:

    1. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
    2. pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            Retain           Bound     default/mongo-persistent-storage-mongo-0   mongo-pv-az              9m

  3. Get the PVC name by following command and note down the PVC name. You have to use this same PVC name while creating new PVC.

    1. kubectl get pvc

    Example Output:

    1. NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    2. mongo-persistent-storage-mongo-0   Lost      pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   0                        mongo-pv-az    4s

  4. Delete StatefulSet application and associated PVCs.

  5. Create a new PVC YAML named newPVC.yaml with same configuration. Specify old PV name belongs to volumeName under the PVC spec.

    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4.   annotations:
    5.     pv.kubernetes.io/bind-completed: "yes"
    6.     pv.kubernetes.io/bound-by-controller: "yes"
    7.     volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-iscsi
    8.   labels:
    9.     environment: test
    10.     openebs.io/replica-anti-affinity: vehicle-db
    11.     role: mongo
    12.   name: mongo-persistent-storage-mongo-0
    13.   namespace: default
    14. spec:
    15.   accessModes:
    16.   - ReadWriteOnce
    17.   resources:
    18.     requests:
    19.       storage: 5G
    20.   storageClassName: mongo-pv-az
    21.   volumeName: pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7
    22. status:
    23.   accessModes:
    24.   - ReadWriteOnce
    25.   capacity:
    26.     storage: 5G

  6. Apply the modified PVC YAML using the following command

    1. kubectl apply -f newPVC.yaml

    Example Output:

    1. NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    2. mongo-persistent-storage-mongo-0   Lost      pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   0                        mongo-pv-az    4s

  7. Get the newly created PVC UID using kubectl get pvc mongo-persistent-storage-mongo-0 -o yaml.

  8. Update the uid under the claimRef in the PV using the following command. The PVC will get attached to the PV after editing the pv with correct uid.

    1. kubectl edit pv pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7

  9. Get the updated PVC status using the following command.

    1. kubectl get pvc

    Example Output:

    1. NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    2. mongo-persistent-storage-mongo-0   Bound     pvc-cc6767b4-52e8-11e9-b1ef-42010a800fe7   5G         RWO            mongo-pv-az    5m

  10. Apply the same StatefulSet application YAML. The pod will come back online by re-using the existing PVC. The application pod status can be get by following command.

  1. kubectl get pods -n <namespace>

Go to top

How to scale up Jiva replica?

From 0.9.0 OpenEBS version, Jiva pod deployment are scheduling with nodeAffinity. For scaling up Jiva replica count, the following steps has to be performed.

  1. Get the deployment details of replica of corresponding Jiva volume using the following command. If it is deployed in openebs namespace, use corresponding namespace appropriately in the following commands.

    1. kubectl get deploy

    Following is an example output.

    1. NAME                                            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    2. percona                                         1         1         1            1           54s
    3. pvc-4cfacfdd-76d7-11e9-9319-42010a800230-ctrl   1         1         1            1           53s
    4. pvc-4cfacfdd-76d7-11e9-9319-42010a800230-rep    1         1         1            1           53s

  2. Edit the corresponding replica deployment of the Jiva volume using the following command.

    1. kubectl edit deploy <replica_deployment_of_corresponding_volume>

    Example:

    1. kubectl edit deploy pvc-4cfacfdd-76d7-11e9-9319-42010a800230-rep

    Perform Step 3 and 4 and then save the changes. It is required to modify the fields of replica count and hostname details where the replica pods has to be scheduled.

  3. Edit replicas value under spec with the required number. In this example, it was replicas: 1 during the initial deployment. With following change, replicas count will change to 2.

    Example:

    1. replicas: 2

  4. Add corresponding hostnames under value in spec.template.spec.affinity.nodeAffinity.nodeSelectorTerms.key.values. The following is the sample snippet for adding the required hostnames. In the following snippet, it is added the hostname of second node in the mentioned path.

    1.     spec:
    2.       affinity:
    3.         nodeAffinity:
    4.           requiredDuringSchedulingIgnoredDuringExecution:
    5.             nodeSelectorTerms:
    6.             - matchExpressions:
    7.               - key: kubernetes.io/hostname
    8.                 operator: In
    9.                 values:
    10.                 - gke-md-jiva-default-pool-15a2475b-bxr5
    11.                 - gke-md-jiva-default-pool-15a2475b-gzx3

  5. After modifying the above changes, save the configuration. With this change , new replica pods will be running and following command will get the details of replica pods.

    1. kubectl get pod -o wide

    The following is an example output.

    1. NAME                                                             READY     STATUS    RESTARTS   AGE    IP           NODE                                   NOMINATED NODE  
    2. percona-66b4fd4ddf-xvswn                                         1/1       Running   0          32m      
    3. pvc-4cfacfdd-76d7-11e9-9319-42010a800230-ctrl-68d94478df-drj6r   2/2       Running   0          32m
    4. pvc-4cfacfdd-76d7-11e9-9319-42010a800230-rep-f9ff69c6-6lcfz      1/1       Running   0          25s
    5. pvc-4cfacfdd-76d7-11e9-9319-42010a800230-rep-f9ff69c6-9jbfm      1/1       Running   0          25s

Go to top

How to install OpenEBS in OpenShift 4.1

In earlier documentation, it was referred to install OpenEBS by disabling SELinux. But, you can install OpenEBS in OpenShift environment without disabling SELinux using the following steps.

  1. Add OpenEBS Service account to the privileged scc of OpenShift.

    1. oc adm policy add-scc-to-user privileged system:serviceaccount:openebs:openebs-maya-operator

    Example output:

    1. scc "privileged" added to: ["system:serviceaccount:openebs:default"]

  2. Find the latest OpenEBS release version from here and download the latest OpenEBS operator YAML in your master node. The latest openebs-operator YAML file can be downloaded using the following way.

    1. wget https://openebs.github.io/charts/openebs-operator-1.1.0.yaml

  3. Apply the modified the YAML using the following command.

    1. kubectl apply -f openebs-operator-1.1.0.yaml

  4. Verify OpenEBS pod status by using kubectl get pods -n openebs

    NAME READY STATUS RESTARTS AGE maya-apiserver-594699887-4x6bj 1/1 Running 0 60m openebs-admission-server-544d8fb47b-lxd52 1/1 Running 0 60m openebs-localpv-provisioner-59f96b699-dpf8l 1/1 Running 0 60m openebs-ndm-4v6kj 1/1 Running 0 60m openebs-ndm-8g226 1/1 Running 0 60m openebs-ndm-kkpk7 1/1 Running 0 60m openebs-ndm-operator-74d9c78cdc-lbtqt 1/1 Running 0 60m openebs-provisioner-5dfd95987b-nhwb9 1/1 Running 0 60m openebs-snapshot-operator-5d58bd848b-94nnt 2/2 Running 0 60m

  5. For provisioning OpenEBS volumes, you have to edit SCC to allow HostPath volumes and Privileged containers. This can be done in two ways.

    • Using “Restricted” SCC
    • Using “Privileged” SCC

    Using “Restricted” SCC

    By default, any/all users (manual, serviceaccount), use the “restricted” securityContextConstraint (SCC). This SCC doesn’t allow:

    • HostPath Volumes
    • Privileged Containers

    Following have to be set to ensure volume replica pods can run on the cluster:

    1. allowHostDirVolumePlugin: true
    2. allowPrivilegeEscalation: true
    3. allowPrivilegedContainer: true

    Run following command from OpenShift cluster.

    1. oc edit scc restricted

    It will show an output similar to the following.

    could not be patched: the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json You can run `oc replace -f /tmp/oc-edit-vvh25.yaml` to try this update again.

    Note: The above command will not patch the SCC directly. It will generate a temporary file and you have to run the mentioned command in the output to update the restricted SCC.

    Using “Privileged” SCC

    In openshift, the users are mapped to “Projects” & SCC are mapped to users (or serviceAccounts). This method is more preferred. In case, where you want your application to run in privileged containers with particular user/serviceaccount, it can be added to the privileged SCC using following command from OpenShift cluster.

    1. oc adm policy add-scc-to-user privileged system:serviceaccount:<project>:<serviceaccountname>

    Example:

    1.  oc adm policy add-scc-to-user privileged system:serviceaccount:openebs:default

    Note:

    • In OpenShift each namespace automatically creates a project - into which one or more users can be created.
    • An oc apply from inside a project will cause all resources to get created with same, i.e., project namespace.

    Example output:

    scc “privileged” added to: [“system:serviceaccount:openebs:default”]

  6. Now,you can provision OpenEBS volumes. More details for provisioning OpenEBS volumes can be obtained from the User Guide section.

Go to top

How to enable Admission-Controller in OpenShift 3.10 and above

The following proceedure will help to enable admission-controller in OpenShift 3.10 and above.

  1. Update the /etc/origin/master/master-config.yaml file with below configuration.

    1. admissionConfig:
    2.   pluginConfig:
    3.     ValidatingAdmissionWebhook: 
    4.       configuration:
    5.         kind: DefaultAdmissionConfig
    6.         apiVersion: v1
    7.         disable: false 
    8.     MutatingAdmissionWebhook: 
    9.       configuration:
    10.         kind: DefaultAdmissionConfig
    11.         apiVersion: v1
    12.         disable: false

  2. Restart the API and controller services using the following commands.

    1. # master-restart api
    2. # master-restart controllers

Go to top

How to setup default PodSecurityPolicy to allow the OpenEBS pods to work with all permissions?

Apply the following YAML in your cluster.

  • Create a Privileged PSP

    1. apiVersion: extensions/v1beta1
    2. kind: PodSecurityPolicy
    3.  metadata:
    4.    name: privileged
    5.    annotations:
    6.      seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
    7.  spec:
    8.    privileged: true
    9.    allowPrivilegeEscalation: true
    10.    allowedCapabilities:
    11.    - '*'
    12.    volumes:
    13.    - '*'
    14.    hostNetwork: true
    15.    hostPorts:
    16.    - min: 0
    17.      max: 65535
    18.    hostIPC: true
    19.    hostPID: true
    20.    runAsUser:
    21.      rule: 'RunAsAny'
    22.    seLinux:
    23.      rule: 'RunAsAny'
    24.    supplementalGroups:
    25.      rule: 'RunAsAny'
    26.    fsGroup:
    27.      rule: 'RunAsAny'

  • Associate the above PSP to a ClusterRole

    1. kind: ClusterRole
    2. apiVersion: rbac.authorization.k8s.io/v1
    3. metadata:
    4.   name: privilegedpsp
    5. rules:
    6. - apiGroups: ['extensions']
    7.   resources: ['podsecuritypolicies']
    8.   verbs:     ['use']
    9.   resourceNames:
    10.   - privileged

  • Associate the above Privileged ClusterRole to OpenEBS Service Account

    1. apiVersion: rbac.authorization.k8s.io/v1
    2. kind: ClusterRoleBinding
    3. metadata:
    4.   annotations:
    5.     rbac.authorization.kubernetes.io/autoupdate: "true"
    6.   name: openebspsp
    7. roleRef:
    8.   apiGroup: rbac.authorization.k8s.io
    9.   kind: ClusterRole
    10.   name: privilegedpsp
    11. subjects:
    12. - kind: ServiceAccount
    13.   name: openebs-maya-operator
    14.   namespace: openebs

  • Proceed to install the OpenEBS. Note that the namespace and service account name used by the OpenEBS should match what is provided in the above ClusterRoleBinding.

Go to top

How to prevent container logs from exhausting disk space?

Container logs, if left unchecked, can eat into the underlying disk space causing disk-pressure conditions leading to eviction of pods running on a given node. This can be prevented by performing log-rotation based on file-size while specifying retention count. One recommended way to do this is by configuring the docker logging driver on the individual cluster nodes. Follow the steps below to enable log-rotation.

  1. Configure the docker configuration file /etc/docker/daemon.json (create one if not already found) with the log-options similar to ones shown below (with desired driver, size at which logs are rotated, maximum logfile retention count & compression respectively):

    1. {
    2.   "log-driver": "json-file",
    3.   "log-opts": {
    4.      "max-size": "400k",
    5.      "max-file": "3",
    6.      "compress": "true"
    7.   }
    8. }

  2. Restart the docker daemon on the nodes. This may cause a temprary disruption of the running containers & cause the node to show up as Not Ready until the daemon has restarted successfully.

    1. systemctl daemon-reload
    2. systemctl restart docker

  3. To verify that the newly set log-options have taken effect, the following commands can be used:

    • At a node-level, the docker logging driver in use can be checked via the following command:

      1. docker info

      The LogConfig section of the output must show the desired values:

      1.  "LogConfig": {
      2.  "Type": "json-file",
      3.  "Config": {}

    • At the individual container level, the log options in use can be checked via the following command:

      1. docker inspect <container-id>

      The LogConfig section of the output must show the desired values:

      1.  "LogConfig": {
      2.         "Type": "json-file",
      3.         "Config": {
      4.             "max-file": "3",
      5.             "max-size": "400k",
      6.             "compress": "true"
      7.          }
      8.  }

  4. To view the current & compressed files, check the contents of the /var/lib/docker/containers/<container-id>/ directory. The symlinks at /var/log/containers/<container-name> refer to the above.

NOTES:

  • The steps are common for Linux distributions (tested on CentOS, RHEL, Ubuntu)

  • Log rotation via the specified procedure is supported by docker logging driver types: json-file (default), local

  • Ensure there are no dockerd cli flags specifying the --log-opts (verify via ps -aux or service definition files in /etc/init.d or /etc/systemd/system/docker.service.d). The docker daemon fails to start if an option is duplicated between the file and the flags, regardless their value.

  • These log-options are applicable only to the containers created after the dockerd restart (which is automatically taken care by the kubelet)

  • The kubectl log reads the uncompressed files/symlinks at /var/log/containers and thereby show rotated/rolled-over logs. If you would like to read the retained/compressed log content as well use docker log command on the nodes. Note that reading from compressed logs can cause temporary increase in CPU utilization (on account of decompression actions performed internally)

  • The log-opt compress: true: is supported from Docker version: 18.04.0. The max-file & max-size opts are supported on earlier releases as well.