Using Rook-Ceph with Pod Security Policies (PSPs)

See the Rook overall PSP document before continuing on here with Ceph specifics.

PodSecurityPolicy

You need at least one PodSecurityPolicy that allows privileged Pod execution. Here is an example that is reasonably pared down for Ceph, though more work to minimize permissions can be done:

  1. apiVersion: policy/v1beta1
  2. kind: PodSecurityPolicy
  3. metadata:
  4.   name: privileged
  5. spec:
  6.   fsGroup:
  7.     rule: RunAsAny
  8.   privileged: true
  9.   runAsUser:
  10.     rule: RunAsAny
  11.   seLinux:
  12.     rule: RunAsAny
  13.   supplementalGroups:
  14.     rule: RunAsAny
  15.   volumes:
  16.     - 'configMap'
  17.     - 'emptyDir'
  18.     - 'projected'
  19.     - 'secret'
  20.     - 'downwardAPI'
  21.     - 'hostPath'
  22.     - 'flexVolume'
  23.   hostPID: true
  24.   # hostNetwork is required for using host networking
  25.   hostNetwork: false

Hint: Allowing hostNetwork usage is required when using hostNetwork: true in the Cluster Resource Definition! You are then also required to allow the usage of hostPorts in the PodSecurityPolicy. The given port range is a minimal working recommendation for a Rook Ceph cluster:

  1.    hostPorts:
  2.      # Ceph msgr2 port
  3.      - min: 3300
  4.        max: 3300
  5.      # Ceph ports
  6.      - min: 6789
  7.        max: 7300
  8.      # Ceph MGR Prometheus Metrics
  9.      - min: 9283
  10.        max: 9283
ClusterRole and ClusterRoleBinding

Next up you require a ClusterRole and a corresponding ClusterRoleBinding, which enables the Rook Agent ServiceAccount to run the rook-ceph-agent Pods on all nodes with privileged rights. Here are the definitions:

  1. # privilegedPSP grants access to use the privileged PSP.
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5.   name: psp:rook
  6. rules:
  7. - apiGroups:
  8.   - policy
  9.   resources:
  10.   - podsecuritypolicies
  11.   resourceNames:
  12.   - privileged
  13.   verbs:
  14.   - use

and

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: rook-ceph-system
  5. ---
  6. # Allow the rook-ceph-system serviceAccount to use the privileged PSP
  7. apiVersion: rbac.authorization.k8s.io/v1
  8. kind: ClusterRoleBinding
  9. metadata:
  10.   name: rook-ceph-system-psp
  11. roleRef:
  12.   apiGroup: rbac.authorization.k8s.io
  13.   kind: ClusterRole
  14.   name: psp:rook
  15. subjects:
  16. - kind: ServiceAccount
  17.   name: rook-ceph-system
  18.   namespace: rook-ceph

Save these definitions to one or multiple yaml files and create them by executing kubectl apply -f <nameOfYourFile>.yaml

You will also require two more RoleBindings for each Rook Cluster you deploy: Create these two RoleBindings in the Namespace you plan to deploy your Rook Cluster into (default is “rook” namespace):

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: rook-ceph
  5. ---
  6. # Allow the default serviceAccount to use the privileged PSP
  7. apiVersion: rbac.authorization.k8s.io/v1
  8. kind: RoleBinding
  9. metadata:
  10.   name: rook-default-psp
  11.   namespace: rook-ceph
  12. roleRef:
  13.   apiGroup: rbac.authorization.k8s.io
  14.   kind: ClusterRole
  15.   name: psp:rook
  16. subjects:
  17. - kind: ServiceAccount
  18.   name: default
  19.   namespace: rook-ceph
  20. ---
  21. # Allow the rook-ceph-osd serviceAccount to use the privileged PSP
  22. apiVersion: rbac.authorization.k8s.io/v1
  23. kind: RoleBinding
  24. metadata:
  25.   name: rook-ceph-osd-psp
  26.   namespace: rook-ceph
  27. roleRef:
  28.   apiGroup: rbac.authorization.k8s.io
  29.   kind: ClusterRole
  30.   name: psp:rook
  31. subjects:
  32. - kind: ServiceAccount
  33.   name: rook-ceph-osd
  34.   namespace: rook-ceph
  35. ---
  36. # Allow the rook-ceph-mgr serviceAccount to use the privileged PSP
  37. apiVersion: rbac.authorization.k8s.io/v1
  38. kind: RoleBinding
  39. metadata:
  40.   name: rook-ceph-mgr-psp
  41.   namespace: rook-ceph
  42. roleRef:
  43.   apiGroup: rbac.authorization.k8s.io
  44.   kind: ClusterRole
  45.   name: psp:rook
  46. subjects:
  47. - kind: ServiceAccount
  48.   name: rook-ceph-mgr
  49.   namespace: rook-ceph