Ceph File System Scrub

CephFS provides the cluster admin (operator) to check consistency of a file systemvia a set of scrub commands. Scrub can be classified into two parts:

  • Forward Scrub: In which the scrub operation starts at the root of the file system(or a sub directory) and looks at everything that can be touched in the hierarchyto ensure consistency.

  • Backward Scrub: In which the scrub operation looks at every RADOS object in thefile system pools and maps it back to the file system hierarchy.

This document details commands to initiate and control forward scrub (referred asscrub thereafter).

Initiate File System Scrub

To start a scrub operation for a directory tree use the following command

  1. ceph tell mds.a scrub start / recursive
  2. {
  3.     "return_code": 0,
  4.     "scrub_tag": "6f0d204c-6cfd-4300-9e02-73f382fd23c1",
  5.     "mode": "asynchronous"
  6. }

Recursive scrub is asynchronous (as hinted by mode in the output above). Scrub tag isa random string that can used to monitor the progress of the scrub operation (explainedfurther in this document).

Custom tag can also be specified when initiating the scrub operation. Custom tags getpersisted in the metadata object for every inode in the file system tree that is beingscrubbed.

  1. ceph tell mds.a scrub start /a/b/c recursive tag0
  2. {
  3.     "return_code": 0,
  4.     "scrub_tag": "tag0",
  5.     "mode": "asynchronous"
  6. }

Monitor (ongoing) File System Scrubs

Status of ongoing scrubs can be monitored using in scrub status command. This commandslists out ongoing scrubs (identified by the tag) along with the path and options used toinitiate the scrub.

  1. ceph tell mds.a scrub status
  2. {
  3.     "status": "scrub active (85 inodes in the stack)",
  4.     "scrubs": {
  5.         "6f0d204c-6cfd-4300-9e02-73f382fd23c1": {
  6.             "path": "/",
  7.             "options": "recursive"
  8.         }
  9.     }
  10. }

status shows the number of inodes that are scheduled to be scrubbed at any point in time,hence, can change on subsequent scrub status invocations. Also, a high level summary ofscrub operation (which includes the operation state and paths on which scrub is triggered)gets displayed in ceph status.

  1. ceph status
  2. [...]
  3.  
  4. task status:
  5.   scrub status:
  6.       mds.0: active [paths:/]
  7.  
  8. [...]

Control (ongoing) File System Scrubs

  • Pause: Pausing ongoing scrub operations results in no new or pending inodes beingscrubbed after in-flight RADOS ops (for the inodes that are currently being scrubbed)finish.
  1. ceph tell mds.a scrub pause
  2. {
  3.     "return_code": 0
  4. }

scrub status after pausing reflects the paused state. At this point, initiating new scruboperations (via scrub start) would just queue the inode for scrub.

  1. ceph tell mds.a scrub status
  2. {
  3.     "status": "PAUSED (66 inodes in the stack)",
  4.     "scrubs": {
  5.         "6f0d204c-6cfd-4300-9e02-73f382fd23c1": {
  6.             "path": "/",
  7.             "options": "recursive"
  8.         }
  9.     }
  10. }
  • Resume: Resuming kick starts a paused scrub operation.
  1. ceph tell mds.a. scrub resume
  2. {
  3.     "return_code": 0
  4. }
  • Abort: Aborting ongoing scrub operations removes pending inodes from the scrubqueue (thereby aborting the scrub) after in-flight RADOS ops (for the inodes thatare currently being scrubbed) finish.
  1. ceph tell mds.a. scrub abort
  2. {
  3.     "return_code": 0
  4. }