Capabilities in CephFS

When a client wants to operate on an inode, it will query the MDS in variousways, which will then grant the client a set of capabilities. Thesegrant the client permissions to operate on the inode in various ways. Oneof the major differences from other network file systems (e.g NFS or SMB) isthat the capabilities granted are quite granular, and it’s possible thatmultiple clients can hold different capabilities on the same inodes.

Types of Capabilities

There are several “generic” capability bits. These denote what sort of abilitythe capability grants.

  1. /* generic cap bits */
  2. #define CEPH_CAP_GSHARED     1  /* client can reads (s) */
  3. #define CEPH_CAP_GEXCL       2  /* client can read and update (x) */
  4. #define CEPH_CAP_GCACHE      4  /* (file) client can cache reads (c) */
  5. #define CEPH_CAP_GRD         8  /* (file) client can read (r) */
  6. #define CEPH_CAP_GWR        16  /* (file) client can write (w) */
  7. #define CEPH_CAP_GBUFFER    32  /* (file) client can buffer writes (b) */
  8. #define CEPH_CAP_GWREXTEND  64  /* (file) client can extend EOF (a) */
  9. #define CEPH_CAP_GLAZYIO   128  /* (file) client can perform lazy io (l) */

These are then shifted by a particular number of bits. These denote a part ofthe inode’s data or metadata on which the capability is being granted:

  1. /* per-lock shift */
  2. #define CEPH_CAP_SAUTH      2 /* A */
  3. #define CEPH_CAP_SLINK      4 /* L */
  4. #define CEPH_CAP_SXATTR     6 /* X */
  5. #define CEPH_CAP_SFILE      8 /* F */

Only certain generic cap types are ever granted for some of those “shifts”,however. In particular, only the FILE shift ever has more than the first twobits.

  1. | AUTH | LINK | XATTR | FILE
  2. 2      4      6       8

From the above, we get a number of constants, that are generated by takingeach bit value and shifting to the correct bit in the word:

  1. #define CEPH_CAP_AUTH_SHARED  (CEPH_CAP_GSHARED  << CEPH_CAP_SAUTH)

These bits can then be or’ed together to make a bitmask denoting a set ofcapabilities.

There is one exception:

  1. #define CEPH_CAP_PIN  1  /* no specific capabilities beyond the pin */

The “pin” just pins the inode into memory, without granting any other caps.

Graphically:

  1. +---+---+---+---+---+---+---+---+
  2. | p | _ |As   x |Ls   x |Xs   x |
  3. +---+---+---+---+---+---+---+---+
  4. |Fs   x   c   r   w   b   a   l |
  5. +---+---+---+---+---+---+---+---+

The second bit is currently unused.

Abilities granted by each cap

While that is how capabilities are granted (and communicated), the importantbit is what they actually allow the client to do:

  • PIN: this just pins the inode into memory. This is sufficient to allow theclient to get to the inode number, as well as other immutable things likemajor or minor numbers in a device inode, or symlink contents.

  • AUTH: this grants the ability to get to the authentication-related metadata.In particular, the owner, group and mode. Note that doing a full permissioncheck may require getting at ACLs as well, which are stored in xattrs.

  • LINK: the link count of the inode

  • XATTR: ability to access or manipulate xattrs. Note that since ACLs arestored in xattrs, it’s also sometimes necessary to access them when checkingpermissions.

  • FILE: this is the big one. These allow the client to access and manipulatefile data. It also covers certain metadata relating to file data – thesize, mtime, atime and ctime, in particular.

Shorthand

Note that the client logging can also present a compact representation of thecapabilities. For example:

  1. pAsLsXsFs

The ‘p’ represents the pin. Each capital letter corresponds to the shiftvalues, and the lowercase letters after each shift are for the actualcapabilities granted in each shift.