File striping

The text below describes how files from Ceph file system clients arestored across objects stored in RADOS.

ceph_file_layout

Ceph distributes (stripes) the data for a given file across a numberof underlying objects. The way file data is mapped to those objectsis defined by the ceph_file_layout structure. The data distributionis a modified RAID 0, where data is striped across a set of objects upto a (per-file) fixed size, at which point another set of objectsholds the file’s data. The second set also holds no more than thefixed amount of data, and then another set is used, and so on.

Defining some terminology will go a long way toward explaining theway file data is laid out across Ceph objects.

    • file
    • A collection of contiguous data, named from the perspective ofthe Ceph client (i.e., a file on a Linux system using Cephstorage). The data for a file is divided into fixed-size“stripe units,” which are stored in ceph “objects.”
    • stripe unit
    • The size (in bytes) of a block of data used in the RAID 0distribution of a file. All stripe units for a file have equalsize. The last stripe unit is typically incomplete–i.e. itrepresents the data at the end of the file as well as unused“space” beyond it up to the end of the fixed stripe unit size.
    • stripe count
    • The number of consecutive stripe units that constitute a RAID 0“stripe” of file data.
    • stripe
    • A contiguous range of file data, RAID 0 striped across “stripecount” objects in fixed-size “stripe unit” blocks.
    • object
    • A collection of data maintained by Ceph storage. Objects areused to hold portions of Ceph client files.
    • object set
    • A set of objects that together represent a contiguous portion ofa file.

Three fields in the ceph_file_layout structure define this mapping:

  1. u32 fl_stripe_unit;
  2. u32 fl_stripe_count;
  3. u32 fl_object_size;

(They are actually maintained in their on-disk format, __le32.)

The role of the first two fields should be clear from thedefinitions above.

The third field is the maximum size (in bytes) of an object used toback file data. The object size is a multiple of the stripe unit.

A file’s data is blocked into stripe units, and consecutive stripeunits are stored on objects in an object set. The number of objectsin a set is the same as the stripe count. No object storing filedata will exceed the file’s designated object size, so after somefixed number of complete stripes, a new object set is used to storesubsequent file data.

Note that by default, Ceph uses a simple striping strategy in whichobject_size equals stripe_unit and stripe_count is 1. This simplyputs one stripe_unit in each object.

Here’s a more complex example:

  1. file size = 1 trillion = 1000000000000 bytes
  2.  
  3. fl_stripe_unit = 64KB = 65536 bytes
  4. fl_stripe_count = 5 stripe units per stripe
  5. fl_object_size = 64GB = 68719476736 bytes

This means:

  1. file stripe size = 64KB * 5 = 320KB = 327680 bytes
  2. each object holds 64GB / 64KB = 1048576 stripe units
  3. file object set size = 64GB * 5 = 320GB = 343597383680 bytes
  4.     (also 1048576 stripe units * 327680 bytes per stripe unit)

So the file’s 1 trillion bytes can be divided into complete objectsets, then complete stripes, then complete stripe units, and finallya single incomplete stripe unit:

  1. - 1 trillion bytes / 320GB per object set = 2 complete object sets
  2.     (with 312805232640 bytes remaining)
  3. - 312805232640 bytes / 320KB per stripe = 954605 complete stripes
  4.     (with 266240 bytes remaining)
  5. - 266240 bytes / 64KB per stripe unit = 4 complete stripe units
  6.     (with 4096 bytes remaining)
  7. - and the final incomplete stripe unit holds those 4096 bytes.

The ASCII art below attempts to capture this:

  1.    _________   _________   _________   _________   _________
  2.   /object  0\ /object  1\ /object  2\ /object  3\ /object  4\
  3.   +=========+ +=========+ +=========+ +=========+ +=========+
  4.   |  stripe | |  stripe | |  stripe | |  stripe | |  stripe |
  5. o |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | stripe 0
  6. b |     0   | |     1   | |     2   | |     3   | |     4   |
  7. j |---------| |---------| |---------| |---------| |---------|
  8. e |  stripe | |  stripe | |  stripe | |  stripe | |  stripe |
  9. c |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | stripe 1
  10. t |     5   | |     6   | |     7   | |     8   | |     9   |
  11.   |---------| |---------| |---------| |---------| |---------|
  12. s |     .   | |     .   | |     .   | |     .   | |     .   |
  13. e       .           .           .           .           .
  14. t |     .   | |     .   | |     .   | |     .   | |     .   |
  15.   |---------| |---------| |---------| |---------| |---------|
  16. 0 |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  17.   |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 1048575
  18.   | 5242875 | | 5242876 | | 5242877 | | 5242878 | | 5242879 |
  19.   \=========/ \=========/ \=========/ \=========/ \=========/
  20.  
  21.    _________   _________   _________   _________   _________
  22.   /object  5\ /object  6\ /object  7\ /object  8\ /object  9\
  23.   +=========+ +=========+ +=========+ +=========+ +=========+
  24.   |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  25. o |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 1048576
  26. b | 5242880 | | 5242881 | | 5242882 | | 5242883 | | 5242884 |
  27. j |---------| |---------| |---------| |---------| |---------|
  28. e |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  29. c |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 1048577
  30. t | 5242885 | | 5242886 | | 5242887 | | 5242888 | | 5242889 |
  31.   |---------| |---------| |---------| |---------| |---------|
  32. s |     .   | |     .   | |     .   | |     .   | |     .   |
  33. e       .           .           .           .           .
  34. t |     .   | |     .   | |     .   | |     .   | |     .   |
  35.   |---------| |---------| |---------| |---------| |---------|
  36. 1 |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  37.   |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 2097151
  38.   | 10485755| | 10485756| | 10485757| | 10485758| | 10485759|
  39.   \=========/ \=========/ \=========/ \=========/ \=========/
  40.  
  41.    _________   _________   _________   _________   _________
  42.   /object 10\ /object 11\ /object 12\ /object 13\ /object 14\
  43.   +=========+ +=========+ +=========+ +=========+ +=========+
  44.   |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  45. o |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 2097152
  46. b | 10485760| | 10485761| | 10485762| | 10485763| | 10485764|
  47. j |---------| |---------| |---------| |---------| |---------|
  48. e |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  49. c |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 2097153
  50. t | 10485765| | 10485766| | 10485767| | 10485768| | 10485769|
  51.   |---------| |---------| |---------| |---------| |---------|
  52. s |     .   | |     .   | |     .   | |     .   | |     .   |
  53. e       .           .           .           .           .
  54. t |     .   | |     .   | |     .   | |     .   | |     .   |
  55.   |---------| |---------| |---------| |---------| |---------|
  56. 2 |  stripe | |  stripe | |  stripe | |  stripe | |  stripe | stripe
  57.   |   unit  | |   unit  | |   unit  | |   unit  | |   unit  | 3051756
  58.   | 15258780| | 15258781| | 15258782| | 15258783| | 15258784|
  59.   |---------| |---------| |---------| |---------| |---------|
  60.   |  stripe | |  stripe | |  stripe | |  stripe | | (partial| (partial
  61.   |   unit  | |   unit  | |   unit  | |   unit  | |  stripe | stripe
  62.   | 15258785| | 15258786| | 15258787| | 15258788| |  unit)  | 3051757)
  63.   \=========/ \=========/ \=========/ \=========/ \=========/