Here we explain the sub-compaction that is used in both Leveled and Universal compactions.

Goal

The goal of sub-compaction is to speed up a compaction job by partitioning it among multiple threads.

When

It is employed when one of the following conditions hold:

• L0 -> Lo where o > 0
  • Why: L0->Lo cannot be run in parallel with another L0->Lo, hence partitioning is the only way to speed it up.
• Universal Compaction, except L0 -> L0.
• Manual Leveled Compaction: Li -> Lo where o > 0
  • Why: Manual compaction invoked by the user is usually not parallelized hence benefits from partitioning.Note: sub-compaction is disabled for leveled if it is not merged with any file from the target level Lo. Refer to Compaction::ShouldFormSubcompactions for details.

How

It currently is based on a heuristic that worked out well so far. The heuristic could be improved in many ways.

• Select boundaries based on the natural boundary of input levels/files.
  • first and last key of L0 files
  • first and last key of non-0, non-last levels
  • first key of each SST file of the last level
• Use Versions::ApproximateSize to estimate the data size in each boundary.
• Merge boundaries to eliminate empty and smaller-than-average ranges.
  • find the average size in each range
  • starting from beginning greedily merge adjacent ranges until their total size exceeds the average