3.9. IO Queue

CouchDB has an internal subsystem that can prioritize IO associated with certain classes of operations. This subsystem can be configured to limit the resources devoted to background operations like internal replication and compaction according to the settings described below.

[ioq]

  • concurrency

    Specifies the maximum number of concurrent in-flight IO requests that the queueing system will submit:

    1. [ioq]
    2. concurrency = 10

  • ratio

    The fraction of the time that a background IO request will be selected over an interactive IO request when both queues are non-empty:

    1. [ioq]
    2. ratio = 0.01

[ioq.bypass]

System administrators can choose to submit specific classes of IO directly to the underlying file descriptor or OS process, bypassing the queues altogether. Installing a bypass can yield higher throughput and lower latency, but relinquishes some control over prioritization. The following classes are recognized:

  • os_process

    Messages on their way to an external process (e.g., couchjs).

  • read

    Disk IO fulfilling interactive read requests.

  • write

    Disk IO required to update a database.

  • view_update

    Disk IO required to update views and other secondary indexes.

  • shard_sync

    Disk IO issued by the background replication processes that fix any inconsistencies between shard copies.

  • compaction

    Disk IO issued by compaction jobs.

Without any configuration CouchDB will enqueue all classes of IO. The default.ini configuration file that ships with CouchDB activates a bypass for each of the interactive IO classes and only background IO goes into the queueing system:

  1. [ioq.bypass]
  2. os_process = true
  3. read = true
  4. write = true
  5. view_update = true
  6. shard_sync = false
  7. compaction = false

3.9.1. Recommendations

The default configuration protects against excessive IO from background operations like compaction disrupting the latency of interactive operations, while maximizing the overall IO throughput devoted to those interactive requests. There are certain situations where this configuration could be sub-optimal:

  • An administrator may want to devote a larger portion of the overall IO bandwidth to compaction in order to stay ahead of the incoming write load. In this it may be necessary to disable the bypass for write (to help with database compaction) and/or view_update (to help with view index compaction) and then increase the ratio to give compaction a higher priority.

  • A server with a large number of views that do not need to be comlpetely up-to-date may benefit from removing the bypass on view_update in order to optimize the latency for regular document read and write operations, and build the views during quieter periods.