Manual compaction

Manual compaction

In Apache Druid, compaction is a special type of ingestion task that reads data from a Druid datasource and writes it back into the same datasource. A common use case for this is to optimally size segments after ingestion to improve query performance.

You can perform manual compaction where you submit a one-time compaction task for a specific interval. Generally, you don’t need to do this if you use automatic compaction, which is recommended for most workloads.

Setting up manual compaction

Compaction tasks merge all segments for the defined interval according to the following syntax:

{
    "type": "compact",
    "id": <task_id>,
    "dataSource": <task_datasource>,
    "ioConfig": <IO config>,
    "dimensionsSpec": <custom dimensionsSpec>,
    "transformSpec": <custom transformSpec>,
    "metricsSpec": <custom metricsSpec>,
    "tuningConfig": <parallel indexing task tuningConfig>,
    "granularitySpec": <compaction task granularitySpec>,
    "context": <task context>
}

Field	Description	Required
`type`	Task type. Set the value to `compact`.	Yes
`id`	Task ID	No
`dataSource`	Data source name to compact	Yes
`ioConfig`	I/O configuration for compaction task. See Compaction I/O configuration for details.	Yes
`dimensionsSpec`	When set, the compaction task uses the specified `dimensionsSpec` rather than generating one from existing segments. See Compaction dimensionsSpec for details.	No
`transformSpec`	When set, the compaction task uses the specified `transformSpec` rather than using `null`. See Compaction transformSpec for details.	No
`metricsSpec`	When set, the compaction task uses the specified `metricsSpec` rather than generating one from existing segments.	No
`segmentGranularity`	Deprecated. Use `granularitySpec`.	No
`tuningConfig`	Tuning configuration for parallel indexing. `awaitSegmentAvailabilityTimeoutMillis` value is not supported for compaction tasks. Leave this parameter at the default value, 0.	No
`granularitySpec`	When set, the compaction task uses the specified `granularitySpec` rather than generating one from existing segments. See Compaction granularitySpec for details.	No
`context`	Task context	No

info

Note: Use granularitySpec over segmentGranularity and only set one of these values. If you specify different values for these in the same compaction spec, the task fails.

To control the number of result segments per time chunk, you can set maxRowsPerSegment or numShards.

info

You can run multiple compaction tasks in parallel. For example, if you want to compact the data for a year, you are not limited to running a single task for the entire year. You can run 12 compaction tasks with month-long intervals.

A compaction task internally generates an index or index_parallel task spec for performing compaction work with some fixed parameters. For example, its inputSource is always the druid input source, and dimensionsSpec and metricsSpec include all dimensions and metrics of the input segments by default.

Compaction tasks typically fetch all relevant segments prior to launching any subtasks, unless the following properties are all set to non-null values. It is strongly recommended to set them to non-null values to maximize performance and minimize disk usage of the compact task:

granularitySpec, with non-null values for each of segmentGranularity, queryGranularity, and rollup
dimensionsSpec
metricsSpec

Compaction tasks exit without doing anything and issue a failure status code in either of the following cases:

If the interval you specify has no data segments loaded.
If the interval you specify is empty.

Note that the metadata between input segments and the resulting compacted segments may differ if the metadata among the input segments differs as well. If all input segments have the same metadata, however, the resulting output segment will have the same metadata as all input segments.