corpora
- Example
- Configuration options

corpora

The corpora element contains all the document corpora used by the workload. You can use document corpora across workloads by copying and pasting any corpora definitions.

Example

The following example defines a single corpus called movies with 11658903 documents and 1544799789 uncompressed bytes:

  "corpora": [
    {
      "name": "movies",
      "documents": [
        {
          "source-file": "movies-documents.json",
          "document-count": 11658903, # Fetch document count from command line
          "uncompressed-bytes": 1544799789 # Fetch uncompressed bytes from command line
        }
      ]
    }
  ]

Configuration options

Use the following options with corpora.

Parameter	Required	Type	Description
`name`	Yes	String	The name of the document corpus. Because OpenSearch Benchmark uses this name in its directories, use only lowercase names without white spaces.
`documents`	Yes	JSON array	An array of document files.
`meta`	No	String	A mapping of key-value pairs with additional metadata for a corpus.

Each entry in the documents array consists of the following options.

Parameter	Required	Type	Description
`source-file`	Yes	String	The file name containing the corresponding documents for the workload. When using OpenSearch Benchmark locally, documents are contained in a JSON file. When providing a `base_url`, use a compressed file format: `.zip`, `.bz2`, `.gz`, `.tar`, `.tar.gz`, `.tgz`, or `.tar.bz2`. The compressed file must have one JSON file containing the name.
`document-count`	Yes	Integer	The number of documents in the `source-file`, which determines which client indexes correlate to which parts of the document corpus. Each N client receives an Nth of the document corpus. When using a source that contains a document with a parent-child relationship, specify the number of parent documents.
`base-url`	No	String	An http(s), Amazon Simple Storage Service (Amazon S3), or Google Cloud Storage URL that points to the root path where OpenSearch Benchmark can obtain the corresponding source file.
`source-format`	No	String	Defines the format OpenSearch Benchmark uses to interpret the data file specified in `source-file`. Only `bulk` is supported.
`compressed-bytes`	No	Integer	The size, in bytes, of the compressed source file, indicating how much data OpenSearch Benchmark downloads.
`uncompressed-bytes`	No	Integer	The size, in bytes, of the source file after decompression, indicating how much disk space the decompressed source file needs.
`target-index`	No	String	Defines the name of the index that the `bulk` operation should target. OpenSearch Benchmark automatically derives this value when only one index is defined in the `indices` element. The value of `target-index` is ignored when the `includes-action-and-meta-data` setting is `true`.
`target-type`	No	String	Defines the document type of the target index targeted in bulk operations. OpenSearch Benchmark automatically derives this value when only one index is defined in the `indices` element and the index has only one type. The value of `target-type` is ignored when the `includes-action-and-meta-data` setting is `true`.
`includes-action-and-meta-data`	No	Boolean	When set to `true`, indicates that the document’s file already contains an `action` line and a `meta-data` line. When `false`, indicates that the document’s file contains only documents. Default is `false`.
`meta`	No	String	A mapping of key-value pairs with additional metadata for a corpus.