Post data to jobs API

Post data to jobs API

Deprecated in 7.11.0.

Posting data directly to anomaly detection jobs is deprecated, in a future major version a datafeed will be required.

New API reference

For the most up-to-date API details, refer to Machine learning anomaly detection APIs.

Sends data to an anomaly detection job for analysis.

Request

POST _ml/anomaly_detectors/<job_id>/_data

Prerequisites

Requires the manage_ml cluster privilege. This privilege is included in the machine_learning_admin built-in role.

Description

The job must have a state of open to receive and process the data.

The data that you send to the job must use the JSON format. Multiple JSON documents can be sent, either adjacent with no separator in between them or whitespace separated. Newline delimited JSON (NDJSON) is a possible whitespace separated format, and for this the Content-Type header should be set to application/x-ndjson.

Upload sizes are limited to the Elasticsearch HTTP receive buffer size (default 100 Mb). If your data is larger, split it into multiple chunks and upload each one separately in sequential time order. When running in real time, it is generally recommended that you perform many small uploads, rather than queueing data to upload larger files.

When uploading data, check the job data counts for progress. The following documents will not be processed:

  • Documents not in chronological order and outside the latency window
  • Records with an invalid timestamp

For each job, data can only be accepted from a single connection at a time. It is not currently possible to post data to multiple jobs using wildcards or a comma-separated list.

Path parameters

<job_id>

(Required, string) Identifier for the anomaly detection job.

Query parameters

reset_start

(Optional, string) Specifies the start of the bucket resetting range.

reset_end

(Optional, string) Specifies the end of the bucket resetting range.

Request body

A sequence of one or more JSON documents containing the data to be analyzed. Only whitespace characters are permitted in between the documents.

Examples

The following example posts data from the it_ops_new_kpi.json file to the it_ops_new_kpi job:

  1. $ curl -s -H "Content-type: application/json"
  2. -X POST http:\/\/localhost:9200/_ml/anomaly_detectors/it_ops_new_kpi/_data
  3. --data-binary @it_ops_new_kpi.json

When the data is sent, you receive information about the operational progress of the job. For example:

  1. {
  2.     "job_id":"it_ops_new_kpi",
  3.     "processed_record_count":21435,
  4.     "processed_field_count":64305,
  5.     "input_bytes":2589063,
  6.     "input_field_count":85740,
  7.     "invalid_date_count":0,
  8.     "missing_field_count":0,
  9.     "out_of_order_timestamp_count":0,
  10.     "empty_bucket_count":16,
  11.     "sparse_bucket_count":0,
  12.     "bucket_count":2165,
  13.     "earliest_record_timestamp":1454020569000,
  14.     "latest_record_timestamp":1455318669000,
  15.     "last_data_time":1491952300658,
  16.     "latest_empty_bucket_timestamp":1454541600000,
  17.     "input_record_count":21435
  18. }

For more information about these properties, see Response body.