Create data frame analytics jobs API

Instantiates a data frame analytics job.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Request

PUT _ml/data_frame/analytics/<data_frame_analytics_id>

Prerequisites

If the Elasticsearch security features are enabled, you must have the following built-in roles and privileges:

  • machine_learning_admin
  • source indices: read, view_index_metadata
  • destination index: read, create_index, manage and index

For more information, see Built-in roles, Security privileges, and Machine learning security privileges.

The data frame analytics job remembers which roles the user who created it had at the time of creation. When you start the job, it performs the analysis using those same roles. If you provide secondary authorization headers, those credentials are used instead.

Description

This API creates a data frame analytics job that performs an analysis on the source indices and stores the outcome in a destination index.

If the destination index does not exist, it is created automatically when you start the job. See Start data frame analytics jobs.

If you supply only a subset of the regression or classification parameters, hyperparameter optimization occurs. It determines a value for each of the undefined parameters.

Path parameters

<data_frame_analytics_id>

(Required, string) Identifier for the data frame analytics job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Request body

allow_lazy_start

(Optional, boolean) Specifies whether this job can start when there is insufficient machine learning node capacity for it to be immediately assigned to a node. The default is false; if a machine learning node with capacity to run the job cannot immediately be found, the Start data frame analytics jobs API returns an error. However, this is also subject to the cluster-wide xpack.ml.max_lazy_ml_nodes setting. See Advanced machine learning settings. If this option is set to true, the API does not return an error and the job waits in the starting state until sufficient machine learning node capacity is available.

analysis

(Required, object) The analysis configuration, which contains the information necessary to perform one of the following types of analysis: classification, outlier detection, or regression.

Properties of analysis

  • classification

    (Required*, object) The configuration information necessary to perform classification.

    Advanced parameters are for fine-tuning classification analysis. They are set automatically by hyperparameter optimization to give the minimum validation error. It is highly recommended to use the default values unless you fully understand the function of these parameters.

    Properties of classification

    • class_assignment_objective

      (Optional, string) Defines the objective to optimize when assigning class labels: maximize_accuracy or maximize_minimum_recall. When maximizing accuracy, class labels are chosen to maximize the number of correct predictions. When maximizing minimum recall, labels are chosen to maximize the minimum recall for any class. Defaults to maximize_minimum_recall.

      dependent_variable

      (Required, string)

      Defines which field of the document is to be predicted. This parameter is supplied by field name and must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable.

      The data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field.

      eta

      (Optional, double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, the smaller the value the longer the training will take. For more information, about shrinkage, see this wiki article. By default, this value is calcuated during hyperparameter optimization.

      feature_bag_fraction

      (Optional, double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

      gamma

      (Optional, double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. The higher the value the more training will prefer smaller trees. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.

      lambda

      (Optional, double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularisation term which applies to leaf weights of the individual trees in the forest. The higher the value the more training will attempt to keep leaf weights small. This makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.

      max_trees

      (Optional, integer) Advanced configuration option. Defines the maximum number of trees the forest is allowed to contain. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

      num_top_classes

      (Optional, integer) Defines the number of categories for which the predicted probabilities are reported. It must be non-negative. If it is greater than the total number of categories, the API reports all category probabilities. Defaults to 2.

      num_top_feature_importance_values

      (Optional, integer) Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, it is zero and no feature importance calculation occurs.

      prediction_field_name

      (Optional, string) Defines the name of the prediction field in the results. Defaults to <dependent_variable>_prediction.

      randomize_seed

      (Optional, long) Defines the seed to the random generator that is used to pick which documents will be used for training. By default it is randomly generated. Set it to a specific value to ensure the same documents are used for training assuming other related parameters (e.g. source, analyzed_fields, etc.) are the same.

      training_percent

      (Optional, integer) Defines what percentage of the eligible documents that will be used for training. Documents that are ignored by the analysis (for example those that contain arrays with more than one value) won’t be included in the calculation for used percentage. Defaults to 100.

  1. `outlier_detection`
  3. (Required\*, object) The configuration information necessary to perform [outlier detection](https://www.elastic.co/guide/en/machine-learning/current/dfa-outlier-detection.html):
  5. Properties of `outlier_detection`
  7. -   `compute_feature_influence`
  9.     (Optional, boolean) Specifies whether the feature influence calculation is enabled. Defaults to `true`.
  11.     `feature_influence_threshold`
  13.     (Optional, double) The minimum outlier score that a document needs to have to calculate its feature influence score. Value range: 0-1 (`0.1` by default).
  15.     `method`
  17.     (Optional, string) The method that outlier detection uses. Available methods are `lof`, `ldof`, `distance_kth_nn`, `distance_knn`, and `ensemble`. The default value is `ensemble`, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.
  19.     `n_neighbors`
  21.     (Optional, integer) Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This deafault behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.
  23.     `outlier_fraction`
  25.     (Optional, double) The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.
  27.     `standardization_enabled`
  29.     (Optional, boolean) If `true`, the following operation is performed on the columns before computing outlier scores: (x\_i - mean(x\_i)) / sd(x\_i). Defaults to `true`. For more information about this concept, see [Wikipedia](https://en.wikipedia.org/wiki/Feature_scaling#Standardization_(Z-score_Normalization)).
  32. `regression`
  34. (Required\*, object) The configuration information necessary to perform [regression](https://www.elastic.co/guide/en/machine-learning/current/dfa-regression.html).
  36. Advanced parameters are for fine-tuning regression analysis. They are set automatically by hyperparameter optimization to give minimum validation error. It is highly recommended to use the default values unless you fully understand the function of these parameters.
  38. Properties of `regression`
  40. -   `dependent_variable`
  42.     (Required, string)
  44.     Defines which field of the document is to be predicted. This parameter is supplied by field name and must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable.
  46.     The data type of the field must be numeric.
  48.     `eta`
  50.     (Optional, double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, the smaller the value the longer the training will take. For more information, about shrinkage, see [this wiki article](https://en.wikipedia.org/wiki/Gradient_boosting#Shrinkage). By default, this value is calcuated during hyperparameter optimization.
  52.     `feature_bag_fraction`
  54.     (Optional, double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.
  56.     `gamma`
  58.     (Optional, double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. The higher the value the more training will prefer smaller trees. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.
  60.     `lambda`
  62.     (Optional, double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularisation term which applies to leaf weights of the individual trees in the forest. The higher the value the more training will attempt to keep leaf weights small. This makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.
  64.     `loss_function`
  66.     (Optional, string) The loss function used during regression. Available options are `mse` (mean squared error), `msle` (mean squared logarithmic error), `huber` (Pseudo-Huber loss). Defaults to `mse`. Refer to [Loss functions for regression analyses](https://www.elastic.co/guide/en/machine-learning/current/dfa-regression.html#dfa-regression-lossfunction) to learn more.
  68.     `loss_function_parameter`
  70.     (Optional, double) A positive number that is used as a parameter to the `loss_function`.
  72.     `max_trees`
  74.     (Optional, integer) Advanced configuration option. Defines the maximum number of trees the forest is allowed to contain. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.
  76.     `num_top_feature_importance_values`
  78.     (Optional, integer) Advanced configuration option. Specifies the maximum number of [feature importance](https://www.elastic.co/guide/en/machine-learning/current/ml-feature-importance.html) values per document to return. By default, it is zero and no feature importance calculation occurs.
  80.     `prediction_field_name`
  82.     (Optional, string) Defines the name of the prediction field in the results. Defaults to `<dependent_variable>_prediction`.
  84.     `randomize_seed`
  86.     (Optional, long) Defines the seed to the random generator that is used to pick which documents will be used for training. By default it is randomly generated. Set it to a specific value to ensure the same documents are used for training assuming other related parameters (e.g. `source`, `analyzed_fields`, etc.) are the same.
  88.     `training_percent`
  90.     (Optional, integer) Defines what percentage of the eligible documents that will be used for training. Documents that are ignored by the analysis (for example those that contain arrays with more than one value) won’t be included in the calculation for used percentage. Defaults to `100`.

analyzed_fields

(Optional, object) Specify includes and/or excludes patterns to select which fields will be included in the analysis. The patterns specified in excludes are applied last, therefore excludes takes precedence. In other words, if the same field is specified in both includes and excludes, then the field will not be included in the analysis.

The supported fields for each type of analysis are as follows:

  • Outlier detection requires numeric or boolean data to analyze. The algorithms don’t support missing values therefore fields that have data types other than numeric or boolean are ignored. Documents where included fields contain missing values, null values, or an array are also ignored. Therefore the dest index may contain documents that don’t have an outlier score.
  • Regression supports fields that are numeric, boolean, text, keyword, and ip. It is also tolerant of missing values. Fields that are supported are included in the analysis, other fields are ignored. Documents where included fields contain an array with two or more values are also ignored. Documents in the dest index that don’t contain a results field are not included in the regression analysis.
  • Classification supports fields that are numeric, boolean, text, keyword, and ip. It is also tolerant of missing values. Fields that are supported are included in the analysis, other fields are ignored. Documents where included fields contain an array with two or more values are also ignored. Documents in the dest index that don’t contain a results field are not included in the classification analysis. Classification analysis can be improved by mapping ordinal variable values to a single number. For example, in case of age ranges, you can model the values as “0-14” = 0, “15-24” = 1, “25-34” = 2, and so on.

If analyzed_fields is not set, only the relevant fields will be included. For example, all the numeric fields for outlier detection. For more information about field selection, see Explain data frame analytics API.

Properties of analyzed_fields

  • excludes

    (Optional, array) An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

    includes

    (Optional, array) An array of strings that defines the fields that will be included in the analysis.

description

(Optional, string) A description of the job.

dest

(Required, object) The destination configuration, consisting of index and optionally results_field (ml by default).

Properties of dest

  • index

    (Required, string) Defines the destination index to store the results of the data frame analytics job.

    results_field

    (Optional, string) Defines the name of the field in which to store the results of the analysis. Defaults to ml.

max_num_threads

(Optional, integer) The maximum number of threads to be used by the analysis. The default value is 1. Using more threads may decrease the time necessary to complete the analysis at the cost of using more CPU. Note that the process may use additional threads for operational functionality other than the analysis itself.

model_memory_limit

(Optional, string) The approximate maximum amount of memory resources that are permitted for analytical processing. The default value for data frame analytics jobs is 1gb. If your elasticsearch.yml file contains an xpack.ml.max_model_memory_limit setting, an error occurs when you try to create data frame analytics jobs that have model_memory_limit values greater than that setting. For more information, see Machine learning settings.

source

(object) The configuration of how to source the analysis data. It requires an index. Optionally, query and _source may be specified.

Properties of source

  • index

    (Required, string or array) Index or indices on which to perform the analysis. It can be a single index or index pattern as well as an array of indices or patterns.

    If your source indices contain documents with the same IDs, only the document that is indexed last appears in the destination index.

    query

    (Optional, object) The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

    _source

    (Optional, object) Specify includes and/or excludes patterns to select which fields will be present in the destination. Fields that are excluded cannot be included in the analysis.

    Properties of _source

    • includes

      (array) An array of strings that defines the fields that will be included in the destination.

      excludes

      (array) An array of strings that defines the fields that will be excluded from the destination.

Examples

Preprocessing actions example

The following example shows how to limit the scope of the analysis to certain fields, specify excluded fields in the destination index, and use a query to filter your data before analysis.

  1. PUT _ml/data_frame/analytics/model-flight-delays-pre
  2. {
  3.   "source": {
  4.     "index": [
  5.       "kibana_sample_data_flights" 
  6.     ],
  7.     "query": { 
  8.       "range": {
  9.         "DistanceKilometers": {
  10.           "gt": 0
  11.         }
  12.       }
  13.     },
  14.     "_source": { 
  15.       "includes": [],
  16.       "excludes": [
  17.         "FlightDelay",
  18.         "FlightDelayType"
  19.       ]
  20.     }
  21.   },
  22.   "dest": { 
  23.     "index": "df-flight-delays",
  24.     "results_field": "ml-results"
  25.   },
  26.   "analysis": {
  27.   "regression": {
  28.     "dependent_variable": "FlightDelayMin",
  29.     "training_percent": 90
  30.     }
  31.   },
  32.   "analyzed_fields": { 
  33.     "includes": [],
  34.     "excludes": [
  35.       "FlightNum"
  36.     ]
  37.   },
  38.   "model_memory_limit": "100mb"
  39. }

Source index to analyze.

This query filters out entire documents that will not be present in the destination index.

The _source object defines fields in the dataset that will be included or excluded in the destination index.

Defines the destination index that contains the results of the analysis and the fields of the source index specified in the _source object. Also defines the name of the results_field.

Specifies fields to be included in or excluded from the analysis. This does not affect whether the fields will be present in the destination index, only affects whether they are used in the analysis.

In this example, we can see that all the fields of the source index are included in the destination index except FlightDelay and FlightDelayType because these are defined as excluded fields by the excludes parameter of the _source object. The FlightNum field is included in the destination index, however it is not included in the analysis because it is explicitly specified as excluded field by the excludes parameter of the analyzed_fields object.

Outlier detection example

The following example creates the loganalytics data frame analytics job, the analysis type is outlier_detection:

  1. PUT _ml/data_frame/analytics/loganalytics
  2. {
  3.   "description": "Outlier detection on log data",
  4.   "source": {
  5.     "index": "logdata"
  6.   },
  7.   "dest": {
  8.     "index": "logdata_out"
  9.   },
  10.   "analysis": {
  11.     "outlier_detection": {
  12.       "compute_feature_influence": true,
  13.       "outlier_fraction": 0.05,
  14.       "standardization_enabled": true
  15.     }
  16.   }
  17. }

The API returns the following result:

  1. {
  2.   "id": "loganalytics",
  3.   "description": "Outlier detection on log data",
  4.   "source": {
  5.     "index": ["logdata"],
  6.     "query": {
  7.       "match_all": {}
  8.     }
  9.   },
  10.   "dest": {
  11.     "index": "logdata_out",
  12.     "results_field": "ml"
  13.   },
  14.   "analysis": {
  15.     "outlier_detection": {
  16.       "compute_feature_influence": true,
  17.       "outlier_fraction": 0.05,
  18.       "standardization_enabled": true
  19.     }
  20.   },
  21.   "model_memory_limit": "1gb",
  22.   "create_time" : 1562265491319,
  23.   "version" : "7.6.0",
  24.   "allow_lazy_start" : false,
  25.   "max_num_threads": 1
  26. }

Regression examples

The following example creates the house_price_regression_analysis data frame analytics job, the analysis type is regression:

  1. PUT _ml/data_frame/analytics/house_price_regression_analysis
  2. {
  3.   "source": {
  4.     "index": "houses_sold_last_10_yrs"
  5.   },
  6.   "dest": {
  7.     "index": "house_price_predictions"
  8.   },
  9.   "analysis":
  10.     {
  11.       "regression": {
  12.         "dependent_variable": "price"
  13.       }
  14.     }
  15. }

The API returns the following result:

  1. {
  2.   "id" : "house_price_regression_analysis",
  3.   "source" : {
  4.     "index" : [
  5.       "houses_sold_last_10_yrs"
  6.     ],
  7.     "query" : {
  8.       "match_all" : { }
  9.     }
  10.   },
  11.   "dest" : {
  12.     "index" : "house_price_predictions",
  13.     "results_field" : "ml"
  14.   },
  15.   "analysis" : {
  16.     "regression" : {
  17.       "dependent_variable" : "price",
  18.       "training_percent" : 100
  19.     }
  20.   },
  21.   "model_memory_limit" : "1gb",
  22.   "create_time" : 1567168659127,
  23.   "version" : "8.0.0",
  24.   "allow_lazy_start" : false
  25. }

The following example creates a job and specifies a training percent:

  1. PUT _ml/data_frame/analytics/student_performance_mathematics_0.3
  2. {
  3.  "source": {
  4.    "index": "student_performance_mathematics"
  5.  },
  6.  "dest": {
  7.    "index":"student_performance_mathematics_reg"
  8.  },
  9.  "analysis":
  10.    {
  11.      "regression": {
  12.        "dependent_variable": "G3",
  13.        "training_percent": 70,  
  14.        "randomize_seed": 19673948271  
  15.      }
  16.    }
  17. }

The percentage of the data set that is used for training the model.

The seed that is used to randomly pick which data is used for training.

Classification example

The following example creates the loan_classification data frame analytics job, the analysis type is classification:

  1. PUT _ml/data_frame/analytics/loan_classification
  2. {
  3.   "source" : {
  4.     "index": "loan-applicants"
  5.   },
  6.   "dest" : {
  7.     "index": "loan-applicants-classified"
  8.   },
  9.   "analysis" : {
  10.     "classification": {
  11.       "dependent_variable": "label",
  12.       "training_percent": 75,
  13.       "num_top_classes": 2
  14.     }
  15.   }
  16. }