Get data frame analytics jobs statistics API

Retrieves usage information for data frame analytics jobs.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Request

GET _ml/data_frame/analytics/<data_frame_analytics_id>/_stats

GET _ml/data_frame/analytics/<data_frame_analytics_id>,<data_frame_analytics_id>/_stats

GET _ml/data_frame/analytics/_stats

GET _ml/data_frame/analytics/_all/_stats

GET _ml/data_frame/analytics/*/_stats

Prerequisites

If the Elasticsearch security features are enabled, you must have the following privileges:

  • cluster: monitor_ml

For more information, see Security privileges and Machine learning security privileges.

Path parameters

<data_frame_analytics_id>

(Optional, string) Identifier for the data frame analytics job. If you do not specify this option, the API returns information for the first hundred data frame analytics jobs.

Query parameters

allow_no_match

(Optional, boolean) Specifies what to do when the request:

  • Contains wildcard expressions and there are no data frame analytics jobs that match.
  • Contains the _all string or no identifiers and there are no matches.
  • Contains wildcard expressions and there are only partial matches.

The default value is true, which returns an empty data_frame_analytics array when there are no matches and the subset of results when there are partial matches. If this parameter is false, the request returns a 404 status code when there are no matches or only partial matches.

from

(Optional, integer) Skips the specified number of data frame analytics jobs. The default value is 0.

size

(Optional, integer) Specifies the maximum number of data frame analytics jobs to obtain. The default value is 100.

Response body

data_frame_analytics

(array) An array of objects that contain usage information for data frame analytics jobs, which are sorted by the id value in ascending order.

Properties of data frame analytics job usage resources

  • analysis_stats

    (object) An object containing information about the analysis job.

    Properties of analysis_stats

    • classification_stats

      (object) An object containing information about the classification analysis job.

      Properties of classification_stats

      hyperparameters

      (object) An object containing the parameters of the classification analysis job.

      Properties of hyperparameters

      alpha

      (double) Regularization factor to penalize deeper trees when training decision trees.

      class_assignment_objective

      (string) Defines the objective to optimize when assigning class labels: maximize_accuracy or maximize_minimum_recall. When maximizing accuracy, class labels are chosen to maximize the number of correct predictions. When maximizing minimum recall, labels are chosen to maximize the minimum recall for any class. Defaults to maximize_minimum_recall.

      downsample_factor

      (double) The value of the downsample factor.

      eta

      (double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, the smaller the value the longer the training will take. For more information, about shrinkage, see this wiki article. By default, this value is calcuated during hyperparameter optimization.

      eta_growth_rate_per_tree

      (double) Specifies the rate at which the eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5%.

      feature_bag_fraction

      (double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

      gamma

      (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. The higher the value the more training will prefer smaller trees. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.

      lambda

      (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularisation term which applies to leaf weights of the individual trees in the forest. The higher the value the more training will attempt to keep leaf weights small. This makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.

      max_attempts_to_add_tree

      (integer) If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.

      max_optimization_rounds_per_hyperparameter

      (integer) A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter.

      max_trees

      (integer) Advanced configuration option. Defines the maximum number of trees the forest is allowed to contain. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

      num_folds

      (integer) The maximum number of folds for the cross-validation procedure.

      num_splits_per_feature

      (integer) Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.

      soft_tree_depth_limit

      (double) Tree depth limit is used for calculating the tree depth penalty. This is a soft limit, it can be exceeded.

      soft_tree_depth_tolerance

      (double) Tree depth tolerance is used for calculating the tree depth penalty. This is a soft limit, it can be exceeded.

      iteration

      (integer) The number of iterations on the analysis.

      timestamp

      (date) The timestamp when the statistics were reported in milliseconds since the epoch.

      timing_stats

      (object) An object containing time statistics about the data frame analytics job.

      Properties of timing_stats

      elapsed_time

      (integer) Runtime of the analysis in milliseconds.

      iteration_time

      (integer) Runtime of the latest iteration of the analysis in milliseconds.

      validation_loss

      (object) An object containing information about validation loss.

      Properties of validation_loss

      fold_values

      (array of strings) Validation loss values for every added decision tree during the forest growing procedure.

      loss_type

      (string) The type of the loss metric. For example, binomial_logistic.

  1. -   `outlier_detection_stats`
  3.     (object) An object containing information about the outlier detection job.
  5.     Properties of `outlier_detection_stats`
  7.     `parameters`
  9.     (object) The list of job parameters specified by the user or determined by algorithmic heuristics.
  11.     Properties of `parameters`
  13.     `compute_feature_influence`
  15.     (boolean) Specifies whether the feature influence calculation is enabled. Defaults to `true`.
  17.     `feature_influence_threshold`
  19.     (double) The minimum outlier score that a document needs to have to calculate its feature influence score. Value range: 0-1 (`0.1` by default).
  21.     `method`
  23.     (string) The method that outlier detection uses. Available methods are `lof`, `ldof`, `distance_kth_nn`, `distance_knn`, and `ensemble`. The default value is `ensemble`, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.
  25.     `n_neighbors`
  27.     (integer) Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This deafault behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.
  29.     `outlier_fraction`
  31.     (double) The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.
  33.     `standardization_enabled`
  35.     (boolean) If `true`, the following operation is performed on the columns before computing outlier scores: (x\_i - mean(x\_i)) / sd(x\_i). Defaults to `true`. For more information about this concept, see [Wikipedia](https://en.wikipedia.org/wiki/Feature_scaling#Standardization_(Z-score_Normalization)).
  37.     `timestamp`
  39.     (date) The timestamp when the statistics were reported in milliseconds since the epoch.
  41.     `timing_stats`
  43.     (object) An object containing time statistics about the data frame analytics job.
  45.     Property of `timing_stats`
  47.     `elapsed_time`
  49.     (integer) Runtime of the analysis in milliseconds.
  52. -   `regression_stats`
  54.     (object) An object containing information about the regression analysis.
  56.     Properties of `regression_stats`
  58.     `hyperparameters`
  60.     (object) An object containing the parameters of the regression analysis.
  62.     Properties of `hyperparameters`
  64.     `alpha`
  66.     (double) Regularization factor to penalize deeper trees when training decision trees.
  68.     `downsample_factor`
  70.     (double) The value of the downsample factor.
  72.     `eta`
  74.     (double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, the smaller the value the longer the training will take. For more information, about shrinkage, see [this wiki article](https://en.wikipedia.org/wiki/Gradient_boosting#Shrinkage). By default, this value is calcuated during hyperparameter optimization.
  76.     `eta_growth_rate_per_tree`
  78.     (double) Specifies the rate at which the `eta` increases for each new tree that is added to the forest. For example, a rate of `1.05` increases `eta` by 5%.
  80.     `feature_bag_fraction`
  82.     (double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.
  84.     `gamma`
  86.     (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. The higher the value the more training will prefer smaller trees. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.
  88.     `lambda`
  90.     (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularisation term which applies to leaf weights of the individual trees in the forest. The higher the value the more training will attempt to keep leaf weights small. This makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. The smaller this parameter the larger individual trees will be and the longer training will take. By default, this value is calculated during hyperparameter optimization.
  92.     `max_attempts_to_add_tree`
  94.     (integer) If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.
  96.     `max_optimization_rounds_per_hyperparameter`
  98.     (integer) A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter.
  100.     `max_trees`
  102.     (integer) Advanced configuration option. Defines the maximum number of trees the forest is allowed to contain. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.
  104.     `num_folds`
  106.     (integer) The maximum number of folds for the cross-validation procedure.
  108.     `num_splits_per_feature`
  110.     (integer) Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.
  112.     `soft_tree_depth_limit`
  114.     (double) Tree depth limit is used for calculating the tree depth penalty. This is a soft limit, it can be exceeded.
  116.     `soft_tree_depth_tolerance`
  118.     (double) Tree depth tolerance is used for calculating the tree depth penalty. This is a soft limit, it can be exceeded.
  120.     `iteration`
  122.     (integer) The number of iterations on the analysis.
  124.     `timestamp`
  126.     (date) The timestamp when the statistics were reported in milliseconds since the epoch.
  128.     `timing_stats`
  130.     (object) An object containing time statistics about the data frame analytics job.
  132.     Propertis of `timing_stats`
  134.     `elapsed_time`
  136.     (integer) Runtime of the analysis in milliseconds.
  138.     `iteration_time`
  140.     (integer) Runtime of the latest iteration of the analysis in milliseconds.
  142.     `validation_loss`
  144.     (object) An object containing information about validation loss.
  146.     Properties of `validation_loss`
  148.     `fold_values`
  150.     (array of strings) Validation loss values for every added decision tree during the forest growing procedure.
  152.     `loss_type`
  154.     (string) The type of the loss metric. For example, `binomial_logistic`.
  157. `assignment_explanation`
  159. (string) For running jobs only, contains messages relating to the selection of a node to run the job.

  • data_counts

    (object) An object that provides counts for the quantity of documents skipped, used in training, or available for testing.

    Properties of data_counts

    • skipped_docs_count

      (integer) The number of documents that are skipped during the analysis because they contained values that are not supported by the analysis. For example, outlier detection does not support missing fields so it skips documents with missing fields. Likewise, all types of analysis skip documents that contain arrays with more than one element.

      test_docs_count

      (integer) The number of documents that are not used for training the model and can be used for testing.

      training_docs_count

      (integer) The number of documents that are used for training the model.

  1. `id`
  3. (string) The unique identifier of the data frame analytics job.
  5. `memory_usage`
  7. (Optional, object) An object describing memory usage of the analytics. It is present only after the job is started and memory usage is reported.
  9. Properties of `memory_usage`
  11. -   `memory_reestimate_bytes`
  13.     (long) This value is present when the `status` is `hard_limit` and it is a new estimate of how much memory the job needs.
  15.     `peak_usage_bytes`
  17.     (long) The number of bytes used at the highest peak of memory usage.
  19.     `status`
  21.     (string) The memory usage status. May have one of the following values:
  23.     -   `ok`: usage stayed below the limit.
  24.     -   `hard_limit`: usage surpassed the configured memory limit.
  26.     `timestamp`
  28.     (date) The timestamp when memory usage was calculated.
  31. `node`
  33. (object) Contains properties for the node that runs the job. This information is available only for running jobs.
  35. Properties of `node`
  37. -   `attributes`
  39.     (object) Lists node attributes such as `ml.machine_memory` or `ml.max_open_jobs` settings.
  41.     `ephemeral_id`
  43.     (string) The ephemeral ID of the node.
  45.     `id`
  47.     (string) The unique identifier of the node.
  49.     `name`
  51.     (string) The node name.
  53.     `transport_address`
  55.     (string) The host and port where transport HTTP connections are accepted.
  58. `progress`
  60. (array) The progress report of the data frame analytics job by phase.
  62. Properties of phase objects
  64. -   `phase`
  66.     (string) Defines the phase of the data frame analytics job. Possible phases:
  68.     -   `reindexing`,
  69.     -   `loading_data`,
  70.     -   `computing_outliers` (for outlier detection only),
  71.     -   `feature_selection` (for regression and classification only),
  72.     -   `coarse_parameter_search` (for regression and classification only),
  73.     -   `fine_tuning_parameters` (for regression and classification only),
  74.     -   `final_training` (for regression and classification only),
  75.     -   `writing_results`,
  76.     -   `inference` (for regression and classification only).
  78.         To learn more about the different phases, refer to [How a data frame analytics job works](https://www.elastic.co/guide/en/machine-learning/current/ml-dfa-phases.html).
  81.     `progress_percent`
  83.     (integer) The progress that the data frame analytics job has made expressed in percentage.
  86. `state`
  88. (string) The status of the data frame analytics job, which can be one of the following values: `analyzing`, `failed`, `reindexing`, `started`, `starting`, `stopped`, `stopping`.

Response codes

404 (Missing resources)

If allow_no_match is false, this code indicates that there are no resources that match the request or only partial matches for the request.

Examples

The following API retrieves usage information for the outlier detection data frame analytics job example:

  1. GET _ml/data_frame/analytics/ecommerce/_stats

The API returns the following results:

  1. {
  2.   "count" : 1,
  3.   "data_frame_analytics" : [
  4.     {
  5.       "id" : "ecommerce",
  6.       "state" : "stopped",
  7.       "progress" : [
  8.         {
  9.           "phase" : "reindexing",
  10.           "progress_percent" : 100
  11.         },
  12.         {
  13.           "phase" : "loading_data",
  14.           "progress_percent" : 100
  15.         },
  16.         {
  17.           "phase" : "analyzing",
  18.           "progress_percent" : 100
  19.         },
  20.         {
  21.           "phase" : "writing_results",
  22.           "progress_percent" : 100
  23.         }
  24.       ],
  25.       "data_counts" : {
  26.         "training_docs_count" : 3321,
  27.         "test_docs_count" : 0,
  28.         "skipped_docs_count" : 0
  29.       },
  30.       "memory_usage" : {
  31.         "timestamp" : 1586905058000,
  32.         "peak_usage_bytes" : 279484
  33.       },
  34.       "analysis_stats" : {
  35.         "outlier_detection_stats" : {
  36.           "timestamp" : 1586905058000,
  37.           "parameters" : {
  38.             "n_neighbors" : 0,
  39.             "method" : "ensemble",
  40.             "compute_feature_influence" : true,
  41.             "feature_influence_threshold" : 0.1,
  42.             "outlier_fraction" : 0.05,
  43.             "standardization_enabled" : true
  44.           },
  45.           "timing_stats" : {
  46.             "elapsed_time" : 245
  47.           }
  48.         }
  49.       }
  50.     }
  51.   ]
  52. }