Anomaly result mapping

When you select the Enable custom result index box on the Custom result index pane, the Anomaly Detection plugin will save the results to an index of your choosing. When the anomaly detector does not detect an anomaly, the result format is as follows:

  1. {
  2.   "detector_id": "kzcZ43wBgEQAbjDnhzGF",
  3.   "schema_version": 5,
  4.   "data_start_time": 1635898161367,
  5.   "data_end_time": 1635898221367,
  6.   "feature_data": [
  7.     {
  8.       "feature_id": "processing_bytes_max",
  9.       "feature_name": "processing bytes max",
  10.       "data": 2322
  11.     },
  12.     {
  13.       "feature_id": "processing_bytes_avg",
  14.       "feature_name": "processing bytes avg",
  15.       "data": 1718.6666666666667
  16.     },
  17.     {
  18.       "feature_id": "processing_bytes_min",
  19.       "feature_name": "processing bytes min",
  20.       "data": 1375
  21.     },
  22.     {
  23.       "feature_id": "processing_bytes_sum",
  24.       "feature_name": "processing bytes sum",
  25.       "data": 5156
  26.     },
  27.     {
  28.       "feature_id": "processing_time_max",
  29.       "feature_name": "processing time max",
  30.       "data": 31198
  31.     }
  32.   ],
  33.   "execution_start_time": 1635898231577,
  34.   "execution_end_time": 1635898231622,
  35.   "anomaly_score": 1.8124904404395776,
  36.   "anomaly_grade": 0,
  37.   "confidence": 0.9802940756605277,
  38.   "entity": [
  39.     {
  40.       "name": "process_name",
  41.       "value": "process_3"
  42.     }
  43.   ],
  44.   "model_id": "kzcZ43wBgEQAbjDnhzGF_entity_process_3",
  45.   "threshold": 1.2368549346675202
  46. }

copy

Response body fields

FieldDescription
detector_idA unique ID for identifying a detector.
schema_versionThe mapping version of the result index.
data_start_timeThe start of the detection range of the aggregated data.
data_end_timeThe end of the detection range of the aggregated data.
feature_dataAn array of the aggregated data points between the data_start_time and data_end_time.
execution_start_timeThe actual start time of the detector for a specific run that produces the anomaly result. This start time includes the window delay parameter that you can set to delay data collection. Window delay is the difference between the execution_start_time and data_start_time.
execution_end_timeThe actual end time of the detector for a specific run that produces the anomaly result.
anomaly_scoreIndicates relative severity of an anomaly. The higher the score, the more anomalous a data point is.
anomaly_gradeA normalized version of the anomaly_score on a scale between 0 and 1.
confidenceThe probability of the accuracy of the anomaly_score. The closer this number is to 1, the higher the accuracy. During the probation period of a running detector, the confidence is low (< 0.9) because of its exposure to limited data.
entityAn entity is a combination of specific category fields’ values. It includes the name and value of the category field. In the previous example, process_name is the category field and one of the processes such as process_3 is the field’s value. The entity field is only present for a high-cardinality detector (where you’ve selected a category field).
model_idA unique ID that identifies a model. If a detector is a single-stream detector (with no category field), it has only one model. If a detector is a high-cardinality detector (with one or more category fields), it might have multiple models, one for each entity.
thresholdOne of the criteria for a detector to classify a data point as an anomaly is that its anomaly_score must surpass a dynamic threshold. This field records the current threshold.

When the imputation option is enabled, the anomaly results include a feature_imputed array showing which features were modified due to missing data. If no features were imputed, then this is excluded.

In the following example anomaly result output, the processing_bytes_max feature was imputed, as shown by the imputed: true status:

  1. {
  2.     "detector_id": "kzcZ43wBgEQAbjDnhzGF",
  3.     "schema_version": 5,
  4.     "data_start_time": 1635898161367,
  5.     "data_end_time": 1635898221367,
  6.     "feature_data": [
  7.         {
  8.             "feature_id": "processing_bytes_max",
  9.             "feature_name": "processing bytes max",
  10.             "data": 2322
  11.         },
  12.         {
  13.             "feature_id": "processing_bytes_avg",
  14.             "feature_name": "processing bytes avg",
  15.             "data": 1718.6666666666667
  16.         },
  17.         {
  18.             "feature_id": "processing_bytes_min",
  19.             "feature_name": "processing bytes min",
  20.             "data": 1375
  21.         },
  22.         {
  23.             "feature_id": "processing_bytes_sum",
  24.             "feature_name": "processing bytes sum",
  25.             "data": 5156
  26.         },
  27.         {
  28.             "feature_id": "processing_time_max",
  29.             "feature_name": "processing time max",
  30.             "data": 31198
  31.         }
  32.     ],
  33.     "execution_start_time": 1635898231577,
  34.     "execution_end_time": 1635898231622,
  35.     "anomaly_score": 1.8124904404395776,
  36.     "anomaly_grade": 0,
  37.     "confidence": 0.9802940756605277,
  38.     "entity": [
  39.         {
  40.             "name": "process_name",
  41.             "value": "process_3"
  42.         }
  43.     ],
  44.     "model_id": "kzcZ43wBgEQAbjDnhzGF_entity_process_3",
  45.     "threshold": 1.2368549346675202,
  46.     "feature_imputed": [
  47.         {
  48.             "feature_id": "processing_bytes_max",
  49.             "imputed": true
  50.         },
  51.         {
  52.             "feature_id": "processing_bytes_avg",
  53.             "imputed": false
  54.         },
  55.         {
  56.             "feature_id": "processing_bytes_min",
  57.             "imputed": false
  58.         },
  59.         {
  60.             "feature_id": "processing_bytes_sum",
  61.             "imputed": false
  62.         },
  63.         {
  64.             "feature_id": "processing_time_max",
  65.             "imputed": false
  66.         }
  67.     ]
  68. }

copy

When an anomaly is detected, the result is provided in the following format:

  1. {
  2.   "detector_id": "fylE53wBc9MCt6q12tKp",
  3.   "schema_version": 0,
  4.   "data_start_time": 1635927900000,
  5.   "data_end_time": 1635927960000,
  6.   "feature_data": [
  7.     {
  8.       "feature_id": "processing_bytes_max",
  9.       "feature_name": "processing bytes max",
  10.       "data": 2291
  11.     },
  12.     {
  13.       "feature_id": "processing_bytes_avg",
  14.       "feature_name": "processing bytes avg",
  15.       "data": 1677.3333333333333
  16.     },
  17.     {
  18.       "feature_id": "processing_bytes_min",
  19.       "feature_name": "processing bytes min",
  20.       "data": 1054
  21.     },
  22.     {
  23.       "feature_id": "processing_bytes_sum",
  24.       "feature_name": "processing bytes sum",
  25.       "data": 5032
  26.     },
  27.     {
  28.       "feature_id": "processing_time_max",
  29.       "feature_name": "processing time max",
  30.       "data": 11422
  31.     }
  32.   ],
  33.   "anomaly_score": 1.1986675882872033,
  34.   "anomaly_grade": 0.26806225550178464,
  35.   "confidence": 0.9607519742565531,
  36.   "entity": [
  37.     {
  38.       "name": "process_name",
  39.       "value": "process_3"
  40.     }
  41.   ],
  42.   "approx_anomaly_start_time": 1635927900000,
  43.   "relevant_attribution": [
  44.     {
  45.       "feature_id": "processing_bytes_max",
  46.       "data": 0.03628638020431366
  47.     },
  48.     {
  49.       "feature_id": "processing_bytes_avg",
  50.       "data": 0.03384479053991436
  51.     },
  52.     {
  53.       "feature_id": "processing_bytes_min",
  54.       "data": 0.058812549572819096
  55.     },
  56.     {
  57.       "feature_id": "processing_bytes_sum",
  58.       "data": 0.10154576265526988
  59.     },
  60.     {
  61.       "feature_id": "processing_time_max",
  62.       "data": 0.7695105170276828
  63.     }
  64.   ],
  65.   "expected_values": [
  66.     {
  67.       "likelihood": 1,
  68.       "value_list": [
  69.         {
  70.           "feature_id": "processing_bytes_max",
  71.           "data": 2291
  72.         },
  73.         {
  74.           "feature_id": "processing_bytes_avg",
  75.           "data": 1677.3333333333333
  76.         },
  77.         {
  78.           "feature_id": "processing_bytes_min",
  79.           "data": 1054
  80.         },
  81.         {
  82.           "feature_id": "processing_bytes_sum",
  83.           "data": 6062
  84.         },
  85.         {
  86.           "feature_id": "processing_time_max",
  87.           "data": 23379
  88.         }
  89.       ]
  90.     }
  91.   ],
  92.   "threshold": 1.0993584705913992,
  93.   "execution_end_time": 1635898427895,
  94.   "execution_start_time": 1635898427803
  95. }

copy

Note that the result includes the following additional field.

FieldDescription
relevant_attributionRepresents the contribution of each input variable. The sum of the attributions is normalized to 1.
expected_valuesThe expected value for each feature.

The detector may be late in detecting an anomaly. For example: The detector observes a sequence of data that alternates between “slow weeks” (represented by the triples {1, 2, 3}) and “busy weeks” (represented by the triples {2, 4, 5}). If the detector comes across a pattern {2, 2, X}, where it has not yet seen the value that X will take, then the detector infers that the pattern is anomalous. However, it cannot determine which 2 is the cause. If X = 3, then the first 2 is the anomaly. If X = 5, then the second 2 is the anomaly. If it is the first 2, then the detector will be late in detecting the anomaly.

When a detector is late in detecting an anomaly, the result includes the following additional fields.

FieldDescription
past_valuesThe actual input that triggered an anomaly. If past_values is null, then the attributions or expected values are from the current input. If past_values is not null, then the attributions or expected values are from a past input (for example, the previous two steps of the data [1,2,3]).
approx_anomaly_start_timeThe approximate time of the actual input that triggered an anomaly. This field helps you understand the time at which a detector flags an anomaly. Both single-stream and high-cardinality detectors do not query previous anomaly results because these queries are costly operations. The cost is especially high for high-cardinality detectors that may have many entities. If the data is not continuous, then the accuracy of this field is low and the actual time at which the detector detects an anomaly can be earlier.
  1. {
  2.   "detector_id": "kzcZ43wBgEQAbjDnhzGF",
  3.   "confidence": 0.9746820962328963,
  4.   "relevant_attribution": [
  5.     {
  6.       "feature_id": "deny_max1",
  7.       "data": 0.07339452532666227
  8.     },
  9.     {
  10.       "feature_id": "deny_avg",
  11.       "data": 0.04934972719948845
  12.     },
  13.     {
  14.       "feature_id": "deny_min",
  15.       "data": 0.01803003656061806
  16.     },
  17.     {
  18.       "feature_id": "deny_sum",
  19.       "data": 0.14804918212089874
  20.     },
  21.     {
  22.       "feature_id": "accept_max5",
  23.       "data": 0.7111765287923325
  24.     }
  25.   ],
  26.   "task_id": "9Dck43wBgEQAbjDn4zEe",
  27.   "threshold": 1,
  28.   "model_id": "kzcZ43wBgEQAbjDnhzGF_entity_app_0",
  29.   "schema_version": 5,
  30.   "anomaly_score": 1.141419389056506,
  31.   "execution_start_time": 1635898427803,
  32.   "past_values": [
  33.     {
  34.       "feature_id": "processing_bytes_max",
  35.       "data": 905
  36.     },
  37.     {
  38.       "feature_id": "processing_bytes_avg",
  39.       "data": 479
  40.     },
  41.     {
  42.       "feature_id": "processing_bytes_min",
  43.       "data": 128
  44.     },
  45.     {
  46.       "feature_id": "processing_bytes_sum",
  47.       "data": 1437
  48.     },
  49.     {
  50.       "feature_id": "processing_time_max",
  51.       "data": 8440
  52.     }
  53.   ],
  54.   "data_end_time": 1635883920000,
  55.   "data_start_time": 1635883860000,
  56.   "feature_data": [
  57.     {
  58.       "feature_id": "processing_bytes_max",
  59.       "feature_name": "processing bytes max",
  60.       "data": 1360
  61.     },
  62.     {
  63.       "feature_id": "processing_bytes_avg",
  64.       "feature_name": "processing bytes avg",
  65.       "data": 990
  66.     },
  67.     {
  68.       "feature_id": "processing_bytes_min",
  69.       "feature_name": "processing bytes min",
  70.       "data": 608
  71.     },
  72.     {
  73.       "feature_id": "processing_bytes_sum",
  74.       "feature_name": "processing bytes sum",
  75.       "data": 2970
  76.     },
  77.     {
  78.       "feature_id": "processing_time_max",
  79.       "feature_name": "processing time max",
  80.       "data": 9670
  81.     }
  82.   ],
  83.   "expected_values": [
  84.     {
  85.       "likelihood": 1,
  86.       "value_list": [
  87.         {
  88.           "feature_id": "processing_bytes_max",
  89.           "data": 905
  90.         },
  91.         {
  92.           "feature_id": "processing_bytes_avg",
  93.           "data": 479
  94.         },
  95.         {
  96.           "feature_id": "processing_bytes_min",
  97.           "data": 128
  98.         },
  99.         {
  100.           "feature_id": "processing_bytes_sum",
  101.           "data": 4847
  102.         },
  103.         {
  104.           "feature_id": "processing_time_max",
  105.           "data": 15713
  106.         }
  107.       ]
  108.     }
  109.   ],
  110.   "execution_end_time": 1635898427895,
  111.   "anomaly_grade": 0.5514172746375128,
  112.   "entity": [
  113.     {
  114.       "name": "process_name",
  115.       "value": "process_3"
  116.     }
  117.   ],
  118.   "approx_anomaly_start_time": 1635883620000
  119. }

copy