Anomaly result mapping

If you enabled custom result index, the anomaly detection plugin stores the results in your own index.

If the anomaly detector doesn’t detect an anomaly, the result has the following format:

  1. {
  2.   "detector_id": "kzcZ43wBgEQAbjDnhzGF",
  3.   "schema_version": 5,
  4.   "data_start_time": 1635898161367,
  5.   "data_end_time": 1635898221367,
  6.   "feature_data": [
  7.     {
  8.       "feature_id": "processing_bytes_max",
  9.       "feature_name": "processing bytes max",
  10.       "data": 2322
  11.     },
  12.     {
  13.       "feature_id": "processing_bytes_avg",
  14.       "feature_name": "processing bytes avg",
  15.       "data": 1718.6666666666667
  16.     },
  17.     {
  18.       "feature_id": "processing_bytes_min",
  19.       "feature_name": "processing bytes min",
  20.       "data": 1375
  21.     },
  22.     {
  23.       "feature_id": "processing_bytes_sum",
  24.       "feature_name": "processing bytes sum",
  25.       "data": 5156
  26.     },
  27.     {
  28.       "feature_id": "processing_time_max",
  29.       "feature_name": "processing time max",
  30.       "data": 31198
  31.     }
  32.   ],
  33.   "execution_start_time": 1635898231577,
  34.   "execution_end_time": 1635898231622,
  35.   "anomaly_score": 1.8124904404395776,
  36.   "anomaly_grade": 0,
  37.   "confidence": 0.9802940756605277,
  38.   "entity": [
  39.     {
  40.       "name": "process_name",
  41.       "value": "process_3"
  42.     }
  43.   ],
  44.   "model_id": "kzcZ43wBgEQAbjDnhzGF_entity_process_3",
  45.   "threshold": 1.2368549346675202
  46. }

Response body fields

FieldDescription
detector_idA unique ID for identifying a detector.
schema_versionThe mapping version of the result index.
data_start_timeThe start of the detection range of the aggregated data.
data_end_timeThe end of the detection range of the aggregated data.
feature_dataAn array of the aggregated data points between the data_start_time and data_end_time.
execution_start_timeThe actual start time of the detector for a specific run that produces the anomaly result. This start time includes the window delay parameter that you can set to delay data collection. Window delay is the difference between the execution_start_time and data_start_time.
execution_end_timeThe actual end time of the detector for a specific run that produces the anomaly result.
anomaly_scoreIndicates relative severity of an anomaly. The higher the score, the more anomalous a data point is.
anomaly_gradeA normalized version of the anomaly_score on a scale between 0 and 1.
confidenceThe probability of the accuracy of the anomaly_score. The closer this number is to 1, the higher the accuracy. During the probation period of a running detector, the confidence is low (< 0.9) because of its exposure to limited data.
entityAn entity is a combination of specific category fields’ values. It includes the name and value of the category field. In the previous example, process_name is the category field and one of the processes such as process_3 is the field’s value. The entity field is only present for a high-cardinality detector (where you’ve selected a category field).
model_idA unique ID that identifies a model. If a detector is a single-stream detector (with no category field), it has only one model. If a detector is a high-cardinality detector (with one or more category fields), it might have multiple models, one for each entity.
thresholdOne of the criteria for a detector to classify a data point as an anomaly is that its anomaly_score must surpass a dynamic threshold. This field records the current threshold.

If an anomaly detector detects an anomaly, the result has the following format:

  1. {
  2.   "detector_id": "fylE53wBc9MCt6q12tKp",
  3.   "schema_version": 0,
  4.   "data_start_time": 1635927900000,
  5.   "data_end_time": 1635927960000,
  6.   "feature_data": [
  7.     {
  8.       "feature_id": "processing_bytes_max",
  9.       "feature_name": "processing bytes max",
  10.       "data": 2291
  11.     },
  12.     {
  13.       "feature_id": "processing_bytes_avg",
  14.       "feature_name": "processing bytes avg",
  15.       "data": 1677.3333333333333
  16.     },
  17.     {
  18.       "feature_id": "processing_bytes_min",
  19.       "feature_name": "processing bytes min",
  20.       "data": 1054
  21.     },
  22.     {
  23.       "feature_id": "processing_bytes_sum",
  24.       "feature_name": "processing bytes sum",
  25.       "data": 5032
  26.     },
  27.     {
  28.       "feature_id": "processing_time_max",
  29.       "feature_name": "processing time max",
  30.       "data": 11422
  31.     }
  32.   ],
  33.   "anomaly_score": 1.1986675882872033,
  34.   "anomaly_grade": 0.26806225550178464,
  35.   "confidence": 0.9607519742565531,
  36.   "entity": [
  37.     {
  38.       "name": "process_name",
  39.       "value": "process_3"
  40.     }
  41.   ],
  42.   "approx_anomaly_start_time": 1635927900000,
  43.   "relevant_attribution": [
  44.     {
  45.       "feature_id": "processing_bytes_max",
  46.       "data": 0.03628638020431366
  47.     },
  48.     {
  49.       "feature_id": "processing_bytes_avg",
  50.       "data": 0.03384479053991436
  51.     },
  52.     {
  53.       "feature_id": "processing_bytes_min",
  54.       "data": 0.058812549572819096
  55.     },
  56.     {
  57.       "feature_id": "processing_bytes_sum",
  58.       "data": 0.10154576265526988
  59.     },
  60.     {
  61.       "feature_id": "processing_time_max",
  62.       "data": 0.7695105170276828
  63.     }
  64.   ],
  65.   "expected_values": [
  66.     {
  67.       "likelihood": 1,
  68.       "value_list": [
  69.         {
  70.           "feature_id": "processing_bytes_max",
  71.           "data": 2291
  72.         },
  73.         {
  74.           "feature_id": "processing_bytes_avg",
  75.           "data": 1677.3333333333333
  76.         },
  77.         {
  78.           "feature_id": "processing_bytes_min",
  79.           "data": 1054
  80.         },
  81.         {
  82.           "feature_id": "processing_bytes_sum",
  83.           "data": 6062
  84.         },
  85.         {
  86.           "feature_id": "processing_time_max",
  87.           "data": 23379
  88.         }
  89.       ]
  90.     }
  91.   ],
  92.   "threshold": 1.0993584705913992,
  93.   "execution_end_time": 1635898427895,
  94.   "execution_start_time": 1635898427803
  95. }

You can see the following additional fields:

FieldDescription
relevant_attributionRepresents the contribution of each input variable. The sum of the attributions is normalized to 1.
expected_valuesThe expected value for each feature.

At times, the detector might detect an anomaly late. Let’s say the detector sees a random mix of the triples {1, 2, 3} and {2, 4, 5} that correspond to slow weeks and busy weeks, respectively. For example 1, 2, 3, 1, 2, 3, 2, 4, 5, 1, 2, 3, 2, 4, 5, … and so on. If the detector comes across a pattern {2, 2, X} and it’s yet to see X, the detector infers that the pattern is anomalous, but it can’t determine at this point which of the 2’s is the cause. If X = 3, then the detector knows it’s the first 2 in that unfinished triple, and if X = 5, then it’s the second 2. If it’s the first 2, then the detector detects the anomaly late.

If a detector detects an anomaly late, the result has the following additional fields:

FieldDescription
past_valuesThe actual input that triggered an anomaly. If past_values is null, the attributions or expected values are from the current input. If past_values is not null, the attributions or expected values are from a past input (for example, the previous two steps of the data [1,2,3]).
approx_anomaly_start_timeThe approximate time of the actual input that triggers an anomaly. This field helps you understand when a detector flags an anomaly. Both single-stream and high-cardinality detectors don’t query previous anomaly results because these queries are expensive operations. The cost is especially high for high-cardinality detectors that might have a lot of entities. If the data is not continuous, the accuracy of this field is low and the actual time that the detector detects an anomaly can be earlier.
  1. {
  2.   "detector_id": "kzcZ43wBgEQAbjDnhzGF",
  3.   "confidence": 0.9746820962328963,
  4.   "relevant_attribution": [
  5.     {
  6.       "feature_id": "deny_max1",
  7.       "data": 0.07339452532666227
  8.     },
  9.     {
  10.       "feature_id": "deny_avg",
  11.       "data": 0.04934972719948845
  12.     },
  13.     {
  14.       "feature_id": "deny_min",
  15.       "data": 0.01803003656061806
  16.     },
  17.     {
  18.       "feature_id": "deny_sum",
  19.       "data": 0.14804918212089874
  20.     },
  21.     {
  22.       "feature_id": "accept_max5",
  23.       "data": 0.7111765287923325
  24.     }
  25.   ],
  26.   "task_id": "9Dck43wBgEQAbjDn4zEe",
  27.   "threshold": 1,
  28.   "model_id": "kzcZ43wBgEQAbjDnhzGF_entity_app_0",
  29.   "schema_version": 5,
  30.   "anomaly_score": 1.141419389056506,
  31.   "execution_start_time": 1635898427803,
  32.   "past_values": [
  33.     {
  34.       "feature_id": "processing_bytes_max",
  35.       "data": 905
  36.     },
  37.     {
  38.       "feature_id": "processing_bytes_avg",
  39.       "data": 479
  40.     },
  41.     {
  42.       "feature_id": "processing_bytes_min",
  43.       "data": 128
  44.     },
  45.     {
  46.       "feature_id": "processing_bytes_sum",
  47.       "data": 1437
  48.     },
  49.     {
  50.       "feature_id": "processing_time_max",
  51.       "data": 8440
  52.     }
  53.   ],
  54.   "data_end_time": 1635883920000,
  55.   "data_start_time": 1635883860000,
  56.   "feature_data": [
  57.     {
  58.       "feature_id": "processing_bytes_max",
  59.       "feature_name": "processing bytes max",
  60.       "data": 1360
  61.     },
  62.     {
  63.       "feature_id": "processing_bytes_avg",
  64.       "feature_name": "processing bytes avg",
  65.       "data": 990
  66.     },
  67.     {
  68.       "feature_id": "processing_bytes_min",
  69.       "feature_name": "processing bytes min",
  70.       "data": 608
  71.     },
  72.     {
  73.       "feature_id": "processing_bytes_sum",
  74.       "feature_name": "processing bytes sum",
  75.       "data": 2970
  76.     },
  77.     {
  78.       "feature_id": "processing_time_max",
  79.       "feature_name": "processing time max",
  80.       "data": 9670
  81.     }
  82.   ],
  83.   "expected_values": [
  84.     {
  85.       "likelihood": 1,
  86.       "value_list": [
  87.         {
  88.           "feature_id": "processing_bytes_max",
  89.           "data": 905
  90.         },
  91.         {
  92.           "feature_id": "processing_bytes_avg",
  93.           "data": 479
  94.         },
  95.         {
  96.           "feature_id": "processing_bytes_min",
  97.           "data": 128
  98.         },
  99.         {
  100.           "feature_id": "processing_bytes_sum",
  101.           "data": 4847
  102.         },
  103.         {
  104.           "feature_id": "processing_time_max",
  105.           "data": 15713
  106.         }
  107.       ]
  108.     }
  109.   ],
  110.   "execution_end_time": 1635898427895,
  111.   "anomaly_grade": 0.5514172746375128,
  112.   "entity": [
  113.     {
  114.       "name": "process_name",
  115.       "value": "process_3"
  116.     }
  117.   ],
  118.   "approx_anomaly_start_time": 1635883620000
  119. }