Status Check in Workflow

In Workflow, the status check could execute specified operations on external systems, such as application systems and monitoring systems, to obtain their statuses, and automatically abort the Workflow when it finds the system is unhealthy. The concept is similar to Container Probes in Kubernetes. This article describes how to execute status checks in Workflow using YAML files.

note

Chaos Mesh does not yet support creating StatusCheck nodes on Chaos Dashboard, so you could only create StatusCheck nodes using YAML for now.

Status Check type

Chaos Mesh only support the HTTP type to execute a status check.

Define a HTTP StatusCheck node

A StatusCheck node sends GET or POST HTTP requests to the specific URL, with custom headers and body, and then determines the result of the request by the conditions in the criteria field.

  1. - name: workflow-status-check
  2.   templateType: StatusCheck
  3.   deadline: 20s
  4.   statusCheck:
  5.     mode: Continuous
  6.     type: HTTP
  7.     intervalSeconds: 1
  8.     timeoutSeconds: 1
  9.     http:
  10.       url: http://123.123.123.123
  11.       method: GET
  12.       criteria:
  13.         statusCode: '200'

In the configuration, you can see a StatusCheck node with HTTP type. The deadline field specifies that this node could be executed for a maximum of 20 seconds. The mode field specifies that this node will execute status checks continuously. The intervalSeconds field specifies a repetition interval of 1 second. The timeoutSeconds field specifies the timeout for each execution.

When Workflow runs to this StatusCheck node, the specified status check would be executed every second. The status check uses the GET method to send an HTTP request to the URL http://123.123.123.123. If the response is returned within 1 second and the status code is 200, this execution succeeds, otherwise it fails.

Status Check results

Each execution of the status check will get an execution result, either Success or Failure. Because a single execution result may not reflect the real situation of the system, due to fluctuations in certain conditions, the final status check result is not determined based on a single execution result.

The StatusCheck node has failureThreshold and successThreshold two fields:

  • When the number of consecutive failed execution results exceeds the failureThreshold, the status check result is considered to be a Failure.
  • When the number of consecutive successful execution results exceeds the successThreshold, the status check result is considered to be a Success.
  1. - name: workflow-status-check
  2.   templateType: StatusCheck
  3.   deadline: 20s
  4.   statusCheck:
  5.     mode: Continuous
  6.     type: HTTP
  7.     successThreshold: 1
  8.     failureThreshold: 3
  9.     http:
  10.       url: http://123.123.123.123
  11.       method: GET
  12.       criteria:
  13.         statusCode: '200'

In the configuration, the StatusCheck node will execute status checks continuously:

  • When the execution result is Success for 1 or more consecutive times, the status check result is considered to be a Success.
  • When the execution result is Failure for 3 or more consecutive times, the status check result is considered to be a Failure.
note

In the following sections, status check fails refers to that status check result is Failure, rather than a single execution result is Failure.

When the Status Check is unsuccessful, abort the Workflow

note

The StatusCheck node only supports aborting the workflow automatically when the status check is unsuccessful. It could not pause or resume the workflow.

When executing chaos experiments, the application system might become unhealthy, this function can be used to restore the application system by quickly ending chaos experiments. To enable the workflow to abort automatically when the status check fails, you can set the abortWithStatusCheck field to true on the StatusCheck node.

  1. - name: workflow-status-check
  2.   templateType: StatusCheck
  3.   deadline: 20s
  4.   abortWithStatusCheck: true
  5.   statusCheck:
  6.     mode: Continuous
  7.     type: HTTP
  8.     http:
  9.       url: http://123.123.123.123
  10.       method: GET
  11.       criteria:
  12.         statusCode: '200'

The status check is considered unsuccessful when any of the following conditions are met:

  • The status check fails.
  • When the StatusCheck node timeout is exceeded, and the status check result is not successful. For example, successThreshold is 1, failureThreshold is 3, and when the timeout is exceeded, there are 2 consecutive failures and 0 successes. Although it does not meet the condition for “status check fails”, it is also considered to be unsuccessful in this case.

Status Check mode

Continuous Status Check

When the mode field is Continuous, it means this StatusCheck node will execute status checks continuously until the node times out or the status check fails.

  1. - name: workflow-status-check
  2.   templateType: StatusCheck
  3.   deadline: 20s
  4.   statusCheck:
  5.     mode: Continuous
  6.     type: HTTP
  7.     intervalSeconds: 1
  8.     successThreshold: 1
  9.     failureThreshold: 3
  10.     http:
  11.       url: http://123.123.123.123
  12.       method: GET
  13.       criteria:
  14.         statusCode: '200'

In the configuration, the StatusCheck node will execute status checks every second, and exit when any of the following conditions are met:

  • The status check fails, i.e. 3 or more consecutive failed execution results
  • Trigger the node timeout after 20 seconds

One time Status Check

When the mode field is Synchronous, it means that this StatusCheck node will exit immediately when the status check result is clear, or when the node times out.

  1. - name: workflow-status-check
  2.   templateType: StatusCheck
  3.   deadline: 20s
  4.   statusCheck:
  5.     mode: Synchronous
  6.     type: HTTP
  7.     intervalSeconds: 1
  8.     successThreshold: 1
  9.     failureThreshold: 3
  10.     http:
  11.       url: http://123.123.123.123
  12.       method: GET
  13.       criteria:
  14.         statusCode: '200'

In the configuration, the StatusCheck node will execute status checks every second, and exit when any of the following conditions are met:

  • The status check succeeds, i.e. 1 or more consecutive successful execution results
  • The status check fails, i.e. 3 or more consecutive failed execution results
  • Trigger the node timeout after 20 seconds

StatusCheck vs HTTP Request Task

Similarities:

  • The StatusCheck node and the HTTP Request Task node (the Task node that executes HTTP requests) are a node type of Workflow.
  • The StatusCheck node and the HTTP Request Task node can obtain information about external systems through HTTP requests.

Differences:

  • The HTTP Request Task node could only send an HTTP once, and cannot send HTTP requests continuously.
  • The HTTP Request Task node cannot affect the status of the workflow when the request fails, such as aborting the workflow.

Field description

For more information about Workflow and Template, refer to Create Chaos Mesh Workflow.

StatusCheck field description

ParameterTypeDescriptionDefault valueRequiredExample
modestringThe execution mode of the status check. Support value: Synchronous/Continuous.NoneYesSynchronous
typestringThe type of the status check. Support value: HTTP.HTTPYesHTTP
durationstringThe duration of the whole status check if the number of failed execution does not exceed the failureThreshold. It is available in both Synchronous and Continuous modes.NoneNo100s
timeoutSecondsintThe timeout seconds when the status check fails.1No1
intervalSecondsintDefines how often (in seconds) to perform an execution of status check.1No1
failureThresholdintThe minimum consecutive failure for the status check to be considered failed.3No3
successThresholdintThe minimum consecutive successes for the status check to be considered successful.1No1
recordsHistoryLimitintThe number of records to retain.100No100
httpHTTPStatusCheckConfigure the detail of the HTTP request to execute.NoneNo

HTTPStatusCheck field description

ParameterTypeDescriptionDefault valueRequiredExample
urlstringThe URL of the HTTP request.NoneYeshttp://123.123.123.123
methodstringThe method of the HTTP request. Support value: GET/POST.GETNoGET
headersmap[string][]stringThe headers of the HTTP request.NoneNo
bodystringThe body of the HTTP request.NoneNo{“a”:”b”}
criteriaHTTPCriteriaDefines how to determine the result of the HTTP StatusCheck.NoneYes

HTTPCriteria field description

ParameterTypeDescriptionDefault valueRequiredExample
statusCodestringThe expected http status code for the request. A statusCode string could be a single code (e.g. 200), or an inclusive range (e.g. 200-400, both 200 and 400 are included).NoneYes200