Status Check in Workflow

In Workflow, the status check could execute specified operations on external systems, such as application systems and monitoring systems, to obtain their statuses, and automatically abort the Workflow when it finds the system is unhealthy. The concept is similar to Container Probes in Kubernetes. This article describes how to execute status checks in Workflow using YAML files.

note

Chaos Mesh does not yet support creating StatusCheck nodes on Chaos Dashboard, so you could only create StatusCheck nodes using YAML for now.

Status Check type

Chaos Mesh only support the HTTP type to execute a status check.

Define a HTTP StatusCheck node

A StatusCheck node sends GET or POST HTTP requests to the specific URL, with custom headers and body, and then determines the result of the request by the conditions in the criteria field.

- name: workflow-status-check
  templateType: StatusCheck
  deadline: 20s
  statusCheck:
    mode: Continuous
    type: HTTP
    intervalSeconds: 1
    timeoutSeconds: 1
    http:
      url: http://123.123.123.123
      method: GET
      criteria:
        statusCode: '200'

In the configuration, you can see a StatusCheck node with HTTP type. The deadline field specifies that this node could be executed for a maximum of 20 seconds. The mode field specifies that this node will execute status checks continuously. The intervalSeconds field specifies a repetition interval of 1 second. The timeoutSeconds field specifies the timeout for each execution.

When Workflow runs to this StatusCheck node, the specified status check would be executed every second. The status check uses the GET method to send an HTTP request to the URL http://123.123.123.123. If the response is returned within 1 second and the status code is 200, this execution succeeds, otherwise it fails.

Status Check results

Each execution of the status check will get an execution result, either Success or Failure. Because a single execution result may not reflect the real situation of the system, due to fluctuations in certain conditions, the final status check result is not determined based on a single execution result.

The StatusCheck node has failureThreshold and successThreshold two fields:

• When the number of consecutive failed execution results exceeds the failureThreshold, the status check result is considered to be a Failure.
• When the number of consecutive successful execution results exceeds the successThreshold, the status check result is considered to be a Success.

- name: workflow-status-check
  templateType: StatusCheck
  deadline: 20s
  statusCheck:
    mode: Continuous
    type: HTTP
    successThreshold: 1
    failureThreshold: 3
    http:
      url: http://123.123.123.123
      method: GET
      criteria:
        statusCode: '200'

In the configuration, the StatusCheck node will execute status checks continuously:

• When the execution result is Success for 1 or more consecutive times, the status check result is considered to be a Success.
• When the execution result is Failure for 3 or more consecutive times, the status check result is considered to be a Failure.

note

In the following sections, status check fails refers to that status check result is Failure, rather than a single execution result is Failure.

When the Status Check is unsuccessful, abort the Workflow

note

The StatusCheck node only supports aborting the workflow automatically when the status check is unsuccessful. It could not pause or resume the workflow.

When executing chaos experiments, the application system might become unhealthy, this function can be used to restore the application system by quickly ending chaos experiments. To enable the workflow to abort automatically when the status check fails, you can set the abortWithStatusCheck field to true on the StatusCheck node.

- name: workflow-status-check
  templateType: StatusCheck
  deadline: 20s
  abortWithStatusCheck: true
  statusCheck:
    mode: Continuous
    type: HTTP
    http:
      url: http://123.123.123.123
      method: GET
      criteria:
        statusCode: '200'

The status check is considered unsuccessful when any of the following conditions are met:

• The status check fails.
• When the StatusCheck node timeout is exceeded, and the status check result is not successful. For example, successThreshold is 1, failureThreshold is 3, and when the timeout is exceeded, there are 2 consecutive failures and 0 successes. Although it does not meet the condition for “status check fails”, it is also considered to be unsuccessful in this case.

Status Check mode

Continuous Status Check

When the mode field is Continuous, it means this StatusCheck node will execute status checks continuously until the node times out or the status check fails.

- name: workflow-status-check
  templateType: StatusCheck
  deadline: 20s
  statusCheck:
    mode: Continuous
    type: HTTP
    intervalSeconds: 1
    successThreshold: 1
    failureThreshold: 3
    http:
      url: http://123.123.123.123
      method: GET
      criteria:
        statusCode: '200'

In the configuration, the StatusCheck node will execute status checks every second, and exit when any of the following conditions are met:

• The status check fails, i.e. 3 or more consecutive failed execution results
• Trigger the node timeout after 20 seconds

One time Status Check

When the mode field is Synchronous, it means that this StatusCheck node will exit immediately when the status check result is clear, or when the node times out.

- name: workflow-status-check
  templateType: StatusCheck
  deadline: 20s
  statusCheck:
    mode: Synchronous
    type: HTTP
    intervalSeconds: 1
    successThreshold: 1
    failureThreshold: 3
    http:
      url: http://123.123.123.123
      method: GET
      criteria:
        statusCode: '200'

In the configuration, the StatusCheck node will execute status checks every second, and exit when any of the following conditions are met:

• The status check succeeds, i.e. 1 or more consecutive successful execution results
• The status check fails, i.e. 3 or more consecutive failed execution results
• Trigger the node timeout after 20 seconds

StatusCheck vs HTTP Request Task

Similarities:

• The StatusCheck node and the HTTP Request Task node (the Task node that executes HTTP requests) are a node type of Workflow.
• The StatusCheck node and the HTTP Request Task node can obtain information about external systems through HTTP requests.

Differences:

• The HTTP Request Task node could only send an HTTP once, and cannot send HTTP requests continuously.
• The HTTP Request Task node cannot affect the status of the workflow when the request fails, such as aborting the workflow.

Field description

For more information about Workflow and Template, refer to Create Chaos Mesh Workflow.

StatusCheck field description

HTTPStatusCheck field description

HTTPCriteria field description