Add a New Chaos Experiment Type

Add a New Chaos Experiment Type

This document describes how to add a new chaos experiment type.

The following walks you through an example of HelloWorldChaos, a new chaos experiment type that prints Hello world! to the log. The steps include:

Step 1: Define the schema of HelloWorldChaos
Step 2: Register the CRD
Step 3: Register the event handler for helloworld objects
Step 4: Build Docker images
Step 5: Run HelloWorldChaos

Step 1: Define the schema of HelloWorldChaos

Add helloworldchaos_types.go to the api/v1alpha1 API directory with the following content:

package v1alpha1
import (
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// +kubebuilder:object:root=true
// +chaos-mesh:experiment
// +chaos-mesh:oneshot=true
// HelloWorldChaos is the Schema for the helloworldchaos API
type HelloWorldChaos struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`
        Spec   HelloWorldChaosSpec   `json:"spec"`
        Status HelloWorldChaosStatus `json:"status,omitempty"`
}
// HelloWorldChaosSpec defines the desired state of HelloWorldChaos
type HelloWorldChaosSpec struct {
        // ContainerSelector specifies the target for injection
        ContainerSelector `json:",inline"`
        // Duration represents the duration of the chaos
        // +optional
        Duration *string `json:"duration,omitempty"`
        // RemoteCluster represents the remote cluster where the chaos will be deployed
        // +optional
        RemoteCluster string `json:"remoteCluster,omitempty"`
}
// HelloWorldChaosStatus defines the observed state of HelloWorldChaos
type HelloWorldChaosStatus struct {
        ChaosStatus `json:",inline"`
}
// GetSelectorSpecs is a getter for selectors
func (obj *HelloWorldChaos) GetSelectorSpecs() map[string]interface{} {
        return map[string]interface{}{
                ".": &obj.Spec.ContainerSelector,
        }
}

This file defines the schema type of HelloWorldChaos, which can be described in a YAML file:

apiVersion: chaos-mesh.org/v1alpha1
kind: HelloWorldChaos
metadata:
  name: <resource name>
  namespace: <namespace>
spec:
  duration: <duration>
#...

Step 2: Register the CRD

You need to register the CRD (Custom Resource Definition) of HelloWorldChaos to interact it with Kubernetes API.

To combine the CRD into manifests/crd.yaml, append config/crd/bases/chaos-mesh.org_helloworldchaos.yaml we generated in the previous step to config/crd/kustomization.yaml:

resources:
  - bases/chaos-mesh.org_podchaos.yaml
  - bases/chaos-mesh.org_networkchaos.yaml
  - bases/chaos-mesh.org_iochaos.yaml
  - bases/chaos-mesh.org_helloworldchaos.yaml # This is the new line

Run make generate in the root directory of Chaos Mesh, which generates a boilerplate of HelloWorldChaos for Chaos Mesh to compile:
```
make generate
```
Then you can see the definition of HelloWorldChaos in manifests/crd.yaml.

Step 3: Register the event handler for helloworldchaos objects

Create a new file controllers/chaosimpl/helloworldchaos/types.go with the following content:

package helloworldchaos
import (
        "context"
        "github.com/go-logr/logr"
        "go.uber.org/fx"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
        impltypes "github.com/chaos-mesh/chaos-mesh/controllers/chaosimpl/types"
        "github.com/chaos-mesh/chaos-mesh/controllers/chaosimpl/utils"
)
var _ impltypes.ChaosImpl = (*Impl)(nil)
type Impl struct {
        client.Client
        Log logr.Logger
        decoder *utils.ContainerRecordDecoder
}
// This corresponds to the Apply phase of HelloWorldChaos. The execution of HelloWorldChaos will be triggered.
func (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {
        impl.Log.Info("Hello world!")
        return v1alpha1.Injected, nil
}
// This corresponds to the Recover phase of HelloWorldChaos. The reconciler will be triggered to recover the chaos action.
func (impl *Impl) Recover(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {
        impl.Log.Info("Goodbye world!")
        return v1alpha1.NotInjected, nil
}
// NewImpl returns a new HelloWorldChaos implementation instance.
func NewImpl(c client.Client, log logr.Logger, decoder *utils.ContainerRecordDecoder) *impltypes.ChaosImplPair {
        return &impltypes.ChaosImplPair{
                Name:   "helloworldchaos",
                Object: &v1alpha1.HelloWorldChaos{},
                Impl: &Impl{
                        Client:  c,
                        Log:     log.WithName("helloworldchaos"),
                        decoder: decoder,
                },
                ObjectList: &v1alpha1.HelloWorldChaosList{},
     }
}
var Module = fx.Provide(
         fx.Annotated{
                 Group:  "impl",
                 Target: NewImpl,
         },
)

Chaos Mesh uses the fx library for dependency injection. To register HelloWorldChaos in the controller manager, add a line to controllers/chaosimpl/fx.go:

var AllImpl = fx.Options(
        gcpchaos.Module,
        stresschaos.Module,
        jvmchaos.Module,
        timechaos.Module,
        helloworldchaos.Module // Add a new line. Make sure you have imported helloworldchaos first.
        //...
)

Then in controllers/types/types.go, append the following content into ChaosObjects:

var ChaosObjects = fx.Supply(
       //...
       fx.Annotated{
               Group: "objs",
               Target: Object{
                       Name:   "helloworldchaos",
                       Object: &v1alpha1.HelloWorldChaos{},
               },
       },
)

Step 4: Build Docker images

Build the production images:
```
make image
```

If you deploy the Kubernetes cluster using minikube, then you need to load images into the cluster:

minikube image load ghcr.io/chaos-mesh/chaos-dashboard:latest
minikube image load ghcr.io/chaos-mesh/chaos-mesh:latest
minikube image load ghcr.io/chaos-mesh/chaos-daemon:latest

Step 5: Run HelloWorldChaos

In this step, you need to deploy Chaos Mesh with your latest changes to test HelloWorldChaos.

kubectl create -f manifests/crd.yaml

You can see HelloWorldChaos is created from the output:

customresourcedefinition.apiextensions.k8s.io/helloworldchaos.chaos-mesh.org created

Now you can get the CRD of HelloWorldChaos using the command below:

kubectl get crd helloworldchaos.chaos-mesh.org

Deploy Chaos Mesh:

helm install chaos-mesh helm/chaos-mesh -n=chaos-mesh --set controllerManager.leaderElection.enabled=false,dashboard.securityMode=false

To verify the deployment is successful, you can check all Pods in the chaos-mesh namespace:

kubectl get pods --namespace chaos-mesh -l app.kubernetes.io/instance=chaos-mesh

Deploy a deployment for testing, we can use an example echo server from minikube docs:

kubectl create deployment hello-minikube --image=kicbase/echo-server:1.0

Wait to see the pod is running:

kubectl get pods

Example output:

NAME                              READY   STATUS    RESTARTS   AGE
hello-minikube-77b6f68484-dg4sw   1/1     Running   0          2m

Create a hello.yaml file with the following content:

apiVersion: chaos-mesh.org/v1alpha1
kind: HelloWorldChaos
metadata:
  name: hello-world
  namespace: chaos-mesh
spec:
  selector:
    labelSelectors:
      app: hello-minikube
  mode: one
  duration: 1h

Run:

kubectl apply -f hello.yaml
# helloworldchaos.chaos-mesh.org/hello-world created

Now you can check if chaos-controller-manager has Hello world! in its logs:

kubectl logs -n chaos-mesh chaos-controller-manager-xxx

Example output:

2023-07-16T06:19:40.068Z INFO records records/controller.go:149 apply chaos {"id": "default/hello-minikube-77b6f68484-dg4sw/echo-server"}
2023-07-16T06:19:40.068Z INFO helloworldchaos helloworldchaos/types.go:26 Hello world!

What’s Next

If you encounter any problems during the process, create an issue in the Chaos Mesh repository.

In the next section, we’ll learn more about how to extend the behavior of HelloWorldChaos.