Add Facilities to Chaos Daemon

In Develop a new chaos, we have added a new chaos type named HelloWorldChaos, which will print hello world in chaos-controller-manager. To actually run the chaos, we need to configure some facilities for Chaos Daemon - so that controller-manager can select the specified Pods according to the chaos configuration and sends the chaos request to the chaos-daemon corresponding to these Pods. Once these are done, the chaos-daemon could run the chaos at last.

This guide covers the following steps:

Add selector for HelloWorldChaos

In Chaos Mesh, we have defined the spec.selector field to specify the scope of the chaos by namespace, labels, annotation, etc. You can refer to Define the Scope of Chaos Experiment for more information. To specify the Pods for HelloWorld chaos:

  1. Add the Spec field in HelloWorldChaos:

    ``` // HelloWorldChaos is the Schema for the helloworldchaos API type HelloWorldChaos struct {

    1. metav1.TypeMeta   `json:",inline"`
    2. metav1.ObjectMeta `json:"metadata,omitempty"`
  1.     // Spec defines the behavior of a pod chaos experiment
  2.     Spec HelloWorldSpec `json:"spec"`
  3. }
  6. type HelloWorldSpec struct {
  7.     Selector SelectorSpec `json:"selector"`
  8. }
  11. // GetSelector is a getter for Selector (for implementing SelectSpec)
  12. func (in *HelloWorldSpec) GetSelector() SelectorSpec {
  13.     return in.Selector
  14. }
  15. ```

  1. Generate boilerplate functions for the spec field. This is required to integrate the resource in Chaos Mesh.

    1. make generate

Implement the gRPC interface

In order for chaos-daemon to accept requests from chaos-controller-manager, a new gRPC interface is required for chaos-controller-manager and chaos-daemon. Take the steps below to add the gRPC interface:

  1. Add the RPC in chaosdaemon.proto.

    ``` service chaosDaemon {

    1. ...
  1.     rpc ExecHelloWorldChaos(ExecHelloWorldRequest) returns (google.protobuf.Empty) {}
  2. }
  5. message ExecHelloWorldRequest {
  6.     string container_id = 1;
  7. }
  8. ```
  12. You will need to update golang code generated by this proto file:
  14. ```
  15. make proto
  16. ```

  1. Implement the gRPC service in chaos-daemon.

    Add a new file named helloworld_server.go under chaosdaemon, with the content as below:

    ``` package chaosdaemon

  1. import (
  2.     "context"
  3.     "fmt"
  6.     "github.com/golang/protobuf/ptypes/empty"
  9.     "github.com/chaos-mesh/chaos-mesh/pkg/bpm"
  10.     pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb"
  11. )
  14. func (s *daemonServer) ExecHelloWorldChaos(ctx context.Context, req *pb.ExecHelloWorldRequest) (*empty.Empty, error) {
  15.     log.Info("ExecHelloWorldChaos", "request", req)
  18.     pid, err := s.crClient.GetPidFromContainerID(ctx, req.ContainerId)
  19.     if err != nil {
  20.         return nil, err
  21.     }
  24.     cmd := bpm.DefaultProcessBuilder("sh", "-c", fmt.Sprintf("echo 'hello' `hostname`")).
  25.         SetNS(pid, bpm.UtsNS).
  26.         SetContext(ctx).
  27.         Build()
  28.     out, err := cmd.Output()
  29.     if err != nil {
  30.         return nil, err
  31.     }
  32.     if len(out) != 0 {
  33.         log.Info("cmd output", "output", string(out))
  34.     }
  37.     return &empty.Empty{}, nil
  38. }
  39. ```
  43. After `chaos-daemon` receives the `ExecHelloWorldChaos` request, `chaos-daemon` will print `hello` to this container's hostname.

  1. Send gRPC requests in reconcile.

    When a CRD object is updated (for example: create or delete), we need to compare the state specified in the object against the actual state, and then perform operations to make the actual cluster state reflect the state specified. This process is called reconcile.

    For HelloworldChaos, chaos-controller-manager needs to send chaos request to chaos-daemon in reconcile. To do this, we need to update the file controllers/helloworldchaos/types.go created in Develop a New Chaos with the content as below:

    ``` package helloworldchaos

  1. import (
  2.     "context"
  3.     "errors"
  4.     "fmt"
  7.     "k8s.io/apimachinery/pkg/runtime"
  8.     ctrl "sigs.k8s.io/controller-runtime"
  11.     "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
  12.     "github.com/chaos-mesh/chaos-mesh/controllers/common"
  13.     "github.com/chaos-mesh/chaos-mesh/controllers/config"
  14.     pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb"
  15.     "github.com/chaos-mesh/chaos-mesh/pkg/router"
  16.     ctx "github.com/chaos-mesh/chaos-mesh/pkg/router/context"
  17.     end "github.com/chaos-mesh/chaos-mesh/pkg/router/endpoint"
  18.     "github.com/chaos-mesh/chaos-mesh/pkg/selector"
  19.     "github.com/chaos-mesh/chaos-mesh/pkg/utils"
  20. )
  23. type endpoint struct {
  24.     ctx.Context
  25. }
  28. // Apply applies helloworld chaos
  29. func (r *endpoint) Apply(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error {
  30.     r.Log.Info("Apply helloworld chaos")
  31.     helloworldchaos, ok := chaos.(*v1alpha1.HelloWorldChaos)
  32.     if !ok {
  33.         return errors.New("chaos is not helloworldchaos")
  34.     }
  37.     pods, err := selector.SelectAndFilterPods(ctx, r.Client, r.Reader, &helloworldchaos.Spec, config.ControllerCfg.ClusterScoped, config.ControllerCfg.TargetNamespace, config.ControllerCfg.AllowedNamespaces, config.ControllerCfg.IgnoredNamespaces)
  38.     if err != nil {
  39.         r.Log.Error(err, "failed to select and filter pods")
  40.         return err
  41.     }
  44.     for _, pod := range pods {
  45.         daemonClient, err := utils.NewChaosDaemonClient(ctx, r.Client, &pod, common.ControllerCfg.ChaosDaemonPort)
  46.         if err != nil {
  47.             r.Log.Error(err, "get chaos daemon client")
  48.             return err
  49.         }
  50.         defer daemonClient.Close()
  51.         if len(pod.Status.ContainerStatuses) == 0 {
  52.             return fmt.Errorf("%s %s can't get the state of container", pod.Namespace, pod.Name)
  53.         }
  56.         containerID := pod.Status.ContainerStatuses[0].ContainerID
  59.         _, err = daemonClient.ExecHelloWorldChaos(ctx, &pb.ExecHelloWorldRequest{
  60.             ContainerId: containerID,
  61.         })
  62.         if err != nil {
  63.             return err
  64.         }
  65.     }
  68.     return nil
  69. }
  72. // Recover means the reconciler recovers the chaos action
  73. func (r *endpoint) Recover(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error {
  74.     return nil
  75. }
  78. // Object would return the instance of chaos
  79. func (r *endpoint) Object() v1alpha1.InnerObject {
  80.     return &v1alpha1.HelloWorldChaos{}
  81. }
  84. func init() {
  85.     router.Register("helloworldchaos", &v1alpha1.HelloWorldChaos{}, func(obj runtime.Object) bool {
  86.         return true
  87.     }, func(ctx ctx.Context) end.Endpoint {
  88.         return &endpoint{
  89.             Context: ctx,
  90.         }
  91.     })
  92. }
  93. ```
  97. > **Notes:**
  98. > 
  99. > In our case here, the `Recover` function does nothing because `HelloWorldChaos` only prints some log and doesn't change anything. You may need to implement the `Recover` function in your development.

Verify your chaos

Now you are all set. It’s time to verify the chaos type you just created. Take the steps below:

  1. Make the Docker image. Refer to Make the Docker image.

  2. Upgrade Chaos Mesh. Since we have already installed Chaos Mesh in Develop a New Chaos, we only need to restart it with the latest image:

    1. kubectl rollout restart deployment chaos-controller-manager -n chaos-testing
    2. kubectl rollout restart daemonset chaos-daemon -n chaos-testing

  1. Deploy the Pods for test:

    1. kubectl apply -f https://raw.githubusercontent.com/chaos-mesh/apps/master/ping/busybox-statefulset.yaml
  1. This command deploys two Pods in the `busybox` namespace.

  1. Create the chaos YAML file:

    1. apiVersion: chaos-mesh.org/v1alpha1
    2. kind: HelloWorldChaos
    3. metadata:
    4.   name: busybox-helloworld-chaos
    5. spec:
    6.   selector:
    7.     namespaces:
    8.       - busybox

  1. Apply the chaos:

    1. kubectl apply -f /path/to/helloworld.yaml

  1. Verify your chaos. There are different logs to check to see whether your chaos works as expected:

    • Check the log of chaos-controller-manager:

      1. kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing
  1.     The log is as follows:
  3.     ```
  4.     2020-09-09T09:13:36.018Z        INFO    controllers.HelloWorldChaos     Reconciling helloworld chaos    {"reconciler": "helloworldchaos"}
  5.     2020-09-09T09:13:36.018Z        INFO    controllers.HelloWorldChaos     Apply helloworld chaos  {"reconciler": "helloworldchaos"}
  6.     ```
  10. -   Check the log of `chaos-daemon`:
  12.     ```
  13.     kubectl logs chaos-daemon-{pod-post-fix} -n chaos-testing
  14.     ```
  18.     The log is as follows:
  20.     ```
  21.     2020-09-09T09:13:36.036Z        INFO    chaos-daemon-server     exec hello world chaos  {"request": "container_id:\"docker://8f2918ee05ed587f7074a923cede3bbe5886277faca95d989e513f7b7e831da5\" "}
  22.     2020-09-09T09:13:36.044Z        INFO    chaos-daemon-server     build command   {"command": "nsenter -u/proc/45664/ns/uts -- sh -c echo 'hello' `hostname`"}
  23.     2020-09-09T09:13:36.058Z        INFO    chaos-daemon-server     cmd output      {"output": "hello busybox-1\n"}
  24.     2020-09-09T09:13:36.064Z        INFO    chaos-daemon-server     exec hello world chaos  {"request": "container_id:\"docker://53e982ba5593fa87648edba665ba0f7da3f58df67f8b70a1354ca00447c00524\" "}
  25.     2020-09-09T09:13:36.066Z        INFO    chaos-daemon-server     build command   {"command": "nsenter -u/proc/45620/ns/uts -- sh -c echo 'hello' `hostname`"}
  26.     2020-09-09T09:13:36.070Z        INFO    chaos-daemon-server     cmd output      {"output": "hello busybox-0\n"}
  27.     ```
  31.     We can see the `chaos-daemon` prints `hello` to these two Pods.