Add Facilities to Chaos Daemon

In Develop a new chaos, we have added a new chaos type named HelloWorldChaos, which will print hello world in chaos-controller-manager. To actually run the chaos, we need to configure some facilities for Chaos Daemon - so that controller-manager can select the specified Pods according to the chaos configuration and sends the chaos request to the chaos-daemon corresponding to these Pods. Once these are done, the chaos-daemon could run the chaos at last.

This guide covers the following steps:

Add selector for HelloWorldChaos

In Chaos Mesh, we have defined the spec.selector field to specify the scope of the chaos by namespace, labels, annotation, etc. You can refer to Define the Scope of Chaos Experiment for more information. To specify the Pods for HelloWorld chaos:

  1. Add the Spec field in HelloWorldChaos:

    ``` // HelloWorldChaos is the Schema for the helloworldchaos API type HelloWorldChaos struct {

    1. metav1.TypeMeta   `json:",inline"`
    2. metav1.ObjectMeta `json:"metadata,omitempty"`
  1.     // Spec defines the behavior of a pod chaos experiment
  2.     Spec HelloWorldSpec `json:"spec"`
  3. }
  6. type HelloWorldSpec struct {
  7.     Selector SelectorSpec `json:"selector"`
  8. }
  11. // GetSelector is a getter for Selector (for implementing SelectSpec)
  12. func (in *HelloWorldSpec) GetSelector() SelectorSpec {
  13.     return in.Selector
  14. }
  15. ```

  1. Generate boilerplate functions for the spec field. This is required to integrate the resource in Chaos Mesh.

    1. make generate

Implement the gRPC interface

In order for chaos-daemon to accept requests from chaos-controller-manager, a new gRPC interface is required for chaos-controller-manager and chaos-daemon. Take the steps below to add the gRPC interface:

  1. Add the RPC in chaosdaemon.proto.

    ``` service chaosDaemon {

    1. ...
  1.     rpc ExecHelloWorldChaos(ExecHelloWorldRequest) returns (google.protobuf.Empty) {}
  2. }
  5. message ExecHelloWorldRequest {
  6.     string container_id = 1;
  7. }
  8. ```
  10. You will need to update golang code generated by this proto file:
  12. ```
  13. make proto
  14. ```

  1. Implement the gRPC service in chaos-daemon.

    Add a new file named helloworld_server.go under chaosdaemon, with the content as below:

    ``` package chaosdaemon

  1. import (
  2.     "context"
  3.     "fmt"
  6.     "github.com/golang/protobuf/ptypes/empty"
  9.     "github.com/chaos-mesh/chaos-mesh/pkg/bpm"
  10.     pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb"
  11. )
  14. func (s *daemonServer) ExecHelloWorldChaos(ctx context.Context, req *pb.ExecHelloWorldRequest) (*empty.Empty, error) {
  15.     log.Info("ExecHelloWorldChaos", "request", req)
  18.     pid, err := s.crClient.GetPidFromContainerID(ctx, req.ContainerId)
  19.     if err != nil {
  20.         return nil, err
  21.     }
  24.     cmd := bpm.DefaultProcessBuilder("sh", "-c", fmt.Sprintf("echo 'hello' `hostname`")).
  25.         SetNS(pid, bpm.UtsNS).
  26.         SetContext(ctx).
  27.         Build()
  28.     out, err := cmd.Output()
  29.     if err != nil {
  30.         return nil, err
  31.     }
  32.     if len(out) != 0 {
  33.         log.Info("cmd output", "output", string(out))
  34.     }
  37.     return &empty.Empty{}, nil
  38. }
  39. ```
  41. After `chaos-daemon` receives the `ExecHelloWorldChaos` request, `chaos-daemon` will print `hello` to this container's hostname.

  1. Send gRPC requests in reconcile.

    When a CRD object is updated (for example: create or delete), we need to compare the state specified in the object against the actual state, and then perform operations to make the actual cluster state reflect the state specified. This process is called reconcile.

    For HelloworldChaos, chaos-controller-manager needs to send chaos request to chaos-daemon in reconcile. To do this, we need to update the file controllers/helloworldchaos/types.go created in Develop a New Chaos with the content as below:

    ``` package helloworldchaos

  1. import (
  2.     "context"
  3.     "errors"
  4.     "fmt"
  7.     "k8s.io/apimachinery/pkg/runtime"
  8.     ctrl "sigs.k8s.io/controller-runtime"
  11.     "github.com/chaos-mesh/chaos-mesh/api/v1alpha1"
  12.     "github.com/chaos-mesh/chaos-mesh/controllers/common"
  13.     pb "github.com/chaos-mesh/chaos-mesh/pkg/chaosdaemon/pb"
  14.     "github.com/chaos-mesh/chaos-mesh/pkg/router"
  15.     ctx "github.com/chaos-mesh/chaos-mesh/pkg/router/context"
  16.     end "github.com/chaos-mesh/chaos-mesh/pkg/router/endpoint"
  17.     "github.com/chaos-mesh/chaos-mesh/pkg/utils"
  18. )
  21. // endpoint is dns-chaos reconciler
  22. type endpoint struct {
  23.     ctx.Context
  24. }
  27. // Apply applies helloworld chaos
  28. func (r *endpoint) Apply(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error {
  29.     r.Log.Info("Apply helloworld chaos")
  30.     helloworldchaos, ok := chaos.(*v1alpha1.HelloWorldChaos)
  31.     if !ok {
  32.         return errors.New("chaos is not helloworldchaos")
  33.     }
  36.     pods, err := utils.SelectPods(ctx, r.Client, r.Reader, helloworldchaos.Spec.GetSelector())
  37.     if err != nil {
  38.         return err
  39.     }
  42.     for _, pod := range pods {
  43.         daemonClient, err := utils.NewChaosDaemonClient(ctx, r.Client, &pod, common.ControllerCfg.ChaosDaemonPort)
  44.         if err != nil {
  45.             r.Log.Error(err, "get chaos daemon client")
  46.             return err
  47.         }
  48.         defer daemonClient.Close()
  49.         if len(pod.Status.ContainerStatuses) == 0 {
  50.             return fmt.Errorf("%s %s can't get the state of container", pod.Namespace, pod.Name)
  51.         }
  54.         containerID := pod.Status.ContainerStatuses[0].ContainerID
  57.         _, err = daemonClient.ExecHelloWorldChaos(ctx, &pb.ExecHelloWorldRequest{
  58.             ContainerId: containerID,
  59.         })
  60.         if err != nil {
  61.             return err
  62.         }
  63.     }
  66.     return nil
  67. }
  70. // Recover means the reconciler recovers the chaos action
  71. func (r *endpoint) Recover(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error {
  72.     return nil
  73. }
  76. // Object would return the instance of chaos
  77. func (r *endpoint) Object() v1alpha1.InnerObject {
  78.     return &v1alpha1.HelloWorldChaos{}
  79. }
  82. func init() {
  83.     router.Register("helloworldchaos", &v1alpha1.HelloWorldChaos{}, func(obj runtime.Object) bool {
  84.         return true
  85.     }, func(ctx ctx.Context) end.Endpoint {
  86.         return &endpoint{
  87.             Context: ctx,
  88.         }
  89.     })
  90. }
  91. ```
  93. > **Notes:**
  94. > 
  95. > In our case here, the `Recover` function does nothing because `HelloWorldChaos` only prints some log and doesn't change anything. You may need to implement the `Recover` function in your development.

Verify your chaos

Now you are all set. It’s time to verify the chaos type you just created. Take the steps below:

  1. Make the Docker image. Refer to Make the Docker image.

  2. Upgrade Chaos Mesh. Since we have already installed Chaos Mesh in Develop a New Chaos, we only need to restart it with the latest image:

    1. kubectl rollout restart deployment chaos-controller-manager -n chaos-testing
    2. kubectl rollout restart daemonset chaos-daemon -n chaos-testing

  3. Deploy the Pods for test:

    1. kubectl apply -f https://raw.githubusercontent.com/chaos-mesh/apps/master/ping/busybox-statefulset.yaml

    This command deploys two Pods in the busybox namespace.

  4. Create the chaos YAML file:

    1. apiVersion: chaos-mesh.org/v1alpha1
    2. kind: HelloWorldChaos
    3. metadata:
    4.   name: busybox-helloworld-chaos
    5. spec:
    6.   selector:
    7.     namespaces:
    8.       - busybox

  5. Apply the chaos:

    1. kubectl apply -f /path/to/helloworld.yaml

  6. Verify your chaos. There are different logs to check to see whether your chaos works as expected:

    • Check the log of chaos-controller-manager:

      1. kubectl logs chaos-controller-manager-{pod-post-fix} -n chaos-testing

      The log is as follows:

      1. 2020-09-09T09:13:36.018Z        INFO    controllers.HelloWorldChaos     Reconciling helloworld chaos    {"reconciler": "helloworldchaos"}
      2. 2020-09-09T09:13:36.018Z        INFO    controllers.HelloWorldChaos     Apply helloworld chaos  {"reconciler": "helloworldchaos"}

    • Check the log of chaos-daemon:

      1. kubectl logs chaos-daemon-{pod-post-fix} -n chaos-testing

      The log is as follows:

      1. 2020-09-09T09:13:36.036Z        INFO    chaos-daemon-server     exec hello world chaos  {"request": "container_id:\"docker://8f2918ee05ed587f7074a923cede3bbe5886277faca95d989e513f7b7e831da5\" "}
      2. 2020-09-09T09:13:36.044Z        INFO    chaos-daemon-server     build command   {"command": "nsenter -u/proc/45664/ns/uts -- sh -c echo 'hello' `hostname`"}
      3. 2020-09-09T09:13:36.058Z        INFO    chaos-daemon-server     cmd output      {"output": "hello busybox-1\n"}
      4. 2020-09-09T09:13:36.064Z        INFO    chaos-daemon-server     exec hello world chaos  {"request": "container_id:\"docker://53e982ba5593fa87648edba665ba0f7da3f58df67f8b70a1354ca00447c00524\" "}
      5. 2020-09-09T09:13:36.066Z        INFO    chaos-daemon-server     build command   {"command": "nsenter -u/proc/45620/ns/uts -- sh -c echo 'hello' `hostname`"}
      6. 2020-09-09T09:13:36.070Z        INFO    chaos-daemon-server     cmd output      {"output": "hello busybox-0\n"}

      We can see the chaos-daemon prints hello to these two Pods.