KernelChaos Experiment

This document describes how to create KernelChaos experiments in Chaos Mesh.

Although KernelChaos targets a certain pod, the performance of other pods are also impacted depending on the specific callchain and frequency. It is because all pods of the same host share the same kernel.

Warning:

This feature is disabled by default. Do not use it in production environment.

Prerequisites

Configuration file

Below is a sample KernelChaos configuration file:

  1. apiVersion: chaos-mesh.org/v1alpha1
  2. kind: KernelChaos
  3. metadata:
  4.   name: kernel-chaos-example
  5.   namespace: chaos-testing
  6. spec:
  7.   mode: one
  8.   selector:
  9.     namespaces:
  10.       - chaos-mount
  11.   failKernRequest:
  12.     callchain:
  13.       - funcname: '__x64_sys_mount'
  14.     failtype: 0

For more sample files, see examples. You can edit them as needed.

Description:

  • mode defines the mode to select pods.

  • selector specifies the target pods for chaos injection. For more details, see Define the Scope of Chaos Experiment.

  • failkernRequest defines the specified injection mode (kmalloc, bio, etc.) with a call chain and an optional set of predicates. The fields are:

    • failtype indicates what to fail, can be set to 0 / 1 / 2.

      • If 0, indicates slab to fail (should_failslab)
      • If 1, indicates alloc_page to fail (should_fail_alloc_page)
      • If 2, indicates bio to fail (should_fail_bio)

      For more information, see fault-injection and inject_example.

    • callchain indicates a special call chain, such as:

      1. ext4_mount
      2. -> mount_subtree
      3.    -> ...
      4.       -> should_failslab
  1.     With an optional set of predicates and an optional set of parameters, which used with predicates. See [call chain and predicate examples](https://github.com/chaos-mesh/bpfki/tree/develop/examples) to learn more. If there is no special call chain, just keep `callchain` empty, which means it will fail at any call chain with slab alloc (eg: kmalloc).
  3.     The challchain's type is an array of frames, the frame has three fields:
  5.     -   **funcname** can be find from kernel source or `/proc/kallsyms`, such as `ext4_mount`.
  6.     -   **parameters** is used with predicate, for example, if you want to inject slab error in `d_alloc_parallel(struct dentry *parent, const struct qstr *name)` with a special name `bananas`, you need to set it to `struct dentry *parent, const struct qstr *name`otherwise omit it.
  7.     -   **predicate** accesses the arguments of this frame, example with parameters's, you can set it to `STRNCMP(name->name, "bananas", 8)` to make inject only with it, or omit it to inject for all d\_alloc\_parallel call chain.
  8. -   **headers** indicates the appropriate kernel headers you need. Eg: "linux/mmzone.h", "linux/blkdev.h" and so on.
  10. -   **probability** indicates the fails with probability. If you want 1%, please set this field with `1`.
  12. -   **times** indicates the max times of fails.

  • duration defines the duration for each chaos experiment. In the sample file above, the time chaos lasts for 10 seconds.

  • scheduler defines the scheduler rules for the running time of the chaos experiment. For more rule information, see robfig/cron

Usage

KernelChaos’s function is similar to inject.py, which guarantees the appropriate erroneous return of the specified injection mode (kmalloc, bio, etc.) given a call chain and an optional set of predicates.

You can read inject_example.txt to learn more.

Below is a sample program:

  1. #include <sys/mount.h>
  2. #include <stdio.h>
  3. #include <string.h>
  4. #include <errno.h>
  5. #include <unistd.h>
  8. int main(void) {
  9.     int ret;
  10.     while (1) {
  11.         ret = mount("/dev/sdc", "/mnt", "ext4",
  12.                 MS_MGC_VAL | MS_RDONLY | MS_NOSUID, "");
  13.         if (ret < 0)
  14.             fprintf(stderr, "%s\n", strerror(errno));
  15.         sleep(1);
  16.         ret = umount("/mnt");
  17.         if (ret < 0)
  18.             fprintf(stderr, "%s\n", strerror(errno));
  19.     }
  20. }

During the injection, the output is similar to this:

  1. > Cannot allocate memory
  2. > Invalid argument
  3. > Cannot allocate memory
  4. > Invalid argument
  5. > Cannot allocate memory
  6. > Invalid argument
  7. > Cannot allocate memory
  8. > Invalid argument
  9. > Cannot allocate memory
  10. > Invalid argument

Limitation

Although we use container_id to limit fault injection, but some behaviors might trigger systemic behaviors. For example:

When failtype is 1, it means that physical page allocation will fail. If the behavior is continuous in a very short time (eg: `while (1) {memset(malloc(1M), '1', 1M)}), the system’s oom-killer will be awakened to release memory. So the container_id will lose limit to oom-killer.