FAQs

Question

Q: If I do not have Kubernetes clusters deployed, can I use Chaos Mesh to create chaos experiments?

No, you can not use Chaos Mesh in this case. But still you can run chaos experiments using command line. Refer to Command Line Usages of Chaos for details.

Q: I have deployed Chaos Mesh and created PodChaos experiments successfully, but I still failed in creating NetworkChaos/TimeChaos Experiment. The log is shown below:

2020-06-18T01:05:26.207Z    ERROR   controllers.TimeChaos   failed to apply chaos on all pods   {"reconciler": "timechaos", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp xx.xx.xx.xx:xxxxx: connect: connection refused\""}

You can try using the parameter: hostNetwork, as shown below:

# vim helm/chaos-mesh/values.yaml, change hostNetwork from false to true
hostNetwork: true

Q: I just saw ERROR: failed to get cluster internal kubeconfig: command "docker exec --privileged kind-control-plane cat /etc/kubernetes/admin.conf" failed with error: exit status 1 when installing Chaos Mesh with kind. How to fix it?

You can try the following command to fix it:

kind delete cluster

then deploy again.

Debug

Q: Experiment not working after chaos is applied

You can debug as described below:

Execute kubectl describe to check the specified chaos experiment resource.

• If there are NextStart and NextRecover fields under spec, then the chaos will be triggered after NextStart is executed.

• If there are no NextStart and NextRecoverfields in spec, run the following command to get controller-manager’s log and see whether there are errors in it.

  kubectl logs -n chaos-testing chaos-controller-manager-xxxxx (replace this with the name of the controller-manager) | grep "ERROR"

  For error message no pod is selected, run the following command to show the labels and check if the selector is desired.

  kubectl get pods -n yourNamespace --show-labels

If the above steps cannot solve the problem or you encounter other related errors in controller’s log, file an issue or message us in the #project-chaos-mesh channel in the CNCF Slack workspace.

IOChaos

Q: Running chaosfs sidecar container failed, and log shows pid file found, ensure docker is not running or delete /tmp/fuse/pid

The chaosfs sidecar container is continuously restarting, and you might see the following logs at the current sidecar container:

2020-01-19T06:30:56.629Z    INFO    chaos-daemon    Init hookfs
2020-01-19T06:30:56.630Z    ERROR   chaos-daemon    failed to create pid file   {"error": "pid file found, ensure docker is not running or delete /tmp/fuse/pid"}
github.com/go-logr/zapr.(*zapLogger).Error

• Cause: Chaos Mesh uses Fuse to inject I/O failures. It fails if you specify an existing directory as the source path for chaos. This often happens when you try to reuse a persistent volume (PV) with the Retain reclaim policy to request a PersistentVolumeClaims (PVC) resource.
• Solution: In this case, use the following command to change the reclaim policy to Delete:

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'