JVMChaos introduction

JVMChaos can inject faults into JVM of the target container, which can be applied for any application that uses JVM as the runtime environment. Currently, JVMChaos uses chaosblade-exec-jvm to inject faults into the JVM. JVMChaos supports the following fault types:

  • Specify return value
  • Method Delay
  • Throw custom exceptions
  • Out of memory
  • Fill JVM Code Cache
  • CPU full load in Java
  • Perform customized Groovy or Java script

Usage restrictions

Currently, Chaos Mesh uses MutatingAdmissionWebhook to modify the Pod definition and loads Java agent using Init Containers instead of loading java agent at runtime. Therefore, there are some restrictions when you use JVMChaos:

  • The Webhook support needs to be enabled in Kubernetes.
  • For Pods that exist before you configure MutatingAdmissionWebhook for the namespace, they will not be affected by JVMChaos.
  • JVM in all containers under namespace will load Java agent at the startup stage, and JVMChaos will not unload Java agent after being deleted. If you hope to clean up the Java agent considering the impact that Java agent may have on program behaviors or performance, you can move the workload out of the namespace.

In addition, creating JVMChaos using Chaos Dashboard is not supported currently.

Create experiments using YAML files

The following example shows you the methods and effects of JVMChaos with a specified return value. The YAML files referred in the following steps can be found in examples/jvm. The default work directory for the following steps is in examples/jvm. The default namespace installed by Chaos Mesh is chaos-testing.

1. Create a namespace and configure MutatingAdmissionWebhook

Create the namespace for the application:

  1. kubectl create ns app

Add the admission-webhook=enabled label for the app namespace, and allow the MutatingAdmissionWebhook of Chaos Mesh to modify Pods under the namespace.

  1. kubectl label ns app admission-webhook=enabled

Prepare a template for modifications to be made by JVMChaos:

  1. kubectl apply -f sidecar-template.yaml
  2. kubectl apply -f sidecar.yaml

2. Create target applications

jvm-chaos-demo is a simple Spring Boot application and here serves as a target application. A target application is defined in example/jvm/app.yaml as follows:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: springboot-jvmchaos-demo
  5.   namespace: app
  6. spec:
  7.   replicas: 1
  8.   selector:
  9.     matchLabels:
  10.       app: springboot-jvmchaos-demo
  11.   template:
  12.     metadata:
  13.       annotations:
  14.         admission-webhook.chaos-mesh.org/request: jvmchaos-sidecar
  15.       creationTimestamp: null
  16.       labels:
  17.         app: springboot-jvmchaos-demo
  18.     spec:
  19.       containers:
  20.         - image: 'gallardot/chaosmesh-jvmchaos-sample:latest'
  21.           imagePullPolicy: IfNotPresent
  22.           name: springboot-jvmchaos-demo

In the above example, the annotation with the value admission-webhook.chaos-mesh.org/request: jvmchaos-sidecar corresponds to the name of ConfigMap in sidecar.yaml of step 1.

Build application deployment:

  1. kubectl apply -f app.yaml

Execute kubectl -n app get pods, and then you can find 1 Pod with a name like springboot-jvmchaos-demo-777d94c5b9-7t7l2 under the namespace app. Wait for READY changes to 1/1 and then execute the following commands:

  1. kubectl -n app get pods

The result is as follows:

  1. NAME                                        READY   STATUS    RESTARTS   AGE
  2. springboot-jvmchaos-demo-777d94c5b9-7t7l2   1/1     Running   0          21s

3. Obseve application behaviors before injecting faults

You can observe the behavior of the jvm-chaos-demo application before injecting faults, for example:

Map the port of Pod to local using kubectl port-forward:

  1. kubectl -n app port-forward pod/springboot-jvmchaos-demo-777d94c5b9-7t7l2 8080:8080

Use curl in another shell session or directly access to http://localhost:8080/hello. Hello firend is expected to be returned:

  1. curl http://localhost:8080/hello
  2. Hello friend

4. Inject JVMChaos and check

The JVMChaos with a specified return value is as follows:

  1. apiVersion: chaos-mesh.org/v1alpha1
  2. kind: JVMChaos
  3. metadata:
  4.   name: jvm-return-example
  5.   namespace: app
  6. spec:
  7.   action: return
  8.   target: jvm
  9.   flags:
  10.     value: 'hello chaos mesh!'
  11.   matchers:
  12.     classname: 'org.chaosmesh.jvm.Application'
  13.     methodname: 'hello'
  14.   mode: one
  15.   selector:
  16.     labelSelectors:
  17.       app: springboot-jvmchaos-demo

JVMChaos modifies the return value of hello method to string hello chaos mesh!.

Inject JVMChaos with a specified value:

  1. kubectl apply -f ./jvm-return-example.yaml

Use curl or directly access to http://localhost:8080/hello, hello chaos mesh! is expected to be returned:

  1. curl http://localhost:8080/hello
  2. hello chaos mesh!

Field description

Parameter Type Description Default value Required Example
action string Indicates the specific fault type. The available fault types include return, script, cfl, oom, ccf, tce, tcf, cpf, tde, and tpf. None Yes return
mode string Indicates how to select Pod. The supported modes include one, all, fixed, fixed-percent, and random-max-percent. None Yes 1
value string Provides parameters for the mode configuration, depending on mode. None No 2
target string Indicates the parameter passed to chaosblade-exec-jvm, representing JVMChaos targets, supporting servlet, psql, jvm, jedis, http, dubbo, rocketmq, tars, mysql, ruid, redisson, rabbitmq, monodb. None Yes jvm
flags map[string]string Indicates parameters passed to chaosblade-exec-jvm and represents the flags of action. None No
matchers map[string]string Indicates parameters passed to chaosblade-execu-jvm and represents the matching of injection points. None No

For the meaning of the value of action, refer to:

Name Meaning
delay Specifies method call delay
return Modifies the return value
script Writes groovy and Java implement scenarios
cfl Java CPU usage overload
oom Out of memory, supporting oom of heap, stack, and metaspaces
ccf JVM code cache fill
tce Throw custom exceptions
cpf Connection pool full
tde Throw the first exception of method declare
tpf Thread pool full

For the details of action, refer to chaos blade document.

For the parameters passed to chaosblade-exec-jvm, Chaos Mesh will merge all fields in flags and matchers as a request body and then send it to chaosblade-exec-jvm. For details, refer to chaosblade-exec-jvm/Protocol.