Simulate Redis Faults

This document introduces how to use Chaosd to simulate Redis faults. This feature uses Golang interfaces in go-redis package and the redis-server command-line tool. You can create experiments either in command-line mode or service mode.

Create experiments using command-line mode

Before creating an experiment, you can run the following command to see the Redis fault types that are supported by Chaosd:

  1. chaosd attack redis -h

The result is as follows:

  1. Redis attack related commands
  2. Usage:
  3. chaosd attack redis [command]
  4. Available Commands:
  5. cache-expiration expire keys in Redis
  6. cache-limit set maxmemory of Redis
  7. cache-penetration penetrate cache
  8. sentinel-restart restart sentinel
  9. sentinel-stop stop sentinel
  10. Flags:
  11. -h, --help help for redis
  12. Global Flags:
  13. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  14. --uid string the experiment ID
  15. Use "chaosd attack redis [command] --help" for more information about a command.

Currently, Chaosd supports simulating cache expiration, cache penetration, cache limit, sentinel restart, and sentinel stop.

Simulate cache expiration using command-line mode

The meaning of this command is the same as EXPIRE in Redis. For more details, refer to the Redis official documentation.

Simulate Redis Faults - 图1note

Currently, Chaosd does not support recovering keys that have executed cache-expiration, so please backup them in advance if you want to recover them.

Commands for cache expiration

  1. chaosd attack redis cache-expiration -h

The result is as follows:

  1. expire keys in Redis
  2. Usage:
  3. chaosd attack redis cache-expiration [flags]
  4. Flags:
  5. -a, --addr string The address of redis server
  6. --expiration string The expiration of the key. A expiration string should be able to be converted to a time duration, such as "5s" or "30m" (default "0")
  7. -h, --help help for cache-expiration
  8. -k, --key string The key to be set a expiration, default expire all keys
  9. --option string The additional options of expiration, only NX, XX, GT, LT supported
  10. -p, --password string The password of server
  11. Global Flags:
  12. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  13. --uid string the experiment ID

Configuration description for cache expiration

Configuration itemAbbreviationTypeDescriptionValue
addrastringThe address and port of Redis server to be injected into the fault, for example 127.0.0.1:6379Default value: “”
expirationNonestringThe specified key will be expired after expiration arrivesDefault value: “0”. Make sure that the string is in the format supported by time.Duration
keykstringThe key to be expiredDefault value: “”, which means the expiration is set for all keys
optionNonestringAdditional options for expiration. Only versions of Redis after 7.0.0 support this flagDefault value: “”. Only NX, XX, GT, and LT are supported
passwordpstringThe password to log in to the serverDefault value: “”

Example for simulating cache expiration

  1. chaosd attack redis cache-expiration -a 127.0.0.1:6379 --option GT --expiration 1m

Simulate cache limit using command-line mode

Commands for cache limit

  1. chaosd attack redis cache-limit -h

The result is as follows:

  1. set maxmemory of Redis
  2. Usage:
  3. chaosd attack redis cache-limit [flags]
  4. Flags:
  5. -a, --addr string The address of redis server
  6. -h, --help help for cache-limit
  7. -p, --password string The password of server
  8. --percent string The percentage of maxmemory
  9. -s, --size string The size of cache (default "0")
  10. Global Flags:
  11. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  12. --uid string the experiment ID

Configuration description for cache limit

Configuration itemAbbreviationTypeDescriptionValue
addrastringThe address and port of Redis server to be injected into the fault, such as 127.0.0.1:6379Default value: “”
passwordpstringThe password to log in to the serverDefault value: “”
percentNonestringSpecifies maxmemory as a percentage of the original valueDefault value: “”
sizesstringSpecifies the size of maxmemoryDefault 0, which means no limitation of memory

Example for simulating cache limit

  1. chaosd attack redis cache-limit -a 127.0.0.1:6379 -s 256M

Simulate cache penetration using command-line mode

This command will send the specified number of GET requests to Redis server as quickly as possible using Redis Pipeline. Since the requested keys do not exist on the Redis server, these requests will cause a cache penetration phenomenon.

Commands for cache penetration

  1. chaosd attack redis cache-penetration -h

The result is as follows:

  1. penetrate cache
  2. Usage:
  3. chaosd attack redis cache-penetration [flags]
  4. Flags:
  5. -a, --addr string The address of redis server
  6. -h, --help help for cache-penetration
  7. -p, --password string The password of server
  8. --request-num int The number of requests
  9. Global Flags:
  10. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  11. --uid string the experiment ID

Configuration description for cache penetration

Configuration itemAbbreviationTypeDescriptionValue
addrastringThe address and port of Redis server to be injected into the fault, such as 127.0.0.1:6379Default value: “”
passwordpstringThe password to log in to the serverDefault value: “”
request-numNoneintSpecifies the number of requests to be sent to the Redis serverDefault value: 0

Example for simulating cache penetration

  1. chaosd attack redis cache-penetration -a 127.0.0.1:6379 --request-num 100000

Simulate Sentinel restart using command-line mode

Commands for Sentinel restart

  1. chaosd attack redis sentinel-restart -h

The result is as follows:

  1. restart sentinel
  2. Usage:
  3. chaosd attack redis sentinel-restart [flags]
  4. Flags:
  5. -a, --addr string The address of redis server
  6. -c, --conf string The config of Redis server
  7. --flush-config Force Sentinel to rewrite its configuration on disk (default true)
  8. -h, --help help for sentinel-restart
  9. -p, --password string The password of server
  10. --redis-path string The path of the redis-server command
  11. Global Flags:
  12. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  13. --uid string the experiment ID

Configuration description for Sentinel restart

Configuration itemAbbreviationTypeDescriptionValue
addrastringThe address and port of Sentinel to be injected into the fault, such as 127.0.0.1:26379Default value: “”
confcstringSpecifies the path of Sentinel config file, this file will be used to revover the SentinelDefault value: “”
flush-configNoneboolForces Sentinel to rewrite its configuration on disk, including the current Sentinel stateDefault value: true
passwordpstringThe password to log in to the serverDefault value: “”
redis-pathNonestringSpecifies the path of redis-server command-line toolDefault value: “”

Example for simulating Sentinel restart

  1. chaosd attack redis sentinel-restart -a 127.0.0.1:26379 --conf /home/redis-test/sentinel-26379.conf

Simulate Sentinel stop using command-line mode

Commands for Sentinel stop

  1. chaosd attack redis sentinel-stop -h

The result is as follows:

  1. stop sentinel
  2. Usage:
  3. chaosd attack redis sentinel-stop [flags]
  4. Flags:
  5. -a, --addr string The address of redis server
  6. -c, --conf string The config path of Redis server
  7. --flush-config Force Sentinel to rewrite its configuration on disk (default true)
  8. -h, --help help for sentinel-stop
  9. -p, --password string The password of server
  10. --redis-path string The path of the redis-server command
  11. Global Flags:
  12. --log-level string the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'
  13. --uid string the experiment ID

Configuration description for Sentinel stop

Configuration itemAbbreviationTypeDescriptionValue
addrastringThe address and port of Sentinel to be injected into the fault, such as 127.0.0.1:26379Default value: “”
confcstringSpecifies the path of Sentinel configuration file, which is used to recover the SentinelDefault value: “”
flush-configNoneboolForces Sentinel to rewrite its configuration on disk, including the current Sentinel stateDefault value: true
passwordpstringThe password to log in to the serverDefault value: “”
redis-pathNonestringSpecifies the path of redis-server command-line toolDefault value: “”

Example for simulating Sentinel restart

  1. chaosd attack redis sentinel-stop -a 127.0.0.1:26379 --conf /home/redis-test/sentinel-26379.conf

Create Redis fault experiments using service mode

To create experiments using the service mode, follow the instructions below:

  1. Run Chaosd in the service mode:

    1. chaosd server --port 31767
  2. Send a POST HTTP request to the /api/attack/redis path of the Chaosd service.

    1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{fault-configuration}'

    In the above command, you need to configure fault-configuration according to the fault types. For the corresponding parameters, refer to the parameters and examples of each fault type in the following sections.

Simulate Redis Faults - 图2note

When running an experiment, remember to record the UID of the experiment. When you want to end the experiment corresponding to the UID, you need to send a DELETE HTTP request to the /api/attack/{uid} path of the Chaosd service.

Simulate cache expiration using service mode

Parameters for simulating cache expiration

ParameterDescriptionTypeValue
actionAction of the experimentstringset to “corrupt”
addrThe address and port of Redis server to be injected into the fault, such as 127.0.0.1:6379stringDefault value: “”
expirationThe specified key will be expired after expiration arrivesstringDefault value: “0”. Make sure that the string is in the format supported by time.Duration
keyThe key to be expiredstringDefault value: “”, which means the expiration is set for all keys
optionAdditional options for expiration. Only versions of Redis after 7.0.0 support this flagstringDefault value: “”. Only NX, XX, GT, and LT are supported
passwordThe password to log in to the serverDefault value: “”

Example for simulating cache expiration using service mode

  1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{"action":"expiration", "expiration":"1m","addr":"127.0.0.1:6379"}'

Simulate cache limit using service mode

Parameters for simulating cache limit

ParameterDescriptionTypeValue
actionAction of the experimentstringset to “cacheLimit”
addrThe address and port of Redis server to be injected into the fault, such as 127.0.0.1:6379stringDefault value: “”
passwordThe password to log in to the serverstringDefault value: “”
percentSpecifies maxmemory as a percentage of the original valuestringDefault value: “”
sizeSpecifies the size of maxmemorystringDefault 0, which means no limitation of memory

Example for simulating cache limit using service mode

  1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{"action":"cacheLimit", ""addr":"127.0.0.1:6379", "percent":"50%"}'

Simulate cache penetration using service mode

Parameters for simulating cache penetration

ParameterDescriptionTypeValue
actionAction of the experimentstringset to “penetration”
addrThe address and port of Redis server to be injected into the fault, such as 127.0.0.1:6379stringDefault value: “”
passwordThe password to log in to the serverstringDefault value: “”
request-numSpecifies the number of requests to be sent to the Redis serverintDefault value: 0

Example for simulating cache penetration using service mode

  1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{"action":"penetration", ""addr":"127.0.0.1:6379", "request-num":"10000"}'

Simulate Sentinel restart using service mode

Parameters for simulating Sentinel restart

ParameterDescriptionTypeValue
actionAction of the experimentstringset to “restart”
addrThe address and port of Sentinel to be injected into the fault, such as 127.0.0.1:26379stringDefault value: “”
confSpecifies the path of Sentinel configuration file, which is used to recover the SentinelstringDefault value: “”
flush-configForces Sentinel to rewrite its configuration on disk, including the current Sentinel stateboolDefault value: true
passwordThe password to log in to the serverstringDefault value: “”
redis-pathSpecifies the path of redis-server command-line toolstringDefault value: “”

Example for simulating Sentinel restart using service mode

  1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{"action":"restart", ""addr":"127.0.0.1:26379", "conf":"/home/redis-test/sentinel-26379.conf"}'

Simulate Sentinel stop using service mode

Parameters for simulating Sentinel stop

ParameterDescriptionTypeValue
actionAction of the experimentstringset to “stop”
addrThe address and port of Sentinel to be injected into the fault, such as 127.0.0.1:26379stringDefault value: “”
confSpecifies the path of Sentinel configuration file, which is used to recover the SentinelstringDefault value: “”
flush-configForces Sentinel to rewrite its configuration on disk, including the current Sentinel stateboolDefault value: true
passwordThe password to log in to the serverstringDefault value: “”
redis-pathSpecifies the path of redis-server command-line toolstringDefault value: “”

Example for simulating Sentinel stop using service mode

  1. curl -X POST 127.0.0.1:31767/api/attack/redis -H "Content-Type:application/json" -d '{"action":"stop", ""addr":"127.0.0.1:26379", "conf":"/home/redis-test/sentinel-26379.conf"}'