This document introduces how to use Chaosd to simulate network faults. The simulations can be completed by modifying network routing and traffic flow control using iptables, ipsets, tc, etc.

Notes

Make sure the NET_SCH_NETEM module is installed in the Linux kernel. If you are using CentOS, you can install the module through the kernel-modules-extra package. Most other Linux distributions have installed it already by default.

Create network fault experiments using command-line mode

This section introduces how to create network fault experiments using command-line mode.

Before creating an experiment, you can run the following command to check the types of network faults supported by Chaosd:

  1. chaosd attack network --help

The output is as follows:

  1. Network attack related commands
  3. Usage:
  4.   chaosd attack network [command]
  6. Available Commands:
  7.   corrupt     corrupt network packet
  8.   delay       delay network
  9.   duplicate   duplicate network packet
  10.   loss        loss network packet
  12. Flags:
  13.   -h, --help   help for network
  15. Global Flags:
  16.       --log-level string   the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'
  18. Use "chaosd attack network [command] --help" for more information about a command.

Currently, you can simulate four experimental scenarios using Chaosd: network corruption, network latency, network duplication, and network loss.

Network corruption

You can run the following command to see the configuration of simulated network corruption using Chaosd:

  1. chaosd attack network corrupt --help

The output is as follows:

  1. corrupt network packet
  3. Usage:
  4.   chaosd attack network corrupt [flags]
  6. Flags:
  7.   -c, --correlation string   correlation is percentage (10 is 10%) (default "0")
  8.   -d, --device string        the network interface to impact
  9.   -e, --egress-port string   only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  10.   -h, --help                 help for corrupt
  11.   -H, --hostname string      only impact traffic to these hostnames
  12.   -i, --ip string            only impact egress traffic to these IP addresses
  13.       --percent string       percentage of packets to corrupt (10 is 10%) (default "1")
  14.   -p, --protocol string      only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  15.   -s, --source-port string   only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  17. Global Flags:
  18.       --log-level string   the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration item Abbreviation Description Value
correlation c The correlation between the percentage of current corrupt occurrence and the previous occurrence. Int. It is a percentage ranging from 0 to 100 (10 is 10%) (“0” by default ).
device d Name of the impacted network interface card. String, such as “eth0”. The value is required.
egress-port e The egress traffic that only impacts specific destination ports. It can only be configured when the protocol is TCP or UDP. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostname H The host name impacted by traffic. String, such as “chaos-mesh.org”.
ip i The IP address impacted by egress traffic. String, such as “123.123.123.123”.
protocol p The IP protocol impacted by traffic. String. Supported protocols: tcp, udp, icmp, all (all network protocols).
source-port s The egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp. String. Use a ‘,’ to delimit the specific port or to indicate the range of the ports, such as “80,8001:8010”.

Network latency

You can run the following command to see the configuration of simulated network latency using Chaosd:

  1. chaosd attack network delay --help

The output is as follows:

  1. delay network
  3. Usage:
  4.   chaosd attack network delay [flags]
  6. Flags:
  7.   -c, --correlation string   correlation is percentage (10 is 10%) (default "0")
  8.   -d, --device string        the network interface to impact
  9.   -e, --egress-port string   only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  10.   -h, --help                 help for delay
  11.   -H, --hostname string      only impact traffic to these hostnames
  12.   -i, --ip string            only impact egress traffic to these IP addresses
  13.   -j, --jitter string        jitter time, time units: ns, us (or µs), ms, s, m, h.
  14.   -l, --latency string       delay egress time, time units: ns, us (or µs), ms, s, m, h.
  15.   -p, --protocol string      only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  16.   -s, --source-port string   only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  18. Global Flags:
  19.       --log-level string   the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration item Abbreviation Description Value
correlation c The correlation between the current latency and the previous one. Int. It is a percentage ranging from 0 to 100 (10 is 10%) (“0” by default).
device d Name of the impacted network interface card. String, such as “eth0”. The value is required.
egress-port e The egress traffic which only impact specific destination ports. It can only be configured when the protocol is TCP or UDP. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostname H The host name impacted by traffic. String, such as “chaos-mesh.org”.
ip i The IP address impacted by egress traffic. String, such as “123.123.123.123”.
jitter j Range of the length of network delay time. String. The time units can be: ns, us (µs), ms, s, m, h, such as “1ms”.
latency l Length of network delay time. String. The time units can be: ns, us (μs), ms, s, m, h, such as “1ms”.
protocol p The IP protocol impacted by traffic. String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-port s The egress traffic that only impacts specified source ports. It can only be configured when the protocol is TCP or UDP. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Network duplication

You can run the following command to see the configuration of simulated network duplication using Chaosd:

  1. chaosd attack network duplicate --help

The output is as follows:

  1. duplicate network packet
  3. Usage:
  4.   chaosd attack network duplicate [flags]
  6. Flags:
  7.   -c, --correlation string   correlation is percentage (10 is 10%) (default "0")
  8.   -d, --device string        the network interface to impact
  9.   -e, --egress-port string   only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  10.   -h, --help                 help for duplicate
  11.   -H, --hostname string      only impact traffic to these hostnames
  12.   -i, --ip string            only impact egress traffic to these IP addresses
  13.       --percent string       percentage of packets to duplicate (10 is 10%) (default "1")
  14.   -p, --protocol string      only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  15.   -s, --source-port string   only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  17. Global Flags:
  18.       --log-level string   the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration item Abbreviation Description Value
correlation c The correlation between the percentage of current duplication occurrence and the previous one. Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “0”).
device d Name of the impacted network interface card. String, such as “eth0”. The value is required.
egress-port e The egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostname H The host name impacted by traffic. String, such as “chaos-mesh.org”.
ip i The IP address impacted by egress traffic. String, such as “123.123.123.123”.
percent none Ratio of network packet duplicate. Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “1”).
protocol p The IP protocol impacted by traffic. String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-port s The egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Network loss

You can run the following command to see the configuration of simulated network loss using Chaosd:

  1. chaosd attack network loss --help

The output is as follows:

  1. loss network packet
  3. Usage:
  4.   chaosd attack network loss [flags]
  6. Flags:
  7.   -c, --correlation string   correlation is percentage (10 is 10%) (default "0")
  8.   -d, --device string        the network interface to impact
  9.   -e, --egress-port string   only impact egress traffic to these destination ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  10.   -h, --help                 help for loss
  11.   -H, --hostname string      only impact traffic to these hostnames
  12.   -i, --ip string            only impact egress traffic to these IP addresses
  13.       --percent string       percentage of packets to drop (10 is 10%) (default "1")
  14.   -p, --protocol string      only impact traffic using this IP protocol, supported: tcp, udp, icmp, all
  15.   -s, --source-port string   only impact egress traffic from these source ports, use a ',' to separate or to indicate the range, such as 80, 8001:8010. It can only be used in conjunction with -p tcp or -p udp
  17. Global Flags:
  18.       --log-level string   the log level of chaosd, the value can be 'debug', 'info', 'warn' and 'error'

The related configuration items are described as follows:

Configuration item Abbreviation Description Value
correlation c The correlation between the percentage of the current network loss and the previous one. Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “0”).
device d Name of the impacted network interface card. String, such as “eth0”. The value is required.
egress-port e The egress traffic that only impacts specified destination ports. It can only be configured when the protocol is TCP or UDP. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.
hostname H The host name impacted by traffic. String, such as “chaos-mesh.org”.
ip i The IP address impacted by egress traffic. String, such as “123.123.123.123”.
percent none Ratio of network packet loss. Int. It is a percentage which range is 0 to 100 (10 is 10%) (default “1”).
protocol p Only impact traffic using this IP protocol. String. It supports the following protocol types: tcp, udp, icmp, all (all network protocols).
source-port s The egress traffic which only impact specific source ports. It can only be configured when the protocol is tcp or udp. String. You need to use a ‘,’ to separate the specific port or to indicate the range of the port, such as “80,8001:8010”.

Examples

Simulate network corruption:

  1. chaosd attack network corrupt -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 4eab1e62-8d60-45cb-ac85-3c17b8ac4825

Simulate network latency:

  1. chaosd attack network delay -d eth0 -i 172.16.4.4 -l 10ms

The output is as follows:

  1. Attack network successfully, uid: 4b23a0b5-e193-4b27-90a7-3e04235f32ab

Simulate network duplication:

  1. chaosd attack network duplicate -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 7bcb74ee-9101-4ae4-82f0-e44c8a7f113c

Simulate network loss:

  1. chaosd attack network loss -d eth0 -i 172.16.4.4 --percent 50

The output is as follows:

  1. Attack network successfully, uid: 1e818adf-3942-4de4-949b-c8499f120265

When running the experiments, remember to save the uid of the experiments. To end a network fault simulation, use recover:

  1. chaosd recover 1e818adf-3942-4de4-949b-c8499f120265

The output is as follows:

  1. Recover 1e818adf-3942-4de4-949b-c8499f120265 successfully

Create network fault experiments using service mode

(To be added)