Jepsen Testing

Jepsen is a tool, written by Kyle Kingsbury, designed to test the partition tolerance of distributed systems. It creates network partitions while fuzzing the system with random operations. The results are analyzed to see if the system violates any of the consistency properties it claims to have.

As part of our Consul testing, we ran a Jepsen test to determine if any consistency issues could be uncovered. In our testing, Consul gracefully recovered from partitions without introducing any consistency issues.

Running the tests

At the moment, testing with Jepsen is rather complex as it requires setting up multiple virtual machines, SSH keys, DNS configuration, and a working Clojure environment. We hope to contribute our Consul testing code upstream and to provide a Vagrant environment for Jepsen testing soon.

Output

Below is the output captured from Jepsen. We ran Jepsen multiple times, and it passed each time. This output is only representative of a single run and has been edited for length. Please reach out on Consul’s Discuss if you would like to reproduce the Jepsen results.

  1. $ lein test :only jepsen.system.consul-test
  3. lein test jepsen.system.consul-test
  4. INFO  jepsen.os.debian - :n5 setting up debian
  5. INFO  jepsen.os.debian - :n3 setting up debian
  6. INFO  jepsen.os.debian - :n4 setting up debian
  7. INFO  jepsen.os.debian - :n1 setting up debian
  8. INFO  jepsen.os.debian - :n2 setting up debian
  9. INFO  jepsen.os.debian - :n4 debian set up
  10. INFO  jepsen.os.debian - :n5 debian set up
  11. INFO  jepsen.os.debian - :n3 debian set up
  12. INFO  jepsen.os.debian - :n1 debian set up
  13. INFO  jepsen.os.debian - :n2 debian set up
  14. INFO  jepsen.system.consul - :n1 consul nuked
  15. INFO  jepsen.system.consul - :n4 consul nuked
  16. INFO  jepsen.system.consul - :n5 consul nuked
  17. INFO  jepsen.system.consul - :n3 consul nuked
  18. INFO  jepsen.system.consul - :n2 consul nuked
  19. INFO  jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
  20. INFO  jepsen.system.consul - :n2 consul nuked
  21. INFO  jepsen.system.consul - :n3 consul nuked
  22. INFO  jepsen.system.consul - :n4 consul nuked
  23. INFO  jepsen.system.consul - :n5 consul nuked
  24. INFO  jepsen.system.consul - :n1 consul nuked
  25. INFO  jepsen.system.consul - :n1 starting consul
  26. INFO  jepsen.system.consul - :n2 starting consul
  27. INFO  jepsen.system.consul - :n4 starting consul
  28. INFO  jepsen.system.consul - :n5 starting consul
  29. INFO  jepsen.system.consul - :n3 starting consul
  30. INFO  jepsen.system.consul - :n3 consul ready
  31. INFO  jepsen.system.consul - :n2 consul ready
  32. INFO  jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
  33. INFO  jepsen.system.consul - :n5 consul ready
  34. INFO  jepsen.system.consul - :n1 consul ready
  35. INFO  jepsen.system.consul - :n4 consul ready
  36. INFO  jepsen.core - Worker 0 starting
  37. INFO  jepsen.core - Worker 2 starting
  38. INFO  jepsen.core - Worker 1 starting
  39. INFO  jepsen.core - Worker 3 starting
  40. INFO  jepsen.core - Worker 4 starting
  41. INFO  jepsen.util - 2   :invoke :read   nil
  42. INFO  jepsen.util - 3   :invoke :cas    [4 4]
  43. INFO  jepsen.util - 0   :invoke :write  4
  44. INFO  jepsen.util - 1   :invoke :write  1
  45. INFO  jepsen.util - 4   :invoke :cas    [4 0]
  46. INFO  jepsen.util - 2   :ok :read   nil
  47. INFO  jepsen.util - 4   :fail   :cas    [4 0]
  48. (Log Truncated...)
  49. INFO  jepsen.util - 4   :invoke :cas    [3 3]
  50. INFO  jepsen.util - 4   :fail   :cas    [3 3]
  51. INFO  jepsen.util - :nemesis    :info   :stop   nil
  52. INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
  53. INFO  jepsen.util - 0   :fail   :read   nil
  54. INFO  jepsen.util - 1   :fail   :write  0
  55. INFO  jepsen.util - :nemesis    :info   :stop   nil
  56. INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
  57. INFO  jepsen.core - nemesis done
  58. INFO  jepsen.core - Worker 3 done
  59. INFO  jepsen.util - 1   :invoke :read   nil
  60. INFO  jepsen.core - Worker 2 done
  61. INFO  jepsen.core - Worker 4 done
  62. INFO  jepsen.core - Worker 0 done
  63. INFO  jepsen.util - 1   :ok :read   3
  64. INFO  jepsen.core - Worker 1 done
  65. INFO  jepsen.core - Run complete, writing
  66. INFO  jepsen.core - Analyzing
  67. (Log Truncated...)
  68. INFO  jepsen.core - Analysis complete
  69. INFO  jepsen.system.consul - :n3 consul nuked
  70. INFO  jepsen.system.consul - :n2 consul nuked
  71. INFO  jepsen.system.consul - :n4 consul nuked
  72. INFO  jepsen.system.consul - :n1 consul nuked
  73. INFO  jepsen.system.consul - :n5 consul nuked
  74. 1964 element history linearizable. :D
  76. Ran 1 tests containing 1 assertions.
  77. 0 failures, 0 errors.
  1. $ lein test :only jepsen.system.consul-test
  3. lein test jepsen.system.consul-test
  4. INFO  jepsen.os.debian - :n5 setting up debian
  5. INFO  jepsen.os.debian - :n3 setting up debian
  6. INFO  jepsen.os.debian - :n4 setting up debian
  7. INFO  jepsen.os.debian - :n1 setting up debian
  8. INFO  jepsen.os.debian - :n2 setting up debian
  9. INFO  jepsen.os.debian - :n4 debian set up
  10. INFO  jepsen.os.debian - :n5 debian set up
  11. INFO  jepsen.os.debian - :n3 debian set up
  12. INFO  jepsen.os.debian - :n1 debian set up
  13. INFO  jepsen.os.debian - :n2 debian set up
  14. INFO  jepsen.system.consul - :n1 consul nuked
  15. INFO  jepsen.system.consul - :n4 consul nuked
  16. INFO  jepsen.system.consul - :n5 consul nuked
  17. INFO  jepsen.system.consul - :n3 consul nuked
  18. INFO  jepsen.system.consul - :n2 consul nuked
  19. INFO  jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
  20. INFO  jepsen.system.consul - :n2 consul nuked
  21. INFO  jepsen.system.consul - :n3 consul nuked
  22. INFO  jepsen.system.consul - :n4 consul nuked
  23. INFO  jepsen.system.consul - :n5 consul nuked
  24. INFO  jepsen.system.consul - :n1 consul nuked
  25. INFO  jepsen.system.consul - :n1 starting consul
  26. INFO  jepsen.system.consul - :n2 starting consul
  27. INFO  jepsen.system.consul - :n4 starting consul
  28. INFO  jepsen.system.consul - :n5 starting consul
  29. INFO  jepsen.system.consul - :n3 starting consul
  30. INFO  jepsen.system.consul - :n3 consul ready
  31. INFO  jepsen.system.consul - :n2 consul ready
  32. INFO  jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
  33. INFO  jepsen.system.consul - :n5 consul ready
  34. INFO  jepsen.system.consul - :n1 consul ready
  35. INFO  jepsen.system.consul - :n4 consul ready
  36. INFO  jepsen.core - Worker 0 starting
  37. INFO  jepsen.core - Worker 2 starting
  38. INFO  jepsen.core - Worker 1 starting
  39. INFO  jepsen.core - Worker 3 starting
  40. INFO  jepsen.core - Worker 4 starting
  41. INFO  jepsen.util - 2   :invoke :read   nil
  42. INFO  jepsen.util - 3   :invoke :cas    [4 4]
  43. INFO  jepsen.util - 0   :invoke :write  4
  44. INFO  jepsen.util - 1   :invoke :write  1
  45. INFO  jepsen.util - 4   :invoke :cas    [4 0]
  46. INFO  jepsen.util - 2   :ok :read   nil
  47. INFO  jepsen.util - 4   :fail   :cas    [4 0]
  48. (Log Truncated...)
  49. INFO  jepsen.util - 4   :invoke :cas    [3 3]
  50. INFO  jepsen.util - 4   :fail   :cas    [3 3]
  51. INFO  jepsen.util - :nemesis    :info   :stop   nil
  52. INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
  53. INFO  jepsen.util - 0   :fail   :read   nil
  54. INFO  jepsen.util - 1   :fail   :write  0
  55. INFO  jepsen.util - :nemesis    :info   :stop   nil
  56. INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
  57. INFO  jepsen.core - nemesis done
  58. INFO  jepsen.core - Worker 3 done
  59. INFO  jepsen.util - 1   :invoke :read   nil
  60. INFO  jepsen.core - Worker 2 done
  61. INFO  jepsen.core - Worker 4 done
  62. INFO  jepsen.core - Worker 0 done
  63. INFO  jepsen.util - 1   :ok :read   3
  64. INFO  jepsen.core - Worker 1 done
  65. INFO  jepsen.core - Run complete, writing
  66. INFO  jepsen.core - Analyzing
  67. (Log Truncated...)
  68. INFO  jepsen.core - Analysis complete
  69. INFO  jepsen.system.consul - :n3 consul nuked
  70. INFO  jepsen.system.consul - :n2 consul nuked
  71. INFO  jepsen.system.consul - :n4 consul nuked
  72. INFO  jepsen.system.consul - :n1 consul nuked
  73. INFO  jepsen.system.consul - :n5 consul nuked
  74. 1964 element history linearizable. :D
  76. Ran 1 tests containing 1 assertions.
  77. 0 failures, 0 errors.