Workflow for writing a Python test

Normally we write functional tests in Python.

  • We use pytest for our Python test infrastructure.
  • Python tests should be added to tests/python/test_xxx.py.

For example, you’ve just added a utility function ti.log10. Now you want to write a test, to test if it functions properly.

Adding a new test case

Look into tests/python, see if there’s already a file suit for your test. If not, feel free to create a new file for it :) So in this case let’s create a new file tests/python/test_logarithm.py for simplicity.

Add a function, the function name must be started with test_ so that pytest could find it. e.g:

  1. import taichi as ti
  3. def test_log10():
  4.     pass

Add some simple code make use of our ti.log10 to make sure it works well. Hint: You may pass/return values to/from Taichi-scope using 0-D tensors, i.e. r[None].

  1. import taichi as ti
  3. def test_log10():
  4.     ti.init(arch=ti.cpu)
  6.     r = ti.var(ti.f32, ())
  8.     @ti.kernel
  9.     def foo():
  10.         r[None] = ti.log10(r[None])
  12.     r[None] = 100
  13.     foo()
  14.     assert r[None] == 2

Execute ti test logarithm, and the functions starting with test_ in tests/python/test_logarithm.py will be executed.

Testing against multiple backends

The above method is not good enough, for example, ti.init(arch=ti.cpu), means that it will only test on the CPU backend. So do we have to write many tests test_log10_cpu, test_log10_cuda, … with only the first line different? No worries, we provide a useful decorator @ti.test:

  1. import taichi as ti
  3. # will test against both CPU and CUDA backends
  4. @ti.test(ti.cpu, ti.cuda)
  5. def test_log10():
  6.     r = ti.var(ti.f32, ())
  8.     @ti.kernel
  9.     def foo():
  10.         r[None] = ti.log10(r[None])
  12.     r[None] = 100
  13.     foo()
  14.     assert r[None] == 2

And you may test against all backends by simply not specifying the argument:

  1. import taichi as ti
  3. # will test against all backends available on your end
  4. @ti.test()
  5. def test_log10():
  6.     r = ti.var(ti.f32, ())
  8.     @ti.kernel
  9.     def foo():
  10.         r[None] = ti.log10(r[None])
  12.     r[None] = 100
  13.     foo()
  14.     assert r[None] == 2

Cool! Right? But that’s still not good enough.

Using ti.approx for comparison with tolerance

Sometimes the math percison could be poor on some backends like OpenGL, e.g. ti.log10(100) may return 2.001 or 1.999 in this case.

To cope with this behavior, we provide ti.approx which can tolerate such errors on different backends, for example 2.001 == ti.approx(2) will return True on the OpenGL backend.

  1. import taichi as ti
  3. # will test against all backends available on your end
  4. @ti.test()
  5. def test_log10():
  6.     r = ti.var(ti.f32, ())
  8.     @ti.kernel
  9.     def foo():
  10.         r[None] = ti.log10(r[None])
  12.     r[None] = 100
  13.     foo()
  14.     assert r[None] == ti.approx(2)

警告

Simply using pytest.approx won’t work well here, since it’s tolerance won’t vary among different Taichi backends. It’ll be likely to fail on the OpenGL backend.

ti.approx also do treatments on boolean types, e.g.: 2 == ti.approx(True).

Great on improving stability! But the test is still not good enough, yet.

Parametrize test inputs

For example, r[None] = 100, means that it will only test the case of ti.log10(100). What if ti.log10(10)? ti.log10(1)?

We may test against different input values using the @pytest.mark.parametrize decorator:

  1. import taichi as ti
  2. import pytest
  3. import math
  5. @pytest.mark.parametrize('x', [1, 10, 100])
  6. @ti.test()
  7. def test_log10(x):
  8.     r = ti.var(ti.f32, ())
  10.     @ti.kernel
  11.     def foo():
  12.         r[None] = ti.log10(r[None])
  14.     r[None] = x
  15.     foo()
  16.     assert r[None] == math.log10(x)

Use a comma-separated list for multiple input values:

  1. import taichi as ti
  2. import pytest
  3. import math
  5. @pytest.mark.parametrize('x,y', [(1, 2), (1, 3), (2, 1)])
  6. @ti.test()
  7. def test_atan2(x, y):
  8.     r = ti.var(ti.f32, ())
  9.     s = ti.var(ti.f32, ())
  11.     @ti.kernel
  12.     def foo():
  13.         r[None] = ti.atan2(r[None])
  15.     r[None] = x
  16.     s[None] = y
  17.     foo()
  18.     assert r[None] == math.atan2(x, y)

Use two separate parametrize to test all combinations of input arguments:

  1. import taichi as ti
  2. import pytest
  3. import math
  5. @pytest.mark.parametrize('x', [1, 2])
  6. @pytest.mark.parametrize('y', [1, 2])
  7. # same as:  .parametrize('x,y', [(1, 1), (1, 2), (2, 1), (2, 2)])
  8. @ti.test()
  9. def test_atan2(x, y):
  10.     r = ti.var(ti.f32, ())
  11.     s = ti.var(ti.f32, ())
  13.     @ti.kernel
  14.     def foo():
  15.         r[None] = ti.atan2(r[None])
  17.     r[None] = x
  18.     s[None] = y
  19.     foo()
  20.     assert r[None] == math.atan2(x, y)

Specifying ti.init configurations

You may specify keyword arguments to ti.init() in ti.test(), e.g.:

  1. @ti.test(ti.cpu, debug=True, log_level=ti.TRACE)
  2. def test_debugging_utils():
  3.     # ... (some tests have to be done in debug mode)

is the same as:

  1. def test_debugging_utils():
  2.     ti.init(arch=ti.cpu, debug=True, log_level=ti.TRACE)
  3.     # ... (some tests have to be done in debug mode)

Exclude some backends from test

Sometimes some backends are not capable of specific tests, we have to exclude them from test:

  1. # Run this test on all backends except for OpenGL
  2. @ti.test(excludes=[ti.opengl])
  3. def test_sparse_tensor():
  4.     # ... (some tests that requires sparse feature which is not supported by OpenGL)

You may also use the extensions keyword to exclude backends without specific feature:

  1. # Run this test on all backends except for OpenGL
  2. @ti.test(extensions=[ti.extension.sparse])
  3. def test_sparse_tensor():
  4.     # ... (some tests that requires sparse feature which is not supported by OpenGL)