Lunch Time Python

Lunch 4: pytest

pytest is a widely used Python test framework, which makes it easy to write small and readable tests, and also offers more advanced features such as fixtures and mocks. There is also a large ecosystem of plugins providing additional functionality.

Press Spacebar to go to the next slide (or ? to see all navigation shortcuts)

Lunch Time Python, Scientific Software Center, Heidelberg University

Why write tests?

• ensure correctness
• maintain correctness
• find bugs earlier and more easily
• allow refactoring without fear
• allow others to contribute without unknowingly breaking stuff
• can complement documentation as examples of use
• give others confidence in your code

pytest installation

• Conda: conda install pytest
• Pip: python -m pip install pytest

First steps

Create a test

1. create a file that starts with test_, e.g. test_math.py
2. add a function to it that starts with test_ and asserts things, e.g.

# in file: test_math.py
def test_add():
        assert 1 + 1 == 2

Run the tests

3. run pytest -v or python -m pytest -v

================ test session starts =======================
platform linux -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0
rootdir: /home/liam/test
plugins: anyio-3.5.0
collected 1 item                                                                                                                                                                                  

test_math.py::test_add PASSED  
                                                       [100%]

====================== 1 passed in 0.00s ====================

What just happened?

• pytest looks for all files that start with test_
• it collects all functions in these files that start with test_
• it runs them all, and reports PASS/FAIL for each assertion in each function

Some things we didn't do

• import a test library
• inherit from some base test class
• use some special assertEqual function
• register our test file or test cases

Simple pytest test strategy

• for each file abc.py, add a test_abc.py
• for each function foo() in abc.py, add a test_foo() to test_abc.py
• inside test_foo() assert things involving foo() that should be true
• that's it

ipytest

disclaimer

• For demonstration purposes, these tests will be written inside this jupyter notebook
• We'll use a helper library ipytest to call pytest on them from inside the notebook
• But generally I would recommend putting functions and tests into files and running pytest directly

if "google.colab" in str(get_ipython()):
    !pip install ipytest -qqq
import ipytest
import pytest

ipytest.autoconfig()

%%ipytest -vv


def test_math():
    assert 1 + 1 == 2

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 1 item

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_math PASSED                                      [100%]

======================================== 1 passed in 0.01s =========================================

Failing tests

%%ipytest -q


def f(x):
    return 2 * x


def g(x):
    return f(x) + 3


def test_math():
    a = 2
    b = 3
    assert f(a) * g(b) == 36

.                                                                                            [100%]

%%ipytest -q


def test_list():
    a = [1, 2, 5, 8]
    b = [1, 2, 5, 8]
    assert a == b

.                                                                                            [100%]

Exceptions

%%ipytest -q


def test_exception():
    my_list = [1, 2, 3]
    with pytest.raises(IndexError):
        my_list[5]

.                                                                                            [100%]

%%ipytest -q


def test_exception():
    my_list = [1, 2, 3]
    with pytest.raises(Exception) as e:
        my_list[5]
    assert e.type == IndexError
    assert "out of range" in str(e.value)

.                                                                                            [100%]

Temporary files

• often need to write to a temporary file in a test
• simply add tmp_path as an argument to your test function
• pytest will provide a unique temporary path object for each test
• this is an example of a fixture

%%ipytest -qs


def test_write(tmp_path):
    print(tmp_path)
    assert str(tmp_path) != ""

/tmp/pytest-of-runner/pytest-0/test_write0
.

Monkey-patching

• a fixture to temporarily modify an object, dict or environment variable
• all modifications are undone after the test is finished
• add monkeypatch as an argument to your test function
• provides various methods, e.g.
  • monkeypatch.setattr(obj, name, value)
  • monkeypatch.setenv(name, value)
  • monkeypatch.syspath_prepend(path)
  • monkeypatch.chdir(path)

%%ipytest -qs

import os


def test_env(monkeypatch):
    assert os.getenv("TEST_API_KEY") == None
    monkeypatch.setenv("TEST_API_KEY", "abc123")
    assert os.getenv("TEST_API_KEY") == "abc123"

.

Fixtures

• a way to provide context (e.g. data or environment) to a test
• test "requests" a fixture by declaring it as an argument
• various built-in fixtures (tmp_path, monkeypatch, ...)
• you can create your own with the @pytest.fixture decorator
• for each test function argument, pytest looks for a fixture with the same name
• fixtures can themselves request other fixtures

%%ipytest -vv

# a fixture to provide some data to a test
@pytest.fixture
def colours():
    return ["red", "green", "blue"]


def test_colours(colours):
    assert colours[0] == "red"

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 1 item

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colours PASSED                                   [100%]

======================================== 1 passed in 0.01s =========================================

%%ipytest -vv


@pytest.fixture
def colours():
    return ["red", "green", "blue"]


# a fixture that itself requests another fixture
@pytest.fixture
def sorted_colours(colours):
    return sorted(colours)


def test_colours(sorted_colours):
    assert sorted_colours[0] == "blue"

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 1 item

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colours PASSED                                   [100%]

======================================== 1 passed in 0.01s =========================================

%%ipytest -vv

# a fixture that uses monkeypatch to set an environment variable
@pytest.fixture
def api_key(monkeypatch):
    monkeypatch.setenv("TEST_API_KEY", "abc123")


def test_missing_api_key():
    assert os.getenv("TEST_API_KEY") == None


def test_api_key(api_key):
    assert os.getenv("TEST_API_KEY") == "abc123"

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 2 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_missing_api_key PASSED                           [ 50%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_api_key PASSED                                   [100%]

======================================== 2 passed in 0.01s =========================================

%%ipytest -vv

# a parameterized fixture: test will be repeated for each parameter
@pytest.fixture(params=["red", "green", "blue", "yellow"])
def colour(request):
    return request.param


def test_colour(colour):
    assert len(colour) >= 3

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 4 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colour[red] PASSED                               [ 25%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colour[green] PASSED                             [ 50%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colour[blue] PASSED                              [ 75%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_colour[yellow] PASSED                            [100%]

======================================== 4 passed in 0.02s =========================================

Test grouping

• put functions into a class whose name begins with Test
• a class can request a fixture, all member functions then have this fixture

%%ipytest -vv


class TestMath:
    def test_add(self):
        assert 1 + 1 == 2

    def test_mul(self):
        assert 2 * 2 == 4

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 2 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::TestMath::test_add PASSED                             [ 50%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::TestMath::test_mul PASSED                             [100%]

======================================== 2 passed in 0.01s =========================================

Marking tests

• mark tests with attributes using @pytest.mark decorator
• common use cases
  • skipif to conditionally skip a test
    • e.g. depending on python version or platform
  • xfail to mark a test that is expected to fail
    • e.g. a test that documents a known bug that is not yet fixed
• can also mark a test class to mark all tests within that class
• can also create your own custom markers

%%ipytest -vv

import sys


@pytest.mark.xfail(reason="bug from issue #123")
def test_add():
    assert 1 + 1 == 3


@pytest.mark.skipif(not sys.platform.startswith("win"), reason="windows only test")
def test_mul():
    assert 2 * 2 == 4

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 2 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_add XFAIL (bug from issue #123)                  [ 50%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_mul SKIPPED (windows only test)                  [100%]

================================== 1 skipped, 1 xfailed in 0.10s ===================================

Parameterizing tests

• can parameterize tests using the @pytest.mark.parameterize decorator
• takes comma-delimeted list of arguments as a string
• followed by a list of tuples of argument values

%%ipytest -vv


@pytest.mark.parametrize("n", [1, 2, 3])
def test_n(n):
    assert n > 0

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 3 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[1] PASSED                                      [ 33%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[2] PASSED                                      [ 66%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[3] PASSED                                      [100%]

======================================== 3 passed in 0.02s =========================================

%%ipytest -vv


@pytest.mark.parametrize("n,n_squared", [(1, 1), (2, 4), (3, 9)])
def test_n(n, n_squared):
    assert n * n == n_squared

======================================= test session starts ========================================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.11/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/lunch-time-python/lunch-time-python
configfile: pytest.ini
plugins: anyio-3.6.2, dash-2.9.3
collecting ... collected 3 items

t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[1-1] PASSED                                    [ 33%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[2-4] PASSED                                    [ 66%]
t_5866e5d8aeb846388fdd9176c5e9e105.py::test_n[3-9] PASSED                                    [100%]

======================================== 3 passed in 0.02s =========================================

Summary

• pytest is very easy to get started with and use
• just writing test functions with assertions already provides a lot of value
• fixtures allow you to provide context to your test functions
• parameterizing fixtures and/or tests can turn a single test into many test cases
• (many) more features at docs.pytest.org