Parameterization from a file in py.test

You can encounter various tools in automated testing. For writing automated tests in Python, one of the most popular solutions at the moment is py.test.

After combing through many pytest-related resources and studying the documentation on the project’s official website, I could not find a direct description of a solution to one of the basic tasks: running tests with test data stored in a separate file. In other words, loading parameters into test functions from a file or files, or parameterizing directly from a file. The details of this procedure are not described anywhere, and the only mention of this possibility is a single line in the pytest documentation.

In this article, I will describe my solution to this task.

---

The task

The main task is to generate test cases as test_input and expected_result parameters for each individual test function, from files whose names correspond to those functions.

Additional tasks:

• choose a human-readable format for files containing test cases;

• retain support for hardcoded test cases;

• display understandable identifiers for each case.

Tools

In this article, I use Python 3 (2.7 will also work), pyyaml, and pytest (version 5+ for Python 3, or 4.6 for Python 2.7), without third-party plugins. I will also use the standard os library.

The file from which we take test cases needs to be structured using a markup language that is easy for humans to understand. In my case, I chose YAML, since it solves the additional task of choosing a human-readable format. In practice, the markup language you need for your dataset files depends entirely on your project’s requirements.

---

Implementation

Since convention is the fundamental pillar of the programming universe, we will have to introduce a few conventions for our solution too.

The hook

To begin with, this solution uses the pytest_generate_tests hook function (wiki), which runs at the test-case generation stage, and its metafunc argument, which allows us to parameterize a function. At this point, pytest goes through each test function and executes the following generation code for it.

Arguments

We need to define an exhaustive list of parameters for the test functions. In my case, a dictionary for test_input and any data type—most often a string or an integer—for expected_result. We need these parameters for use in metafunc.parametrize(...).

Parameterization

This function fully reproduces the behavior of the parameterization fixture @pytest.mark.parametrize, which takes as its first argument a string listing the test function’s arguments—in our case, "test_input, expected_result"—and a list of data it iterates over to create our test cases, for example [(1, 2), (2, 4), (3, 6)].

In practice, it looks like this:

@pytest.mark.parametrize("test_input, expected_result", [(1, 2), (2, 4), (3, 6)])
def test_multiplication(test_input, expected_result):
    assert test_input * 2 == expected_result

In our case, we will specify this in advance:

# Наша реализация...
return metafunc.parametrize("test_input, expected", test_cases) # или `[(1, 2), (2, 4), (3, 6)]`

Filtering

This also means distinguishing test functions that need data loaded from a file from those using static or dynamic data. We will apply this filtering before parsing information from the file.

The filters themselves can be anything, for example:

• A function marker named yaml:

# Откидываем варианты вообще без каких-либо маркеров
if not hasattr(metafunc.function, 'pytestmark'):
    return

# Берем все маркеры нынешней функции и их имена вносим в список
mark_names = [ mark.name for mark in metafunc.function.pytestmark ]
# Пропускаем эту функцию, если в списке нет выбранного нами маркера
if 'yaml' not in mark_names:
    return

Alternatively, the same filter can be implemented like this:

# Создаем пустой маркер и ищем такой же в маркерах функции
if Mark(name=’yaml’, args=(), kwargs={}) not in metafunc.function.pytestmark:
    return

• The function argument test_input:

# Пропускаем все функции, у которых нет аргумента test_input
if 'test_input' not in metafunc.fixturenames:
        return

This option suited me best.

---

The result

We only need to finish the part where we parse data from the file. This is straightforward with yaml, as well as json, xml, and so on, so let’s put everything together.

# conftest.py
import os
import yaml
import pytest

def pytest_generate_tests(metafunc):

    # Пропускаем все функции, у которых нет аргумента test_input
    if 'test_input' not in metafunc.fixturenames:
        return

    # Определяем директорию текущего файла
    dir_path = os.path.dirname(os.path.abspath(metafunc.module.__file__))

    # Определяем путь к файлу с данными
    file_path = os.path.join(dir_path, metafunc.function.__name__ + '.yaml')
    # Открываем выбранный файл
    with open(file_path) as f:
        test_cases = yaml.full_load(f)

    # Предусматриваем неправильную загрузку и пустой файл
    if not test_cases:
        raise ValueError("Test cases not loaded")

    return metafunc.parametrize("test_input, expected_result", test_cases)

We write a test script roughly like this:

# test_script.py
import pytest

def test_multiplication(test_input, expected_result):
    assert test_input * 2 == expected_result

And the data file:

# test_multiplication.yaml
  - !!python/tuple [1,2]
  - !!python/tuple [1,3]
  - !!python/tuple [1,5]
  - !!python/tuple [2,4]
  - !!python/tuple [3,4]
  - !!python/tuple [5,4]

We get this list of test cases:

 pytest /test_script.py --collect-only
======================== test session starts ========================
platform linux -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: /pytest_habr
collected 6 items                                                                                                                                                           
<Module test_script.py>
  <Function test_multiplication[1-2]>
  <Function test_multiplication[1-3]>
  <Function test_multiplication[1-5]>
  <Function test_multiplication[2-4]>
  <Function test_multiplication[3-4]>
  <Function test_multiplication[5-4]>

======================== no tests ran in 0.04s ========================

Running the script gives this result: 4 failed, 2 passed, 1 warnings in 0.11s

---

Additional tasks

We could end the article here, but to make things a little more complex, I will add more convenient identifiers, different data parsing, and marking of each individual test case to our function.

So, straight to the code:

# conftest.py
import os
import yaml
import pytest

def pytest_generate_tests(metafunc):

    def generate_id(input_data, level):
        level += 1

        # Выбираем как это будет выглядеть
        INDENTS = {
            # level: (levelmark, addition_indent)
            1: ('_', ['', '']),
            2: ('-', ['[', ']'])
        }
        COMMON_INDENT = ('-', ['[', ']'])

        levelmark, additional_indent = INDENTS.get(level, COMMON_INDENT)

        # Если глубже второго уровня - идентификатором становится тип данных
        if level > 3:
            return additional_indent[0] + type(input_data).__name__ + additional_indent[1]

        # Возвращаем простые данные
        elif isinstance(input_data, (str, bool, float, int)):
            return str(input_data)

        # Разбираем список
        elif isinstance(input_data, (list, set, tuple)):
            # Погружаемся в список, чтобы проверить те данные, что внутри
            list_repr = levelmark.join(
                [ generate_id(input_value, level=level) \
                                        for input_value in input_data ])
            return additional_indent[0] + list_repr + additional_indent[1]

        # Ключи словаря переводим в строку
        elif isinstance(input_data, dict):
            return '{' + levelmark.join(input_data.keys()) + '}'

        # Или ничего для ничего
        else:
            return None

    # Пропускаем все функции, у которых нет аргумента test_input
    if 'test_input' not in metafunc.fixturenames:
        return

    # Определяем директорию текущего файла
    dir_path = os.path.dirname(os.path.abspath(metafunc.module.__file__))

    # Определяем путь к файлу с данными
    file_path = os.path.join(dir_path, metafunc.function.__name__ + '.yaml')
    # Открываем выбранный файл
    with open(file_path) as f:
        raw_test_cases = yaml.full_load(f)

    # Предусматриваем неправильную загрузку и пустой файл
    if not raw_test_cases:
        raise ValueError("Test cases not loaded")

    # Тут будут наши тест-кейсы
    test_cases = []

    # Проходим по нашим сырым данным
    for case_id, test_case in enumerate(raw_test_cases):
        # Ищем список маркеров
        marks = [ getattr(pytest.mark, name) for name in test_case.get("marks", []) ]

        # Берем идентификатор из данных, либо сгенерируем
        case_id = test_case.get("id", generate_id(test_case["test_data"], level=0))

        # Добавляем в наш список сгенерированный из тестовых данных pytest.param
        test_cases.append(pytest.param(*test_case["test_data"], marks=marks, id=case_id))

    return metafunc.parametrize("test_input, expected_result", test_cases)

Accordingly, we change what our YAML file looks like:

# test_multiplication.yaml
-
  test_data: [1, 2]
  id: 'one_two'
-
  test_data: [1,3]
  marks: ['xfail']
-
  test_data: [1,5]
  marks: ['skip']
-
  test_data: [2,4]
  id: "it's good"
  marks: ['xfail']
-
  test_data: [3,4]
  marks: ['negative']
-
  test_data: [5,4]
  marks: ['more_than']

Then the description changes to:

<Module test_script.py>
  <Function test_multiplication[one_two]>
  <Function test_multiplication[1_3]>
  <Function test_multiplication[1_5]>
  <Function test_multiplication[it's good]>
  <Function test_multiplication[3_4]>
  <Function test_multiplication[5_4]>

And running it gives: 2 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 2 warnings in 0.12s

P.S.: warnings occur because the custom markers are not registered in pytest.ini

Taking the topic further

I am ready to discuss questions like these in the comments:

• What is the best way to write the YAML file?

• What format is most convenient for storing test data?

• What else does a test case need at the generation stage?

• Does every case need an identifier?