Perhaps inspired by Brian Okken’s pytest book, I have been experimenting with a new approach to writing code. Most of my work consists of a long list of one-off scripts which serve a single purpose: moving data around, performing some relatively simple NLP operation, etc. While they will likely be run a few times (e.g., adding additional data, fixing bugs, etc.), they are unlikely to serve much use beyond the scope of the current project, whence they will be unceremoniously interred, their ashes scattered amongst the winds, and, should ever our paths meet again in this world, I will balk with disgust and wonder what poor neophyte programmer wrote that crap.
As might be readily inferred, this code is barely legible, untested (except in the rather subjective ‘this output makes sense and looks right’), and bears documentation mostly in the form of ‘todos’ which will never be addressed. From this sorry state, I’ve been attempting to implement better practices with the goal of improved code re-use (both for myself and collaborators), as well as increasing the confidence (and reduce debugging) of even single-use code (which, as far as I’m concerned, is never actually single-use).
One of the changes I’ve made in this approach has been to code from a perspective of testing. Instead of solely designing functions with the viewpoint reducing layers, I’ve included a new heuristic: will adding this function make the script more testable?
On this journey, I started incorporating text much earlier into the testing process, and quite naturally found the use of pytest’s fixture. This have mostly felt rather foreign to me, but as I was leaning close to something akin to test-driven development, I began formatting my tests more narrowly. Before, my test was typically part of a conclusion, or of adding some new case, so I tested everything at once:
@pytest.mark.parametrize(('text', 'exp'), [
('test this text', {'words': 3, 'characters': 12, ...: ...}),
...
])
def test_counting(text, exp):
assert count_stuff(text) == exp
This approach is less helpful when attempting to test count_stuff while actively writing the function. This won’t work when I’ve trying get count_stuff to focus on (e.g.) just counting the number of words. Rather, every line will fail: ‘characters’ will ever be not present or incorrect.
No, instead, I want my pytest to be happy and display that happy green line when I’ve succeeded. Let’s change that.
@pytest.mark.parametrize(('text', 'exp'), [
('test this text', 3),
...
])
def test_counting_words(text, exp):
d = count_stuff(text)
assert d['words'] == exp
Yes! Success. But as I expand this process, I run into a slight issue:
@pytest.mark.parametrize(('text', 'exp'), [
('test this text', 12),
...
])
def test_counting_characters(text, exp):
d = count_stuff(text)
assert d['characters'] == exp
For each of these, I will likely want to use at least some of the same underlying set of text (although not wanting them exactly identical if, e.g., certain characters are proving troublesome for characters, I may want to just add that to the characters parametrization. (Also, bear in mind that the text I’m working with tends to be quite a bit longer…)
Two solutions come to mind:
- Place these functions into a test class as some field. (This, however, seems more prone to confusion…)
- Use fixtures. (See below.)
There are probably better solutions (send me an email if you have one, and I’ll give it a try).
Using fixtures:
# FAILS: This doesn't work, but is the gist
@pytest.fixture
def text1():
return 'test this text'
@pytest.mark.parametrize(('text', 'exp'), [
(text1, 3), # can't use fixtures in this setting
('test another\ntext', 3),
...
])
def test_counting_words(text, exp):
d = count_stuff(text)
assert d['words'] == exp
@pytest.mark.parametrize(('text', 'exp'), [
(text1, 12), # can't use fixtures in this setting
('test another\ntext', 15),
...
])
def test_counting_characters(text, exp):
d = count_stuff(text)
assert d['characters'] == exp
The above doesn’t quite work (pytest doesn’t evaluate text1), though I like the idea, and it feels pytest-y — not that I can claim to know what pytest-y would actually mean.
Kindly, [the pytest-lazy-fixture plugin](https://pypi.org/project/pytest-lazy-fixture/) exists to help us out. pip install pytest-lazy-fixture. Now, we can add pytest.lazy_fixture around the text1 fixture:
@pytest.fixture
def text1():
return 'test this text'
@pytest.mark.parametrize(('text', 'exp'), [
(pytest.lazy_fixture('text1'), 3), # using lazy_fixture
('test another\ntext', 3),
...
])
def test_counting_words(text, exp):
d = count_stuff(text)
assert d['words'] == exp
@pytest.mark.parametrize(('text', 'exp'), [
(pytest.lazy_fixture('text1'), 12), # using lazy_fixture
('test another\ntext', 15),
...
])
def test_counting_characters(text, exp):
d = count_stuff(text)
assert d['characters'] == exp
And now we get pytest’s green light to tackle the next item.