kr41

Dmitry Vakhrushev about programming and related things

Tag “Testing”

I released GreenRocket library on October 2012. It is a dead simple implementation of Observer design pattern, which I use in almost all of my projects. I thought, there was nothing to improve. But my recent project heavily uses the library. And I get tired to write tests that checks signals. This is how them look like:

from nose import tools
# I use Nose for testing my code

from myproject import MySignal, some_func
# ``MySignal`` inherits ``greenrocket.Signal``
# ``some_func`` must fire ``MySignal`` as its side-effect


def test_some_func():
    log = []                    # Create log for fired signals

    @MySignal.subscribe         # Subscribe a dummy handler
    def handler(signal):
        log.append(signal)      # Put fired signal to the log

    some_func()                 # Call the function to test

    # Test fited signal from the log
    tools.eq_(len(log), 1)
    tools.eq_(log[0].x, 1)
    tools.eq_(log[0].y, 2)
    # ...and so on

There are four lines of utility code. And it is boring. So I added helper class Watchman to the library to make it testing friendly. This is how it works:

from greenrocket import Watchman

from myproject import MySignal, some_func


def test_some_code_that_fires_signal():
    watchman = Watchman(MySignal)           # Create a watchman for MySignal
    some_func()
    watchman.assert_fired_with(x=1, y=2)    # Test fired signal

Just one line of utility code and one line for actual test! I have already rewritten all of my tests. So if you are using the library, it’s time to upgrade. If you don’t, then try it out.

When you develop a library, which should work with a number of Python versions, Tox is obvious choice. However, I start using it even in application developing, where single Python version is used. Because it helps me significantly reduce efforts of documentation writing. How? It transparently manages virtual environments.

For instance, you work on backend, and your colleague works on frontend. This guy is CSS ninja, but knows nothing about Python. So you have to explain him or her how to start the application in development mode. You can do this in two ways.

The first one is to write an instruction. The instruction should explain how to setup virtual environment, activate it, setup application in development mode, and run it. In 9 cases of 10 it blows non-pythonista’s mind away. Moreover, writing docs is tedious. Who likes writing docs, when you can write a script?!

And this is the second way. You can write a script, which creates virtual environment, activates it, installs application, and runs it. But this is exactly what Tox does.

Here is how I do it. The following tox.ini file is from the project I am working on now. It is a Pyramid application, which I test against Python 3.3 and 3.4, but develop using Python 3.4.

[tox]
envlist=py33,py34

[testenv]
deps=
    -r{toxinidir}/tests/requires.txt
    flake8
commands=
    nosetests
    flake8 application
    flake8 tests

[testenv:dev]
envdir=devenv
basepython=python3.4
usedevelop=True
deps=
    -r{toxinidir}/tests/requires.txt
    waitress
commands={toxinidir}/scripts/devenv.sh {posargs}

Take a notice on the section [testenv:dev]. It launches devenv.sh script passing command line arguments, which are not processed by Tox itself. Here is the script:

#!/bin/bash

test() {
    nosetests "$@"
}

serve() {
    pserve --reload "configs/development.ini"
}

cmd="$1"
shift

if [[ -n "$cmd" ]]
then
    $cmd "$@"
fi

And here is an example of the manual:

  1. Install Tox.

  2. To run application use:

    $ tox -e dev serve

  3. To run all tests using development environment:

    $ tox -e dev test

  1. To run single test using development environment:

    $ tox -e dev test path/to/test.py

  2. To run complete test suite and code linting:

    $ tox

That’s it. Pretty simple. I copy it from project to project and my teammates are happy. You can even eliminate the first item from the list above, using Vagrant and installing Tox on provision stage. But there is a bug in Distutils, which breaks Tox within Vagrant. Use this hack to make it work.

It is always easy and fun to do something, if you have right tools. Writing tests is not exception. Here is my toolbox, all things at one place. I hope, the following text will save somebody’s time and Google’s bandwidth.

Here we go.

Flake8

It is a meta tool, which tests code using PyFlakes and pep8. The first one is a static analyzer and the second one is a code style checker. They can be used separately, but I prefer they work as a team. It helps to find stupid errors such as unused variables or imports, typos in names, undefined variables, and so on. It also helps to keep code consistent according to PEP 8 Style Guide for Python Code, which is critical for code-style nazis like me. The usage is quite simple:

$ flake8 coolproject
coolproject/module.py:97:1: F401 'shutil' imported but unused
coolproject/module.py:625:17: E225 missing whitespace around operato
coolproject/module.py:729:1: F811 redefinition of function 'readlines' from line 723
coolproject/module.py:1028:1: F841 local variable 'errors' is assigned to but never used

Additionally, Flake8 includes complexity checker, but I never use it. However, it can help to decrease WTFs per minute during code review, I guess.

Nose

It is a unit-test framework, an extension of traditional unittest. I never use the last one itself, so I cannot adequately compare it with Nose. However, at a glance, Nose-based tests is more readable and compact. But it is only my subjective opinion.

Another benefit of Nose is its plugins. Some of them I use from time to time, but there are two, which I use unconditionally on each my project: doctest and cover.

The doctest plugin collects test scenarios from source code doc-strings and run them using Doctest library. It helps to keep doc-strings consistent with the code they describe. It also good place for unit tests for simple functions and classes. If test cases are not too complex, it will be enough to cover the code directly in the doc-string.

The cover plugin calculates test coverage and generates reports like this one:

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
coolproject                  20      4    80%   33-35, 39
coolproject.module           56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%

Such reports help to check test cases themselves and significantly improve the quality of ones. The cover plugin uses Coverage tool behind the scene, so you have to manually install it.

Nose is perfectly integrated with Setuptools. By the way, that is another reason to use the last one. I prefer to store Nose settings in the setup.cfg file, which usually looks like this:

[nosetests]
verbosity=2
with-doctest=1
with-coverage=1
cover-package=coolproject
cover-erase=1

It makes Nose usage very simple:

$ nosetests
tests.test1 ... ok
tests.test2 ... ok
tests.test3 ... ok
Doctest: coolproject.function1 ... ok
Doctest: coolproject.module.function2 ... ok

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
coolproject                  20      4    80%   33-35, 39
coolproject.module           56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%
Ran 5 tests in 0.021s

OK

Mocks

There is no way to write tests without mocks. At the most cases, Mock library is all what you need. However, there are other useful libraries, that can be helpful in particular cases.

  • Venusian can help to mock decorated functions deferring decorator action on the separate step.
  • FreezeGun is a neat mock for date and time. There is nothing you cannot do using Mock library, but it has already been done. So, just use it.
  • Responses is a mock for Requests library. If you develop client for third-party REST-service using Requests, that is what you need.

Additionally, I strongly recommend to look over the perfect article Python Mock Gotchas by Alex Marandon.

Tox

Tox gets things together and makes them run against different Python versions. It is like a command center for all testing infrastructure. It automatically creates virtual environments for specified Python versions, installs test dependencies and runs tests. And all of these is done using single command tox.

For example, tox.ini described bellow sets up testing for Python 3.3, 3.4, and PyPy, using Nose for unit tests and Flake8 for static analysis of source code of project itself, as well as source code of unit tests.

[tox]
envlist=py33,py34,pypy

[testenv]
deps=
    nose
    coverage
    flake8
commands=
    nosetests
    flake8 coolproject
    flake8 tests

The usage of this tool is not limited by tests only. But it deserves a separate article, so I will write it soon.

Conclusion

I am pretty sure the list above is not complete. And there is a lot of awesome testing libraries that make life easier. So, post your links in the comments. I will try to keep the article updated.