Testing with Pytest
At AHL, automated testing is a key tool to help us produce high quality code. For our Python based tests we rely heavily on Pytest. In this article we look at why we chose Pytest and how we use it.
Here at AHL we are dedicated to automated software tests. In the same spirit of automation that we apply to systematic trading, automated software tests verify that any changes to our code are rigorously checked for bugs and regressions of behaviour.
To help us achieve this, we are heavy users of the Pytest testing framework. Its simple function-based syntax and powerful feature set are attractive to new and experienced developers alike. A modular plugin system with its re-usable fixtures have allowed us to grow a very large library of tests with minimal duplication in setup code.
Here’s some of our favourite features of Pytest:
Tests Are Just Functions
Instead of constructing boilerplate classes, tests can be expressed as simple functions. Given this short function under test, here’s a comparison between the default UnitTest implementation and
## Contents of analytics.py import numpy def fastsum(array_like): return numpy.sum(array_like, axis=0)
import unittest import numpy import analytics class TestAnalytics(unittest.TestCase): def test_fastsum(self): self.assertEqual(analytics.fastsum(numpy.array([1, 2, 3]), 6)
import numpy import analytics def test_fastsum(): assert analytics.fastsum(numpy.array([1, 2, 3])) == 6
assert statement has been overloaded to provide in-depth comparison information about what you’re asserting.
$ pytest test_analytics.py ========================== test session starts =========================== platform linux2 -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 collected 1 items test_analytics.py F ================================ FAILURES ================================ ________________________________ test_sum ________________________________ def test_fastsum(): > assert analytics.fastsum([1, 2, 3]) == 7 E assert 6 == 7 E + where 6 = analytics.fastsum([1, 2, 3]) test_analytics.py:4: AssertionError ========================= 1 failed in 0.01 seconds ======================
Data-driven test cases
Often a function under test can have many different types of inputs. It is laborious to write out each permutation of inputs and outputs as separate tests, so
pytest has a great feature for creating data-driven parametrized test cases.
import pytest import numpy as np import analytics ([1, 2, 3], 6), # Integers ([1.1, 2.2, 3.3], 6.6), # Floats (np.array([1, 2, 3]), 6), # Numpy Arrays # ... etc ([1.0, np.nan, 3.0], np.nan), # Nan Handling ]) def test_fastsum(fn_in, expected_result): result = analytics.fastsum(fn_in) if np.isnan(expected_result): assert np.isnan(result) else: assert result == expected_result
Powerful Re-usable Fixtures
Simple, parametrized test functions and smart assertions are only the start of why we like
pytest. The real power comes from its dependency-injection-style test fixtures.
- A test fixture is an object that is created by the test framework with some initial state and passed into any tests that have requested it by specifying the fixture name as a function argument.
- Test fixtures have a scope, which determines their lifetime within the test run. The scope can be one of:
function: a single test function
UnitTeststyle test class
module: a single Python test module
session: the entire session, from when
pyteststarts up till it finishes running all the tests
- Fixtures have setup and teardown, that is run at the start and end of their lifetime.
Here’s a simple example that creates an in-memory SQLite database session and hands it to the test function:
import pytest from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker def db_session(): engine = create_engine('sqlite:///:memory:', echo=True) Session = sessionmaker(bind=engine) return Session() def test_select(db_session): """ the 'db_session' argument here is matched to the name of the fixture above """ db_session.execute('select name from user') ...
Fixtures Can Depend On Other Fixtures
In the same way that test functions depend on fixtures by specifying them by name as arguments, fixtures can depend on each other by specifying other fixture names in their own arguments. These dependencies will form a graph of objects that are created and torn down in the correct graph-order by test runner.
One can quickly see that this encourages libraries of re-usable test code that can be assembled to run complex integration tests with many moving parts. Here’s an example:
import pytest import backend import frontend def backend_server(): # --- Setup ------ server = backend.Server() server.start() yield server # --- Teardown --- server.stop() def frontend_client(backend_server): client = frontend.Client(host=server.host, port=server.port) client.connect() return client def test_client(frontend_client): frontend_client.login('admin', 'password') ...
- In this example we have a session-scoped server fixture that is only created once, and a frontend client that is created once per test function.
| Session Scope | Function Scope | |--------------------|--------------------------------------| | backend_server <--|-- frontend_client <----- test_client | | | |
- The frontend client depends on the server fixture. This both ensures that the server is running at the time the client is needed, and also means the client can pull configuration details like host and port numbers out of the server class.
- The test itself only asks for the client;
pytestwill ensure the server is started in time for the client to connect.
There is a gotcha here: fixture dependencies must obey scope precedence. You cannot have a session-scoped fixture depending on a function-scoped fixture, because the function-based fixture may have been destroyed within the lifetime of the session-based one.
Good Practice - Distribute Fixtures With The Code
As codebases grow, it makes sense for the teams that maintain a project to distribute the test fixtures for that project along with the normal code if its used by another team’s project. This way the fixtures remain ‘first class citizens’ and changes are propogated to the tests of other projects.
An example layout for a PnL service could be:
pnl_service/ __init__.py server.py client.py fixtures.py # this module contains a fixture called 'pnl_client'
Then in another project’s tests that use the PnL service, it can use the fixtures maintained by the PnL team.
import attribution # This tells pytest that we want to use the fixtures from # the other project pytest_plugins = 'pnl_service.fixtures' def test_attribution(pnl_client): pnl_data = pnl_client.get_pnl(product='TEST_PRODUCT') ftl_data = attribution.get_market_attribution('FTL') assert ftl_data == [ .... ]
Open Sourced: Here’s Some We Created Earlier
There are lots of plugins for Pytest that provide fixtures and other functionality, including many built-in ones.
Many of the fixtures we’ve written over the years are generally useful and as such have been open-sourced. They are all available in the GitHub repository pytest-plugins. Here’s the list of plugins:
||Extensible server-running framework with a suite of well-known databases and webservices included: mongodb, redis, rethinkd, Jenkins, Apache httpd, Xvfb|
||Unix shell and environment management tools|
||Profiling plugin with tabular heat graph output and gprof support for C-Extensions|
||DevPI server runnning fixture for testing package management code|
||Pyramid server fixture for running up servers in integration tests|
||Selenium webdriver fixture for testing web applications|
||Create and teardown virtual environments, run tests and commands in the scope of the virtualenv|
||PyQT application fixture|
||TCP Listener/Reciever for testing remote systems|
||Git repository fixture|
||SVN repository fixture|
||Configuration tools for Py.test fixtures|
||Makes py.test’s parametrize output a little more verbose|
How to get them
All of these plugins are available on PyPI and can be installed individually using pip:
$ pip install pytest-server-fixtures
If you love testing your code as much we do, go ahead and install Pytest and get testing! It’s easy to get started and satisfying to know your software is in good hands when the tests all pass. Here’s some references for further reading:
- Pytest Official Documentation
- Pytest Plugins on PyPI
- Pytest Development Blog
- Using Pytest with Jenkins
Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. This material is proprietary information of the Company and its affiliates and may not be reproduced or otherwise disseminated in whole or in part without prior written consent from the Company. The Company believes the content to be accurate. However accuracy is not warranted or guaranteed. The Company does not assume any liability in the case of incorrectly reported or incomplete information. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.