Testing with Pytest

At AHL, automated testing is a key tool to help us produce high quality code. For our Python based tests we rely heavily on Pytest. In this article we look at why we chose Pytest and how we use it.

26 JANUARY 2018

Why Pytest?

Here at AHL we are dedicated to automated software tests. In the same spirit of automation that we apply to systematic trading, automated software tests verify that any changes to our code are rigorously checked for bugs and regressions of behaviour.

To help us achieve this, we are heavy users of the Pytest testing framework. Its simple function-based syntax and powerful feature set are attractive to new and experienced developers alike. A modular plugin system with its re-usable fixtures have allowed us to grow a very large library of tests with minimal duplication in setup code.

Here’s some of our favourite features of Pytest:

Tests Are Just Functions

Instead of constructing boilerplate classes, tests can be expressed as simple functions. Given this short function under test, here’s a comparison between the default UnitTest implementation and pytest.


  ## Contents of analytics.py
  import numpy

  def fastsum(array_like):
      return numpy.sum(array_like, axis=0)
  

UnitTest


  import unittest
  import numpy
  import analytics

  class TestAnalytics(unittest.TestCase):
      def test_fastsum(self):
          self.assertEqual(analytics.fastsum(numpy.array([1, 2, 3]),  6)
  

PyTest


  import numpy
  import analytics

  def test_fastsum():
      assert analytics.fastsum(numpy.array([1, 2, 3])) == 6
  

Smart Assertions

The built-in assert statement has been overloaded to provide in-depth comparison information about what you’re asserting.

UnitTest


  $ pytest test_analytics.py
  ========================== test session starts ===========================
  platform linux2 -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
  collected 1 items

  test_analytics.py F

  ================================ FAILURES ================================
  ________________________________ test_sum ________________________________

      def test_fastsum():
  >       assert analytics.fastsum([1, 2, 3]) == 7
  E       assert 6 == 7
  E        +  where 6 = analytics.fastsum([1, 2, 3])

  test_analytics.py:4: AssertionError
  =========================  1 failed in 0.01 seconds ======================
  

Data-driven test cases

Often a function under test can have many different types of inputs. It is laborious to write out each permutation of inputs and outputs as separate tests, so pytest has a great feature for creating data-driven parametrized test cases.


  import pytest
  import numpy as np
  import analytics

  @pytest.mark.parametrize('fn_in,expected_result',[
       ([1, 2, 3], 6),                  # Integers
       ([1.1, 2.2, 3.3], 6.6),          # Floats
       (np.array([1, 2, 3]), 6),        # Numpy Arrays
       # ... etc
       ([1.0, np.nan, 3.0], np.nan),    # Nan Handling
  ])
  def test_fastsum(fn_in, expected_result):
      result = analytics.fastsum(fn_in)
      if np.isnan(expected_result):
          assert np.isnan(result)
      else:
          assert result == expected_result
  

Powerful Re-usable Fixtures

Simple, parametrized test functions and smart assertions are only the start of why we like pytest. The real power comes from its dependency-injection-style test fixtures.

  • A test fixture is an object that is created by the test framework with some initial state and passed into any tests that have requested it by specifying the fixture name as a function argument.
  • Test fixtures have a scope, which determines their lifetime within the test run. The scope can be one of:
    • function : a single test function
    • class: a UnitTest style test class
    • module: a single Python test module
    • session: the entire session, from when pytest starts up till it finishes running all the tests
  • Fixtures have setup and teardown, that is run at the start and end of their lifetime.

Here’s a simple example that creates an in-memory SQLite database session and hands it to the test function:


  import pytest
  from sqlalchemy import create_engine
  from sqlalchemy.orm import sessionmaker

  @pytest.fixture
  def db_session():
      engine = create_engine('sqlite:///:memory:', echo=True)
      Session = sessionmaker(bind=engine)
      return Session()

  def test_select(db_session):
      """ the 'db_session' argument here is matched to the name of the
          fixture above
      """
      db_session.execute('select name from user')
      ... 
  

Fixtures Can Depend On Other Fixtures

In the same way that test functions depend on fixtures by specifying them by name as arguments, fixtures can depend on each other by specifying other fixture names in their own arguments. These dependencies will form a graph of objects that are created and torn down in the correct graph-order by test runner.

One can quickly see that this encourages libraries of re-usable test code that can be assembled to run complex integration tests with many moving parts. Here’s an example:


  import pytest

  import backend
  import frontend

  @pytest.yield_fixture(scope=session)
  def backend_server():
      # --- Setup ------
      server = backend.Server()
      server.start()
      yield server
      # --- Teardown ---
      server.stop()

  @pytest.fixture
  def frontend_client(backend_server):
      client = frontend.Client(host=server.host, port=server.port)
      client.connect()
      return client


  def test_client(frontend_client):
      frontend_client.login('admin', 'password')
      ...
  
  • In this example we have a session-scoped server fixture that is only created once, and a frontend client that is created once per test function.
  |   Session Scope    |            Function Scope            |
  |--------------------|--------------------------------------|
  |  backend_server <--|-- frontend_client <----- test_client |
  |                    |                                      |

  • The frontend client depends on the server fixture. This both ensures that the server is running at the time the client is needed, and also means the client can pull configuration details like host and port numbers out of the server class.
  • The test itself only asks for the client; pytest will ensure the server is started in time for the client to connect.

There is a gotcha here: fixture dependencies must obey scope precedence. You cannot have a session-scoped fixture depending on a function-scoped fixture, because the function-based fixture may have been destroyed within the lifetime of the session-based one.

Good Practice - Distribute Fixtures With The Code

As codebases grow, it makes sense for the teams that maintain a project to distribute the test fixtures for that project along with the normal code if its used by another team’s project. This way the fixtures remain ‘first class citizens’ and changes are propogated to the tests of other projects.

An example layout for a PnL service could be:

pnl_service/
    __init__.py
    server.py
    client.py
    fixtures.py   # this module contains a fixture called 'pnl_client'
  

Then in another project’s tests that use the PnL service, it can use the fixtures maintained by the PnL team.


  import attribution

  # This tells pytest that we want to use the fixtures from
  # the other project
  pytest_plugins = 'pnl_service.fixtures'

  def test_attribution(pnl_client):
      pnl_data = pnl_client.get_pnl(product='TEST_PRODUCT')
      ftl_data = attribution.get_market_attribution('FTL')
      assert ftl_data == [ .... ]
  

 

Open Sourced: Here’s Some We Created Earlier

There are lots of plugins for Pytest that provide fixtures and other functionality, including many built-in ones.
Many of the fixtures we’ve written over the years are generally useful and as such have been open-sourced. They are all available in the GitHub repository pytest-plugins. Here’s the list of plugins:

Plugin Description
pytest-server-fixtures Extensible server-running framework with a suite of well-known databases and webservices included: mongodb, redis, rethinkd, Jenkins, Apache httpd, Xvfb
pytest-shutil Unix shell and environment management tools
pytest-profiling Profiling plugin with tabular heat graph output and gprof support for C-Extensions
pytest-devpi-server DevPI server runnning fixture for testing package management code
pytest-pyramid-server Pyramid server fixture for running up servers in integration tests
pytest-webdriver Selenium webdriver fixture for testing web applications
pytest-virtualenv Create and teardown virtual environments, run tests and commands in the scope of the virtualenv
pytest-qt-app PyQT application fixture
pytest-listener TCP Listener/Reciever for testing remote systems
pytest-git Git repository fixture
pytest-svn SVN repository fixture
pytest-fixture-config Configuration tools for Py.test fixtures
pytest-verbose-parametrize Makes py.test’s parametrize output a little more verbose

How to get them

All of these plugins are available on PyPI and can be installed individually using pip:


    $ pip install pytest-server-fixtures
  

 

Further Reading

If you love testing your code as much we do, go ahead and install Pytest and get testing! It’s easy to get started and satisfying to know your software is in good hands when the tests all pass. Here’s some references for further reading:

Back to Technology

Latest Technology articles

How we centralized and simplified the capture of core dumps in Docker using common Linux tools.

Technology Team

How do you perform logging in software that represents a large mathematical model? If the model isn't behaving as expected, is it the code or the data that's at fault?

Technology Team

Important information

Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. This material is proprietary information of the Company and its affiliates and may not be reproduced or otherwise disseminated in whole or in part without prior written consent from the Company. The Company believes the content to be accurate. However accuracy is not warranted or guaranteed. The Company does not assume any liability in the case of incorrectly reported or incomplete information. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.

Please update your browser

Unfortunately we no longer support Internet Explorer 8, 7 and older for security reasons.

Please update your browser to a later version and try to access our site again.

Many thanks.