Logging in Large Mathematical Models
How do you perform logging in software that represents a large mathematical model? If the model isn't behaving as expected, is it the code or the data that's at fault?
Logging is crucial in any large system, but numerical programs add some unique challenges. You can't simply write a large numeric array to a text file: neither a human nor a computer can easily read that.
At Man AHL, we've solved this with a special diagnostic log format, affectionately known as a 'diag'.
Suppose we have some code of the form:
import numpy as np def complex_model(val): return val + 1.0 def apply_complex_model(arr): return np.apply_along_axis(complex_model, 0, arr)
If we're getting unexpected values from
apply_complex_model, does that indicate an issue with
complex_model or the value we passed in for
It's tempting to print
arr to the log, but that doesn't scale for large values. We also want to plot bad values, so we can eyeball the data.
We modify the code to log the value itself:
import numpy as np import ahl.diags as diags def apply_complex_model(arr): # In practice we provide decorators for common use cases # like logging inputs and outputs. with diags.prefix('complex_model'): diags.log("input", arr) output = np.apply_along_axis(complex_model, 0, arr) diags.log("output", output) return output
This has many advantages:
Interactivity: We can load up the diag in ipython and examine it.
Visibility: We can visualise the actual data that was used when the program ran. Inputs are typically timeseries, which lend themselves to plotting.
Reproducibility: Since we have the interesting inputs, we can re-run our
apply_complex_model function with these inputs. If we're bugfixing, we can run our new implementation against the same inputs.
Large mathematical models often have large inputs and outputs. Loading the entire diag into memory would be slow and resource intensive.
We store diags as efficiently serialised data in HDF5 files. HDF5 allows us to only load the values from the diag that we're interested in, without reading the whole file. This keeps loading snappy.
A diag looks like a nested dict of dicts:
>>> from ahl.diags import import_diag >>> my_diag = import_diag("~/example_diag.h5") >>> my_diag['complex_model']['input'].value array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Since we often examine them in ipython, we provide a more convenient API that aids tab-completion:
>>> my_diag.complex_model.input.value array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Good logs are readily available, and diags are no exception. We store our diags in the 'diags repo': a shared filesystem that's available to all our researchers and developers.
Every time a model runs, its diag is stored in this shared directory. This is incredibly powerful for debugging, and we've built reporting tools on top of the diags repo.
It's not easy to find diag files in this directory directly. With tens of thousands of files, it's hard to find the diag you're interested in.
We monitor the directory with a 'diags indexer' tool. When new diags are available, we update a Mongo database with diag metadata. We then provide a Python object that queries this database.
This database allows us to load diags according to specific constraints:
>>> my_diag = diags.repo.by_user.jdoe.last
This is great for discoverability, we can just press tab to interactively see what data is available:
# Which strategies are running in live? >>> diags.repo.by_platform.live.by_strategy.<TAB> # Which markets are we trading in preprod? >>> diags.repo.by_platform.preprod.by_market.<TAB>
The diag has all the advantage of a log, but it's structured and easy to build upon. More importantly, it contains real Python objects, so it's easy to examine. It's become a ubiquitous part of our tooling.
Opinions expressed are those of the author and may not be shared by all personnel of Man Group plc (‘Man’). These opinions are subject to change without notice, are for information purposes only and do not constitute an offer or invitation to make an investment in any financial instrument or in any product to which the Company and/or its affiliates provides investment advisory or any other financial services. Any organisations, financial instrument or products described in this material are mentioned for reference purposes only which should not be considered a recommendation for their purchase or sale. Neither the Company nor the authors shall be liable to any person for any action taken on the basis of the information provided. Some statements contained in this material concerning goals, strategies, outlook or other non-historical matters may be forward-looking statements and are based on current indicators and expectations. These forward-looking statements speak only as of the date on which they are made, and the Company undertakes no obligation to update or revise any forward-looking statements. These forward-looking statements are subject to risks and uncertainties that may cause actual results to differ materially from those contained in the statements. The Company and/or its affiliates may or may not have a position in any financial instrument mentioned and may or may not be actively trading in any such securities. This material is proprietary information of the Company and its affiliates and may not be reproduced or otherwise disseminated in whole or in part without prior written consent from the Company. The Company believes the content to be accurate. However accuracy is not warranted or guaranteed. The Company does not assume any liability in the case of incorrectly reported or incomplete information. Unless stated otherwise all information is provided by the Company. Past performance is not indicative of future results.