Hash Comparison Mode

This how-to guide will show you how to use the hash comparison mode of pytest-mpl.

In this mode, the hash of the image is compared to the hash of the baseline image. Only the hash value of the baseline image, rather than the full image, needs to be stored in the repository. This means that the repository size is reduced, and the images can be regenerated if necessary. This approach does however make it more difficult to visually inspect any changes to the images.

If your goal is to not commit any images to the code repository, then you should consider using hybrid mode instead. In this mode, the hashes can be stored in the code repository, while the baseline images are stored in a separate repository and accessed through a URL when testing.

Generating baseline hashes

Once a suite of image comparison tests have been written, baseline hashes should be generated by setting --mpl-generate-hash-library:

pytest --mpl-generate-hash-library=your_project/tests/hashes.json

It is important to visually inspect the figures before generating baseline hashes. So, as well as generating baseline hashes, this command runs baseline image comparison tests. If no baseline images exist in the default directory, this command will fail.

A better option is to generate baseline images along with the baseline hashes to ensure that the images are as expected, even if you do not wish to use them for comparison:

pytest \
  --mpl-generate-hash-library=your_project/tests/mpl35_ft261.json \
  --mpl-generate-path=baseline

To assist with inspecting the generated images (and hashes), a HTML summary report can be generated by setting --mpl-generate-summary:

pytest \
  --mpl-generate-hash-library=test_hashes.json \
  --mpl-generate-path=baseline \
  --mpl-results-path=results \
  --mpl-generate-summary=html,json

HTMLJSON

You should choose a directory within you repository to store the baseline hashes. It’s usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. mpl35_ft261.json. The hash library file should then be committed to the repository.

Running hash comparison tests

When running the tests, the --mpl flag should be used along with a configured hash library path to enable baseline hash comparison testing:

pytest --mpl \
  --mpl-hash-library=your_project/tests/mpl35_ft261.json

Optionally, a HTML summary report can be generated by setting --mpl-generate-summary:

pytest --mpl \
  --mpl-hash-library=your_project/tests/mpl35_ft261.json \
  --mpl-results-path=results \
  --mpl-generate-summary=html,json

HTMLJSON

The --mpl-results-path flag can be used to set the directory where the generated HTML summary will be stored. If this is not set, the images will be stored in a temporary directory.

Continue reading

pytest-mpl has many configuration options that can be used to customize the behavior of the hash comparison mode. Only a few of the most commonly used options are covered in this guide. See the configuration options documentation for full details.