Hybrid Mode: Hashes and Images#

This how-to guide will show you how to use the hybrid mode of pytest-mpl.

For a full description of the hybrid mode, see the hybrid mode section of the get started guide. In summary, hybrid mode uses both baseline images and hashes. First, the hash of the image is compared to the hash of the baseline image. If the hashes match, the test passes. If the hashes do not match, the test fails.

The difference with hybrid mode is that a baseline image comparison will also be carried out if the hashes do not match, or always if this has been configured. The purpose of the additional image comparison (which does not affect the test result) is to allow the user to visually inspect the difference between the baseline image and the image generated by the test.

In order to keep the code repository size small, it is recommended to store the baseline hashes in the code repository, and the baseline images in a separate repository. The baseline hashes should be updated where appropriate in PRs to the code repository. However, the baseline images are not updated in these PRs. Instead, they should be updated once the PR has been merged, preferably by a CI job.

Another benefit of only updating the baseline images once the PR has been merged is that the PR tests will show the difference between the remote baseline images and the images generated by the PR. Even though the tests will pass when the baseline hash matches, the images will still be compared and the difference will be shown in the HTML test summary report, which is useful when reviewing the PR.

Generating baseline hashes and images#

Once a suite of image comparison tests have been written, baseline hashes and images should be generated by setting --mpl-generate-path and --mpl-generate-hash-library:

pytest \
  --mpl-generate-hash-library=your_project/tests/test_hashes.json \
  --mpl-generate-path=baseline \
  --mpl-results-path=results \
  --mpl-generate-summary=html,json

HTML JSON

Open the HTML summary file and inspect the figures to ensure that the baseline images are correct. If they are, the baseline hashes can be committed to the code repository. It’s usually a good idea to encode the Matplotlib version and the FreeType version in the filename, e.g. mpl35_ft261.json. The baseline images should be copied to a separate repository; preferably within a version specific directory, e.g. mpl35_ft261/.

Running hash comparison tests#

When running the tests, the --mpl flag should be used along with a configured hash library path and baseline image path to enable hybrid mode testing:

pytest --mpl \
  --mpl-hash-library=your_project/tests/mpl35_ft261.json \
  --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
  --mpl-results-path=results \
  --mpl-generate-summary=html,json

HTML JSON

The --mpl-results-path flag can be used to set the directory where the generated HTML summary will be stored. If this is not set, the images will be stored in a temporary directory.

Notice that the baseline image path is set to a URL, which is the location of the baseline images in the separate repository. When the baseline image comparison is carried out, the baseline images will be downloaded from this URL.

It is recommended to create a CI job that updates the baseline images in the separate repository once the PR has been merged. The CI job should run when new commits are pushed to the default branch. The baseline images should only be regenerated and updated if the tests pass in the hash comparison mode.

Aside: basic HTML summary

This is what the basic HTML summary looks like for the same test as above:

pytest --mpl \
  --mpl-hash-library=your_project/tests/mpl35_ft261.json \
  --mpl-baseline-path=https://raw.githubusercontent.com/your-org/your-project-figure-tests/mpl35_ft261/ \
  --mpl-results-path=results \
  --mpl-generate-summary=basic-html,json

Basic HTML JSON

Continue reading#

pytest-mpl has many configuration options that can be used to customize the behavior of the hybrid mode. Only a few of the most commonly used options are covered in this guide. See the configuration options documentation for full details.