Learn how to automate website visual regression testing using a screenshot API. Step-by-step setup, best practices & CI/CD integration.
What Is Website Visual Regression Testing?
Visual regression testing is a technique used to identify unintended visual changes in a user interface (UI) after updates or modifications. It helps ensure that the appearance of an application remains consistent over time by comparing baseline screenshots (previous UI state) with new screenshots after code modifications. It helps identify issues such as layout shifts, missing elements, or altered styles that may not be easily caught by traditional functional testing. Some common visual regressions are:
- Layout shifts Unexpected movement of elements (e.g. text blocks jump when ads load).
- Broken or missing assets Images, icons or fonts fail to load, showing “alt” text or blank squares.
- Styling inconsistencies Color changes, border misplacements or font‐size regressions due to CSS cascade issues.
- Overlapping or clipped elements Buttons, text or inputs overlap, or get cut off at container boundaries.
- Responsive‐breakpoint errors Components that look fine at one viewport width but overflow or collapse at another.
- Missing or shifted whitespace Padding/margin changes that break the intended flow or alignment of content.
How Screenshot APIs Simplify Visual Regression?
Visual regression testing involves capturing screenshots (baseline images) of the UI and then comparing them with new screenshots after changes to the codebase. Any differences are flagged as potential regressions.
The process is often automated using specialized tools that can highlight discrepancies between the baseline and the new UI, enabling developers to quickly locate and fix issues.
Visual regression testing is commonly integrated into Continuous Integration/Continuous Deployment (CI/CD) pipelines, so visual integrity is maintained with every code update.
Using Capture for visual regression testing
Capture offers a blazing fast screenshot API that is perfect for this job. Using Capture's screenshot API we can fetch screenshot of staged version of an app and compare it with screenshot of production version of the app to see if changes are acceptable, or we could have a baseline version stored to compare against a change and verify if there is a regression. Let's try to build both approach.
Test 1: Comparing staging vs production
To make this work we have to make sure we've a deployed version of staging and production app. For this example, we are going to use the following endpoints for testing:
Staging: https://example.com/hello-staging Production: https://example.com/hello
First, we're going to generate the screenshot URL for both these pages using Capture, you can find instructions here.
Staging screenshot: https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/09c84600d453495a9bbf778889f0a443/image?url=https%3A%2F%2Fexample.com%2Fhello-staging Production screenshot: https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/ab4b2ac5d12f104c1dafbd29a378892e/image?url=https%3A%2F%2Fexample.com%2Fhello
We're going to download these images and then compare them using ImageMagick
curl -o staging.png https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/09c84600d453495a9bbf778889f0a443/image?url=https%3A%2F%2Fexample.com%2Fhello-staging
curl -o prod.png https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/ab4b2ac5d12f104c1dafbd29a378892e/image?url=https%3A%2F%2Fexample.com%2Fhello
compare -metric AE staging.png prod.png diff.png
You'll see the result 0 (0)% denoting that the images are identical and we can verify the same from the images. In some cases there could be minor differences and its always good to allow a certain margin for errors, a small percentage should suffice.
Test 2: Compare baseline against staging
This should be straightforward as we've seen the previous example, we need to take screenshot of the updated version of our page and then compare it with baseline.
curl -o staging.png https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/09c84600d453495a9bbf778889f0a443/image?url=https%3A%2F%2Fexample.com%2Fhello-staging
compare -metric AE staging.png baseline.png diff.png
Integrate with CI/CD
Now that we've learned how the testing works, we can integrate it into our continuous integration pipeline so that we can run these tests every-time we're changing our web pages. Here is a script that can compare two pages and check the diff against a threshold.
#!/bin/bash
# Define the threshold for acceptable difference (in pixels)
# Adjust this value based on your acceptable level of visual change
THRESHOLD=100
# Download the staging and production screenshots
curl -o staging.png https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/09c84600d453495a9bbf778889f0a443/image?url=https%3A%2F%2Fexample.com%2Fhello-staging
curl -o prod.png https://cdn.capture.page/e1ab7054-dabc-48d6-a33f-c18038aac1c8/ab4b2ac5d12f104c1dafbd29a378892e/image?url=https%3A%2F%2Fexample.com%2Fhello
# Compare the images and capture the Absolute Error (AE)
# The error value is typically the first number in the output
ERROR=$(compare -metric AE staging.png prod.png diff.png 2>&1 | awk '{print $1}')
# Check if the error is greater than the threshold
if (( ERROR > THRESHOLD )); then
echo "Visual difference (AE: $ERROR) is above the threshold ($THRESHOLD)."
echo "Check diff.png for details."
exit 1 # Exit with a non-zero status code to indicate failure
else
echo "Visual difference (AE: $ERROR) is within the acceptable threshold ($THRESHOLD)."
exit 0 # Exit with a zero status code to indicate success
fi
When we run the script we to compare our test staging against production, we will see:
Visual difference (AE: 0) is within the acceptable threshold (100).
And we've successfully passed the test as the difference in within our threshold.
Best Practices for Effective Visual Regression
What we've tried above is comparing two pages entirely against a staging vs current production version scenario, although this might work for some pages such as legal pages or pages that change infrequently, for most pages we would have frequent changes for the overall pages. So its important to verify certain elements in the page or certain components alone and not the entire page.
Capture lets you do that by capturing certain elements in a page. For example I could check that header always has a Sign Up button by capturing the screenshot of that specific part of the page using selector options in Capture. You can see the full list of screenshot options here: https://docs.capture.page/docs/screenshot-options
It might also require some trial and error to determine the correct threshold to determine a test pass or failing. We've to make sure that we account for subtle variants and do not alert too many false positives.
I would also recommend running tests in multiple device dimensions so that we can make sure our UI behaves as expected in all devices. This can be achieved by specifying custom viewport dimensions (vw && vh) and custom user agent (userAgent) while taking screenshot. This can help in minimising responsive breakpoint errors which are a very common visual regression.
If you'd like to try Capture for visual regression testing, you can get started here: https://capture.page