Skip to content

Better way to attach visual regression diffs to Allure test reporter #888

@saransh88

Description

@saransh88

Is your feature request related to a problem? Please describe.
I am using @wdio/visual-service for visual regression testing.
While trying to integrate visual diff results into Allure reports, I couldn't find a straightforward way to attach the baseline, actual, and diff images to the test results.

There doesn’t seem to be a public API or built-in support today to easily hook into the visual diff results and attach them to the Allure reporter.

Describe the solution you'd like
I would like to see it be seem less similar to how test and screenshot per tests work

Describe alternatives you've considered
Right now, I’m manually parsing the output.json file generated by the visual service.
From there, I extract the paths to the baseline, actual, and diff images, and manually attach them to Allure like this:

// Read the output.json file
            const outputFilePath = path.join(process.cwd(), 'tmp', 'actual', 'output.json');
            const outputData = JSON.parse(fs.readFileSync(outputFilePath, 'utf-8'));

            // Find the relevant test data
            const testData = outputData.find(entry => entry.data.some(t => t.test === test.title));
            if (testData) {
                const testDetails = testData.data.find(t => t.test === test.title);

                // Loop through all tags for the test
                testDetails.data.forEach(tagData => {
                    const { actualFilePath, baselineFilePath, diffFilePath } = tagData.fileData;

                    if (actualFilePath && baselineFilePath) {
                        const expected = fs.readFileSync(baselineFilePath, 'base64');
                        const actual = fs.readFileSync(actualFilePath, 'base64');
                        const diff = diffFilePath ? fs.readFileSync(diffFilePath, 'base64') : null;

                        // Wrap in a JSON, encode as string
                        const content = JSON.stringify({
                            expected: `data:image/png;base64,${expected}`,
                            actual: `data:image/png;base64,${actual}`,
                            diff: diff ? `data:image/png;base64,${diff}` : null,
                        });

                        // Attach to the test report
                        allureReporter.addAttachment('Visual regression result', content, 'application/vnd.allure.image.diff');
                    } else {
                        console.warn(`Missing file paths for tag: ${tagData.tag}`);
                    }
                });
            }
        }

Question
Is there an easier or more official way today to attach visual regression results from @wdio/visual-service to the Allure test reporter?

If not, would this be something better solved on the WebdriverIO side (visual service API) or on the Allure reporter side (supporting attachments for visual diffs automatically)?

Additional context
The goal is to automatically capture and attach visual diff artifacts (baseline, actual, diff) to each test result without requiring manual work inside each test file.

This is what I was able to achieve with the my workaround
https://github.com/user-attachments/assets/2a1ea296-4bc4-4c25-9185-6c11b77b3204

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions