Standard reporter interface

[jzaefferer]

We want to provide a reporter interface that we can share with other testing tools like Jasmine and Mocha, to make it easier for integration tools - like Karma, browserstack-runner, grunt plugins - to hook into QUnit and other tools. Anyone writing a new JavaScript testing tool should be encouraged to provide the same interface, making it more reasonable to support these tools.

This requires a bunch of research. So far we identified two components that need to be standardised:

• needs a data format ala JUnit XML (here's an unofficial xsd to validate the output) or TAP, that is flexible enough to support QUnit and others. Specifically, Jasmine and other BDD styles support (more or less) infinite nesting, which QUnit doesn't
• need a list of events that provide this data. Something like done should provide the full datastructure, other events should help to output progress

Related tickets:

• (testDone should provide aggregated assertion results #351) testDone should provide aggregated assertion results
• (Implement reporter interface #405) Implement reporter interface
• (Implement QUnit callbacks event listener style #422) Implement QUnit callbacks event listener style
• (QUnit.done should provide aggregated results #472) QUnit.done should provide aggregated results

[JamesMGreene]

I reviewed Mocha's reporter interface first.

Important details:

• All reporters seem to inherit from a base reporter, although it doesn't appear that this is an actual requirement. The base reporter is responsible for providing many base utilities, such diff-ing displays, etc.
• The reporters are setup as an event listener that listens for the following events to be emitted:
  • runner.on('start', fn) → QUnit.begin(fn);
  • runner.on('suite', fn) → Mostly equivalent to QUnit.moduleStart(fn) except that Mocha suites can be infinitely nested. The nesting is simply achieved by treating suites as a stack: if a new suite event is emitted without a suite end event for the previous suite, the new suite is a child of the previous suite.
  • runner.on('test', fn) → QUnit.testStart(fn)
  • runner.on('pass', fn) → Equivalent to QUnit.testDone(fn) with a "success" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('fail', fn) → Equivalent to QUnit.testDone(fn) with a "failure" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('pending', fn) → Equivalent to QUnit.testDone(fn) with a "pending" status. This is a convenience event only as the same info could be obtained via the test end event.
  • runner.on('test end', fn) → QUnit.testDone(fn)
  • runner.on('suite end', fn) → Mostly equivalent to QUnit.moduleDone(fn) except that Mocha suites can be infinitely nested.
  • runner.on('end', fn) → QUnit.done(fn)

There are also two more events that relate to the before, beforeEach, afterEach, and after functions:

• runner.on('hook', fn)
• runner.on('hook end', fn)

The reporter can be set by calling mocha.reporter(reporter) or via the Mocha constructor options [if you're using Mocha programmatically].

Mocha will only allow a single reporter.

Mocha does not have an event for assertions.

[JamesMGreene]

Jasmine reporters are classes that can implement up to 6 core eventing methods which will be invoked during the appropriate times:

• reporter.jasmineStarted → QUnit.begin
• reporter.suiteStarted → Mostly equivalent to QUnit.moduleStart except that Jasmine suites can be infinitely nested.
• reporter.specStarted → QUnit.testStart
• reporter.specDone → QUnit.testDone
• reporter.suiteDone → Mostly equivalent to QUnit.moduleDone except that Jasmine suites can be infinitely nested.
• reporter.jasmineDone → QUnit.done

Reporters are added via jasmine.getEnv().addReporter(reporter).

Jasmine will allow using multiple reporters simultaneously.

Jasmine does not have an event for assertions.

[jzaefferer]

That is looking promising, since the overall structure in all three projects is the same. QUnit only has one level of nesting for "suites", so the common standard should include the nesting, even if we won't make use of it.

The EventEmmiter style that Mocha implements and we've discussed previously (#422) seems like a good starting point.

Both mocha and jasmine use "suite" for the component we call "module". That reminds me of the discussion in #190. Not that I have any interest in reopening that once more.

[Krinkle]

Are we looking for a standard reporting interface (e.g. the EventEmitter interface, and a standardised set of event names and arguments, which HTML or CLI formatters can use to display results and indicate progress). Or for a standard test suite summary format (e.g. like a JSON variant of jUnit TestSuite XML output) that one would retrieve at once to store and display results (either all, or just the failures).

I suppose both are interesting, but they're different things. I think the latter is more valuable for now (for grunt plugins and browserstack/saucelabs interpreters). The way to get that summary is probably not a big deal and can be slightly different per test framework (it's unlikely they'd factor out that one line of code to be the same between all frameworks). It'd be like QUnit.on( "<eventname>", function( summary ) { callback( summary ); });

[Krinkle]

Regarding the summary format, see also jquery/testswarm#183.

[Krinkle]

After today's QUnit meeting, @JamesMGreene put together a proposal: https://gist.github.com/JamesMGreene/9786246

---

Couple of thoughts on https://gist.github.com/JamesMGreene/9786246/90f16610b02b8267d18f61b7648b6364ca84da39:

• I'm not sure if the universal reporter interface should distinguish between suites and tests. Not every test framework might have these. And the ones that do, implement them quite differently. For example, in Jasmine suites and tests can be nested. Is there a use case of separating these?
• I'd recommend using names that are also valid identifiers in javascript (ideally both es3 and es5, but at least es5) so that we don't need bracket syntax and string literals all over the place.
• Though this is just coding style, knowing examples sometimes get blindly copied and are hard to kill once they get spread, I'd reduce the examples to 1) Not do typeof function check (just plain boolean check), 2) Use dot notation, 3) Not create local variables for each of them.

[leobalter]

@Krinkle: I did some changes based on your suggestions: https://gist.github.com/leobalter/226637d148577ac9b5c3

I did some research about reserved words as well to check for irregularities, everything is ok.

[leobalter]

@garu is a friend who is a Perl developers used to the TAP specification. I've talked to him and he gently agreed to help prototyping the methods for the TAP output.

Btw, now I also saw that we should expand the methods exportation to handle the passed data in each of the callback arguments. Probably Jasmine and Mocha are both different from QUnit too.

[JamesMGreene]

Cross-reference issues:

• Share a standard Reporter interface? mochajs/mocha#1326
• Share a standard Reporter interface? jasmine/jasmine#659
• Share a standard Reporter interface? theintern/intern#257