Consider deprecating `expect { }.not_to raise_error(SpecificErrorClass)`

[myronmarston]

An expression like expect { }.not_to raise_error is fine, as it expresses intent (particularly when paired with a corresponding example that demonstrates when a particular error is raised), but expect { }.not_to raise_error(SpecificErrorClass) is dangerous. It's prone to producing false positives. Consider:

expect {
  expect(some_helper(foo)).to eq(:bar)
}.not_to raise_error(SomeError)

With this in place, it's impossible for the inner expectation to ever cause the example to fail. When the inner expectation fails, it raises a RSpec::Expectations::ExpectationNotMetError, but this is not a SomeError, so the outer expectation causes it to pass.

Thus, I think we should deprecate a specific error class with the negative raise_error matching. In 3.0 it'd be nice to disallow it entirely.

[JonRowe]

Alternatively we could make it fail on a RSpec::Expectations::ExceptationNotMetError, or on any Error.

[fables-tales]

I'm in favour of removal of this: people either expect some specific error or no error whatsoever. It seems backwards to me to have a spec that passes when the vast majority of errors are thrown, apart from the one we're expecting not to happen.

[myronmarston]

Alternatively we could make it fail on a RSpec::Expectations::ExceptationNotMetError

This would improve things, but it doesn't solve the problem. Consider if the spec is like this:

expect {
  some_object.do_something
}.not_to raise_error(SomeError)

Later, we rename the do_something method to do_something_else, and run our specs to find all the places we need to update the caller. Problem is, the inner expression now raises a NoMethodError, and the expectation allows that to pass because NoMehtodError is not a SomeError...so this spec is now testing absolutely nothing.

or on any Error.

Sure, we could do that, but that's not what the expectation expresses, is it? I think it's confusing to do something entirely different from what the expectation expresses. I don't think people actually ever really want this -- they're just thinking "this used to raise exception X", but I'm fixing it so it doesn't do that anymore -- and they use this expectation without thinking of the consequences. (See rspec/rspec-core#846 for an example -- it's what prompted me to open this issue).

[cupakromer]

After considering this and looking at the code bases I'm working on. The places were not_to raise_error are used, aren't necessary and should be re-written to be more explicit.

👍

[JonRowe]

I'd be happy for the behaviour to be deprecated and removed to be honest, I was just providing some alternative thought. If it's kept I think we should change the behaviour to catch only the specific named error, or all errors if none are named. That way it would be more explicit in it's intent and should create less false positives.

[hron84]

As previous I still keep my opinion: if code raises error other than we expected it is a problem of the example or problem of RSpec?

We really want to fix developer errors too?

with @myronmarston example:

expect {
  some_object.do_something
}.not_to raise_error(SomeError)

Developer wants to express the wrapped code should not raise an error what specified. The naive developer assumes if error is raised what is not a SomeError, e.g. NameError the expectation not activated, but the error itself is raised ahead. It means the example is fails with E(xception) and not F(ailed) (and of course, definitelly not pass). Another situation - what would be covered with another exampel, or code needs fix - but it's a developer's job, not of the test framework.

If we fail if any error happened it looks like a good thing - but if developer wants to fail on any error, then he not specify an error class in the method - the current codebase provides a way for this. If developer specify an error class then he wants to example pass when - and only when - the block is not raises that specific error. For example. if I want to check a templating system to finds templates correctly, I can expect to not raises a TemplateNotFoundError. But it should raise another error, because - for example - the codebase is not completed - but template loaded correctly.

[myronmarston]

if code raises error other than we expected it is a problem of the example or problem of RSpec?

It could be either: RSpec, like all software, has bugs, and it could be a bug in RSpec causing an error to be raised. Or it could be a problem with the example itself (or the code the example tests). Regardless, I don't think to silence it.

The naive developer assumes if error is raised what is not a SomeError, e.g. NameError the expectation not activated, but the error itself is raised ahead.

I'm not 100% sure what you mean by this, but it sounds like you're saying a naive developer assumes this expectation concerns itself only with the named exception, and would allow other types of exceptions to propagate?

I don't really know what a naive developer expects. Regardless, I don't think the behavior of this expectation is ever something you'd want. (I think folks who do want it don't quite understand what it is doing yet).

Another situation - what would be covered with another exampel, or code needs fix - but it's a developer's job, not of the test framework.

Of course it's the developer's job to fix problems in their code. But if the test framework is hiding errors from them (as this expectation does), they won't know they have a problem that needs fixing. This is a bad thing.

[hron84]

I'm not 100% sure what you mean by this, but it sounds like you're saying a naive developer assumes this
expectation concerns itself only with the named exception, and would allow other types of exceptions to
propagate?

Exactly, sorry for my poor English.

There are many libs what communicates the problems with the developer via exceptions/errors. Especially in Ruby, devs usually do not have a C-like approach returning with nil, false, or so if error happened, instead raises an error.

What I want to say to code not raises a specific error? I allow the code to raise any other error because I know the code is not complete yet (e.g. because I follow TDD pattern), and code may raise NoMethodError, SyntaxError, and so on, it is a natural thing. But, what I want to express: this code is not raises SomeError, what is means I didn't made a specific type of mistake. But it is not means I fixed all errors in the code, it means the code should not ran into a well-defined problem (what causes SomeError being raised).

Back to ERB-trimming topic: ERB parser is raises SyntaxError if ERB.new called with wrong arguments. Even if it is a sign of poor architecture of ERB parser (it should check the template syntax an raise a TemplateParsingError instead), as a client of it I must accept this behavior. However, this means no way to ensure the trimming option is passed to ERB templating than checking the ERB parser code is not raises SyntaxError if I pass a template with trimming 'dash' (and without a syntax error, of course). However, the template parser code should raise any other error, e.g. NameError if I follow TDD pattern and I call another function what's not defined/implemented yet. Or the tester block can raise ExpectationNotMetError, because it is definitely not a SyntaxError.

[cupakromer]

@hron84 I see where you're coming from. I had that very same thought process previously. However, the test quickly becomes brittle.

Sure, you know now that you aren't creating a SyntaxError. You also don't know what other errors you are creating. Testing for the lack of the SyntaxError doesn't actually test that you did what you want; just that you didn't cause a syntax error. You want to be testing for the expected side-effect or outcome. If you get a SyntaxError you know you did something wrong. When that error goes away, you did something correct and your test, should be for what you did (outcome / side-effect).

Regarding TDD. I don't see this as an issue. Depending on how tight and small your TDD loop is, the argument can be made that you would only get the missing method exception as you implement the next feature. By your argument, you should have a test that checks that no exception is raised after you implement that method. However, that would mean you should then start adding test to make sure each possible error (that you know of) is not raised as you are testing. That's not going to add value to the test suite in the long run.

By not testing for an exception and testing the side-effect / outcome you have a solid test that demonstrates the behavior of the code. It will at the same time implicitly test for things like SyntaxError, NameError, etc.

Consider the scenario where in the future you or a colleague replaces ERB with something else (possibly a contrived scenario, but it illustrates the point of brittleness). The exception test will pass falsely. It is no longer testing it's original intent. It does not fail because no syntax errors were made, but the code that could cause a syntax error is no longer there. By instead just testing the outcomes, when you change the behavior of the method, that will fail and you/your colleague can then delete the out of date test.

[hron84]

@cupakromer

Problem: in ERB, no side-effect / outcome, if no trimming argument specified. The SyntaxError is the only thing you can test! So template.result(binding) raises an error and does not returns with anything. And no syntax checking method in ERB templating engine, because it is not a high-level templating language. ERB is basically translates template tags to some wrapper ruby code and evaluates it with eval, that's all. It is not like a HAML or SLiM what builds a syntax tree and checks it before compile/run the template code itself. ERB has very simple and very dumb implementation to use it anywhere without spending a performance for it.

Usually, I accept to not test errors, because if they appears that means something definitely wrong happened. If an error is raised, there is a mistake in the code or in the logic. (SyntaxError is even worse, because any Ruby code can raise it if there is a - syntax error. This is why this example was problematic.)
However, there is some external library with bad behaviors what I - as an application developer - cannot and should not change. My example is tried to show there is some situation where I should need to exclude some error from checking.

To be clear: I usually do not do things like checking to not raise SyntaxError. It is a crazy and useless idea, especially because - as an experienced developer - I know humans make syntax error too easily. I taken this example because I talked about it in other RSpec issue previously and I kept it in my mind.
Furthermore, I usually do use the complete expect { }.to_not raise_error very-very rarely as usually I want to all errors be raised from tests because it helps me to fix all errors in my code.

[fables-tales]

Closed by #244