Fail Tests on write to System.out

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Fail Tests on write to System.out

JUnit - User mailing list
Hi,

I am looking for a way to fail unittests when they write to System.out
or System.err. I found different ways to achieve that and tried a
solution with a RunListener, because these can be configured using the
maven-surefire-plugin. Now I discovered, that RunListeners can't really
fail tests because fail() throws an AssertionException and Exceptions
remove the RunListener. Is there a better way to discover output?

To summarize what I want:
* a way to discover output via System.out without preventing Logger
functionality
* this way should not involve huge code changes on the testclasses (
like adding TestRules everywhere)
* all unittests which produced output should be reported as "failing"
when you run with `mvn test`

I tried to answer these questions on SO but got only a few helpful
responses, so I thought I ask the specialists here and get more
opinions. For reference the SO questions:
* http://stackoverflow.com/questions/33214683
* http://stackoverflow.com/questions/33207996

Any help on this topic is very appreciated.

Regards
Andreas Worm




Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
Hi Andreas,

This is not going to be useful if you're using plain JUnit but you can look
at the solution in randomizedtesting -- we use it in Lucene and Solr and it
works fairly well.

You do need to have a common superclass or repeat the rule annotation on
every class this should be applied to, unfortunately. This is the
annotation:

https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L206

this annotation is used for suppressing it (for a class):

https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L352-L365

and this is rule declaration chain:

https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L581

The scenario in Lucene is quite complex, for example you want to suppress
certain checks on failures (other rules are permitted to write diagnostics
to sysout if something fails), but you can look for the inspiration on how
it was done here:

https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleLimitSysouts.java

If you peek at the source code and the comments you'll see the difficulties
this entails (early-on stolen references to System.out, running with a
security manager, etc.).

Hope this helps somehow,
Dawid


On Wed, Oct 21, 2015 at 2:07 PM, Andreas Worm [hidden email]
[junit] <[hidden email]> wrote:

>
>
> Hi,
>
> I am looking for a way to fail unittests when they write to System.out
> or System.err. I found different ways to achieve that and tried a
> solution with a RunListener, because these can be configured using the
> maven-surefire-plugin. Now I discovered, that RunListeners can't really
> fail tests because fail() throws an AssertionException and Exceptions
> remove the RunListener. Is there a better way to discover output?
>
> To summarize what I want:
> * a way to discover output via System.out without preventing Logger
> functionality
> * this way should not involve huge code changes on the testclasses (
> like adding TestRules everywhere)
> * all unittests which produced output should be reported as "failing"
> when you run with `mvn test`
>
> I tried to answer these questions on SO but got only a few helpful
> responses, so I thought I ask the specialists here and get more
> opinions. For reference the SO questions:
> * http://stackoverflow.com/questions/33214683
> * http://stackoverflow.com/questions/33207996
>
> Any help on this topic is very appreciated.
>
> Regards
> Andreas Worm
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
Hi Andreas.

I find it difficult to find out whether a call to System.out is being okay or not in your setup. are you saying that you want to suppress calls to System.out for your test execution entirely? it will be difficult to find out if it was your code that called it or if it was some kind of the testing framework for example.

The only way I currently see is to wrap your test execution with some kind of filter. The mechanism for that in JUnit is the Rule annotation. To avoid the definition of a rule in each test class your could introduce a base class that supports this for all your tests - still not very flexible.

I would recommend to avoid System.out to be used in the code directly. Instead use some kind of writer that you can mock out entirely for the tests. if that is not possible you probably have to rely on rules for the moment.

All the best

Stefan


> Am 22.10.2015 um 08:24 schrieb Dawid Weiss [hidden email] [junit] <[hidden email]>:
>
>
> Hi Andreas,
>
> This is not going to be useful if you're using plain JUnit but you can look at the solution in randomizedtesting -- we use it in Lucene and Solr and it works fairly well.
>
> You do need to have a common superclass or repeat the rule annotation on every class this should be applied to, unfortunately. This is the annotation:
>
> https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L206
>
> this annotation is used for suppressing it (for a class):
>
> https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L352-L365
>
> and this is rule declaration chain:
>
> https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java#L581
>
> The scenario in Lucene is quite complex, for example you want to suppress certain checks on failures (other rules are permitted to write diagnostics to sysout if something fails), but you can look for the inspiration on how it was done here:
>
> https://github.com/apache/lucene-solr/blob/trunk/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleLimitSysouts.java
>
> If you peek at the source code and the comments you'll see the difficulties this entails (early-on stolen references to System.out, running with a security manager, etc.).
>
> Hope this helps somehow,
> Dawid
>
>
>> On Wed, Oct 21, 2015 at 2:07 PM, Andreas Worm [hidden email] [junit] <[hidden email]> wrote:
>>  
>> Hi,
>>
>> I am looking for a way to fail unittests when they write to System.out
>> or System.err. I found different ways to achieve that and tried a
>> solution with a RunListener, because these can be configured using the
>> maven-surefire-plugin. Now I discovered, that RunListeners can't really
>> fail tests because fail() throws an AssertionException and Exceptions
>> remove the RunListener. Is there a better way to discover output?
>>
>> To summarize what I want:
>> * a way to discover output via System.out without preventing Logger
>> functionality
>> * this way should not involve huge code changes on the testclasses (
>> like adding TestRules everywhere)
>> * all unittests which produced output should be reported as "failing"
>> when you run with `mvn test`
>>
>> I tried to answer these questions on SO but got only a few helpful
>> responses, so I thought I ask the specialists here and get more
>> opinions. For reference the SO questions:
>> * http://stackoverflow.com/questions/33214683
>> * http://stackoverflow.com/questions/33207996
>>
>> Any help on this topic is very appreciated.
>>
>> Regards
>> Andreas Worm
>
>

image1.jpeg (114K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
In reply to this post by JUnit - User mailing list
Thanks Dawid
> You do need to have a common superclass or repeat the rule annotation
> on every class this should be applied to, unfortunately.
limiting the amount of output to a configurable number of bytes is
pretty cool but as you said, I have to touch every testclass to make
them obey the rules. Though, there seems to be no better way. So far,
thanks a lot, the whole TestRuleLimitSysouts class is very interesting.

--
"Shhh.... I'm charging my laser."

    // Andreas Worm - headissue GmbH - Germany
  \//  https://headissue.com

Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
> limiting the amount of output to a configurable number of bytes is pretty
> cool but as you
>

We didn't look for the coolness factor here, actually. In practice it's
fairly typical that developers will want to sysout stuff for debugging (in
tests). This is fine, unless the sysouts are huge, in which case this takes
resources (the runner spills outputs to disk, because there are multiple
parallel JVMs, etc.) and time. Hence the "soft limit".

As for preventing System.out calls in your production code (I suspect you
wish to enforce loggers or some other kind of infrastructure), take a look
at Uwe Schindler's forbidden-APIs tool, which parses bytecode and can fail
the build if an unwanted API is called. This is very useful because you can
apply this selectively, suppress inside code, etc.

https://github.com/policeman-tools/forbidden-apis/

Dawid
Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
I think I'd just use System.setErr and setOut to change the streams to one
that whose contents I could track and assert against.



On Thu, Oct 22, 2015 at 12:48 PM, Dawid Weiss [hidden email] [junit]
<[hidden email]> wrote:

>
>
>
>
>> limiting the amount of output to a configurable number of bytes is pretty
>> cool but as you
>>
>
> We didn't look for the coolness factor here, actually. In practice it's
> fairly typical that developers will want to sysout stuff for debugging (in
> tests). This is fine, unless the sysouts are huge, in which case this takes
> resources (the runner spills outputs to disk, because there are multiple
> parallel JVMs, etc.) and time. Hence the "soft limit".
>
> As for preventing System.out calls in your production code (I suspect you
> wish to enforce loggers or some other kind of infrastructure), take a look
> at Uwe Schindler's forbidden-APIs tool, which parses bytecode and can fail
> the build if an unwanted API is called. This is very useful because you can
> apply this selectively, suppress inside code, etc.
>
> https://github.com/policeman-tools/forbidden-apis/
>
> Dawid
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
In reply to this post by JUnit - User mailing list
The policeman-tools  were recommended in the Stackoverflow questions
too. I may find time to try them out next week. If they don't prove
useful, I will use TestRules. Again thanks for all the answers and
especially thanks Dawid for sharing his TestRule.

Regards
Andreas

On 22/10/15 18:48, Dawid Weiss [hidden email] [junit] wrote:

>
>     limiting the amount of output to a configurable number of bytes is
>     pretty cool but as you
>
>
> We didn't look for the coolness factor here, actually. In practice
> it's fairly typical that developers will want to sysout stuff for
> debugging (in tests). This is fine, unless the sysouts are huge, in
> which case this takes resources (the runner spills outputs to disk,
> because there are multiple parallel JVMs, etc.) and time. Hence the
> "soft limit".
>
> As for preventing System.out calls in your production code (I suspect
> you wish to enforce loggers or some other kind of infrastructure),
> take a look at Uwe Schindler's forbidden-APIs tool, which parses
> bytecode and can fail the build if an unwanted API is called. This is
> very useful because you can apply this selectively, suppress inside
> code, etc.
>
> https://github.com/policeman-tools/forbidden-apis/
>
> Dawid
>
>

--
"Shhh.... I'm charging my laser."

    // Andreas Worm - headissue GmbH - Germany
  \//  https://headissue.com

Reply | Threaded
Open this post in threaded view
|

Re: Fail Tests on write to System.out

JUnit - User mailing list
In reply to this post by JUnit - User mailing list
Have you taken a look at:

 System Rules
 

 It's and interesting approach that continues to expand on the notion of collections of Rules for reducing test code duplication.  We've only looked at this for about a week at this point...
 

 System Rules http://stefanbirkner.github.io/system-rules/ 
 
 System Rules http://stefanbirkner.github.io/system-rules/ A collection of JUnit rules for testing code that uses java.lang.System.
 
 
 
 View on stefanbirkner.github.io http://stefanbirkner.github.io/system-rules/ 
 Preview by Yahoo