Hi,

While looking at some of the test failures it occurred to me that it
would be great to have a tiny addition to the junit output for
succesfull tests. Now if a test succeeds it only prints out something
like this:

   [junit4] Suite: org.apache.solr.analysis.TestKeepFilterFactory
   [junit4] Completed on J0 in 0.22s, 1 test

If that also had a time stamp when the test started it would be, in
some cases, helpful to see what other tests were running at the same
time. I think this information is implicitly available because the
tests output when they finish but the proposed change would make it
more obvious and easier to spot.

--
 Sami Siren

---------------------------------------------------------------------

>   [junit4] Suite: org.apache.solr.analysis.TestKeepFilterFactory
>   [junit4] Completed on J0 in 0.22s, 1 test
>
> If that also had a time stamp when the test started it would be, in
> some cases, helpful to see what other tests were running at the same

The event log contains this information, it's the console report that
doesn't print it.
Lack of space is the primary reason but so is the question how
accurate the timestamp should be and what format
should it be in...

This is also related to another issue that's on the list -- hung tests
https://github.com/carrotsearch/randomizedtesting/issues/106

There should be some sort of heartbeat to report what's going on. I'll
be thinking how to solve this once we
publish a release (of Carrot2 and our internal stuff).

Dawid

---------------------------------------------------------------------

On Wed, Jun 13, 2012 at 10:45 AM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>>   [junit4] Suite: org.apache.solr.analysis.TestKeepFilterFactory
>>   [junit4] Completed on J0 in 0.22s, 1 test
>>
>> If that also had a time stamp when the test started it would be, in
>> some cases, helpful to see what other tests were running at the same
>
> The event log contains this information, it's the console report that
> doesn't print it.
> Lack of space is the primary reason but so is the question how
> accurate the timestamp should be and what format
> should it be in...

For my purposes the actual format (and after thinking about this
little more: the output in succesful test run) is less important than
seeing what were the tests running while a failing test was running. I
see that there's now output like this:

NOTE: All tests run in this JVM: [TestArabicFilters,
SoftAutoCommitTest, TestPatternTokenizerFactory, NoCacheHeaderTest,
SpellingQueryConverterTest,
TestDictionaryCompoundWordTokenFilterFactory,
TestJapaneseKatakanaStemFilterFactory, TestXIncludeConfig,
TestNorwegianLightStemFilterFactory, TestLRUCache,
TestStopFilterFactory, UniqFieldsUpdateProcessorFactoryTest,
TestPerFieldSimilarity, MBeansHandlerTest, TestLFUCache,
CopyFieldTest, TestPluginEnable, TestGroupingSearch, SuggesterTest]

So how about if something similar was printed from different aspect:

NOTE: All tests that were running at the same time: [TestA, TestB, TestC]

--
 Sami Siren

---------------------------------------------------------------------

> NOTE: All tests run in this JVM: [TestArabicFilters,
> SoftAutoCommitTest, TestPatternTokenizerFactory, NoCacheHeaderTest,
> SpellingQueryConverterTest,

I think (didn't check) this is printed from within a single JVM (a
rule in LuceneTestCase, actually), not by the framework.

> So how about if something similar was printed from different aspect:
>
> NOTE: All tests that were running at the same time: [TestA, TestB, TestC]

I think this would be noisy -- if you have multiple jvms it's normal
that they execute concurrently so there'd be a lot of overlap. I admit
it crossed my mind at some point to add a "gantt-like" chart of tests
execution flow over forked jvms to the html5 report. This is doable
and would be probably clearer to investigate compared to textual
output. Something for version 2.0 :)

Dawid

---------------------------------------------------------------------

On Wed, Jun 13, 2012 at 11:41 AM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> So how about if something similar was printed from different aspect:
>>
>> NOTE: All tests that were running at the same time: [TestA, TestB, TestC]
>
> I think this would be noisy -- if you have multiple jvms it's normal
> that they execute concurrently so there'd be a lot of overlap.

This would be visible only for failed tests (no point in printing it
for anything else)

> I admit
> it crossed my mind at some point to add a "gantt-like" chart of tests
> execution flow over forked jvms to the html5 report. This is doable
> and would be probably clearer to investigate compared to textual
> output. Something for version 2.0 :)

cool! when is it coming out? :)

--
 Sami Siren

---------------------------------------------------------------------

> This would be visible only for failed tests (no point in printing it
> for anything else)

Yes, but it'd require higher-level analysis (cross-jvm). Another
element of difficulty is that I also had a plan to distribute suites
physically to different machines at some point (and the overlap
wouldn't make sense then; nor would make timestamps...).

> cool! when is it coming out? :)

As soon as all the issues are solved. Feel free to contribute ;)

Dawid

---------------------------------------------------------------------

In my opinion, the separate JVMs should not produce test failures or affect each other, because every JVM gets its own temporary directory for running tests and creating indexes.

Lucene respects this completely, so running 2 test suites in parallel acts like running them on different machines or different users. The only problem is a few Solr tests, that don’t use the correct temporary directory and write to solr/core/src/test (maybe that’s already fixed).

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Dawid Weiss
> Sent: Wednesday, June 13, 2012 11:11 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Possible test framework improvement
>
> > This would be visible only for failed tests (no point in printing it
> > for anything else)
>
> Yes, but it'd require higher-level analysis (cross-jvm). Another element of
> difficulty is that I also had a plan to distribute suites physically to different
> machines at some point (and the overlap wouldn't make sense then; nor would
> make timestamps...).
>
> > cool! when is it coming out? :)
>
> As soon as all the issues are solved. Feel free to contribute ;)
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

I don't think any tests are writing to the source tree anymore: see
https://issues.apache.org/jira/browse/SOLR-3112,
https://issues.apache.org/jira/browse/SOLR-3268
Jenkins runs 'svn status' after all tests execute and fails the build
if its not clean.

So I think the only likely possibilities for test interference across
JVMS are things like using same temp directory outside of the source
tree or under some svn-ignore'd part of the source tree, or using the
same network port numbers, etc.

On Wed, Jun 13, 2012 at 5:16 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> In my opinion, the separate JVMs should not produce test failures or affect each other, because every JVM gets its own temporary directory for running tests and creating indexes.
>
> Lucene respects this completely, so running 2 test suites in parallel acts like running them on different machines or different users. The only problem is a few Solr tests, that don’t use the correct temporary directory and write to solr/core/src/test (maybe that’s already fixed).
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
>> Dawid Weiss
>> Sent: Wednesday, June 13, 2012 11:11 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Possible test framework improvement
>>
>> > This would be visible only for failed tests (no point in printing it
>> > for anything else)
>>
>> Yes, but it'd require higher-level analysis (cross-jvm). Another element of
>> difficulty is that I also had a plan to distribute suites physically to different
>> machines at some point (and the overlap wouldn't make sense then; nor would
>> make timestamps...).
>>
>> > cool! when is it coming out? :)
>>
>> As soon as all the issues are solved. Feel free to contribute ;)
>>
>> Dawid
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
>> commands, e-mail: [EMAIL PROTECTED]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

--
lucidimagination.com

---------------------------------------------------------------------

> So I think the only likely possibilities for test interference across
> JVMS are things like using same temp directory outside of the source
> tree or under some svn-ignore'd part of the source tree, or using the
> same network port numbers, etc.

Temp directories shouldn't be a problem if they're acquired properly
(cwd is different for each jvm, temp file creation should be atomic?).

D.

---------------------------------------------------------------------

On Wed, Jun 13, 2012 at 10:13 AM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> So I think the only likely possibilities for test interference across
>> JVMS are things like using same temp directory outside of the source
>> tree or under some svn-ignore'd part of the source tree, or using the
>> same network port numbers, etc.
>
> Temp directories shouldn't be a problem if they're acquired properly
> (cwd is different for each jvm, temp file creation should be atomic?).
>

Right, i just meant in the theoretical sense: for example, if
foo-dependency-1.0.jar does stuff with java.io.tmpdir it could create
an issue correct? or does the test runner also set this sysprop to the
"sandbox" dir for slave jvms (if it doesn't already but can, that
would be very cool, more safety).

--
lucidimagination.com

---------------------------------------------------------------------

> foo-dependency-1.0.jar does stuff with java.io.tmpdir it could create
> an issue correct? or does the test runner also set this sysprop to the
> "sandbox" dir for slave jvms (if it doesn't already but can, that
> would be very cool, more safety).

True, we could alter java.io.tmpdir. There is actually a snippet of
code in randomized testing that creates new temporary directories and
files on demand in a way that their names contain creation timestamps
(simplifies connecting temporary files to tests which created them...)
but I didn't want to change everything at once to make the learning
curve easier.

I agree java.io.tmpdir could be redirected though, it makes sense.

Dawid

---------------------------------------------------------------------

: In my opinion, the separate JVMs should not produce test failures or
: affect each other, because every JVM gets its own temporary directory
: for running tests and creating indexes.

I don't think anyone would disagree with that opinion -- but having a
common opinion doesn't magically make it a fact.

The entire point of sami's suggestion is that if a test fails, it would be
nice to have the info needed to determine what other tests were running at
the same time, regardless of the JVM, so you can investigate wether
something in those tests is the cause of the problem and fix them.

ie: tests in seperate JVMs shouldn't affect eachother, but in spite of
best intentions they might, and having timestamps would help discover
that.
-Hoss

---------------------------------------------------------------------

> ie: tests in seperate JVMs shouldn't affect eachother, but in spite of
> best intentions they might, and having timestamps would help discover
> that.

They do -- look at the html report, it contains timestamps. I'll add
them to the txt report in the next version.

Dawid

---------------------------------------------------------------------

hmm, actually the html report (and generate-test-reports) displays
this information nicely so i don't know if this is necessary.

On Wed, Jun 13, 2012 at 9:55 AM, Sami Siren <[EMAIL PROTECTED]> wrote:
> Hi,
>
> While looking at some of the test failures it occurred to me that it
> would be great to have a tiny addition to the junit output for
> succesfull tests. Now if a test succeeds it only prints out something
> like this:
>
>   [junit4] Suite: org.apache.solr.analysis.TestKeepFilterFactory
>   [junit4] Completed on J0 in 0.22s, 1 test
>
> If that also had a time stamp when the test started it would be, in
> some cases, helpful to see what other tests were running at the same
> time. I think this information is implicitly available because the
> tests output when they finish but the proposed change would make it
> more obvious and easier to spot.
>
> --
>  Sami Siren

---------------------------------------------------------------------