Ah... such a relief to finally have it committed even if this means
the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene
and make it available as a separate project for reuse in other
projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for
randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
with an aggregation of events, balancing, logs, different (from
standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on
github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
RandomizedRunner. It also modifies build scripts to run junit4-ant
instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is
gone. This change was motivated by the fact that the field's value was
read from places where it shouldn't be read, passed to places where it
shouldn't be passed, etc. Instead, the Random instance for a given
scope (see below) can be acquired from a static method in
LuceneTestCase called random(). In the essence, you can just add
brackets around your previous random field references and it _should_
work out of the box. There are differences though: Random object
returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random
instance in a static scope (@BeforeClass) to a field and reusing it in
a test, passing a Random instance from one thread to another, saving a
Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the
scope (test method, hook) it was acquired for. You _can_ call random()
from non-test threads -- they will be given their own thread-local
Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master
seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed
to 'tests.iters' (you'll get a build failure and a message if you try
to use the former one). I could add a fallback but I'd rather not
introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a
duplicate of a given test (or tests), including any @Before/@After
hooks and setups. In theory, this means it is now possible to
reiterate ANY test, no matter how complex. If it doesn't depend on
static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3
-Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in
TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each
executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three
times, but they all count individually (so the reported total is 9
tests). What's also clearly seen is that the master seed is constant
for all tests but each repetition gets a (predictable) derivative of
the random seed. This way you can reiterate N times over a test, each
time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3
-Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs

Hi Dawid,

I've been running the new test setup, so far without any problems - nice! - and I noticed a couple of things:

- tests.threadspercpu seems to behave like a maximum JVM count rather a core multiplier - is that right?  If so, maybe this param should be renamed?

- How does "ant -verbose" differ from/replace "ant -Dtests.verbose" ?  Are these complementary?

Thanks for all your work on this!

Would you like me to take a crack at modifying the RunningTests wiki page?

Steve

-----Original Message-----
From: Dawid Weiss [mailto:[EMAIL PROTECTED]]
Sent: Sunday, April 15, 2012 10:45 AM
To: [EMAIL PROTECTED]
Subject: [Important] New test framework, differences, features, etc.

Ah... such a relief to finally have it committed even if this means the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene and make it available as a separate project for reuse in other projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an aggregation of events, balancing, logs, different (from standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with RandomizedRunner. It also modifies build scripts to run junit4-ant instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is gone. This change was motivated by the fact that the field's value was read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc. Instead, the Random instance for a given scope (see below) can be acquired from a static method in LuceneTestCase called random(). In the essence, you can just add brackets around your previous random field references and it _should_ work out of the box. There are differences though: Random object returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random instance in a static scope (@BeforeClass) to a field and reusing it in a test, passing a Random instance from one thread to another, saving a Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the scope (test method, hook) it was acquired for. You _can_ call random() from non-test threads -- they will be given their own thread-local Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed to 'tests.iters' (you'll get a build failure and a message if you try to use the former one). I could add a fallback but I'd rather not introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a given test (or tests), including any @Before/@After hooks and setups. In theory, this means it is now possible to reiterate ANY test, no matter how complex. If it doesn't depend on static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three times, but they all count individually (so the reported total is 9 tests). What's also clearly seen is that the master seed is constant for all tests but each repetition gets a (predictable) derivative of the random seed. This way you can reiterate N times over a test, each time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.

Hi Steve,

> - tests.threadspercpu seems to behave like a maximum JVM count rather a core multiplier - is that right?  If so, maybe this param should be renamed?

You're right, I don't know how it came to be threadspercpu... it is
actually the number of forked JVMs -- it should be "tests.jvms"
(sounds good?). There is no reliable way to tell if a cpu is running
in hyperthreading mode or with real threads (multicpu motherboards,
for example) and from my experiments it seems that going into
hyperthreading does not speed up things (possibly because Java has its
own concurrent threads anyway so everything's saturated already).

I'll rename this attribute -- thanks for spotting this.

> - How does "ant -verbose" differ from/replace "ant -Dtests.verbose" ? ��Are these complementary?

"ant -verbose" is a global switch to run ant's logger in verbose mode.
So you'll get tons of logs from tasks other than junit4 runner, but
including it. I just reused ant's infrastructure here. -Dtests.verbose
is project (lucene/solr) specific.

> Would you like me to take a crack at modifying the RunningTests wiki page?

Oh, this would be awesome if you could. I am wiki challenged...

Dawid

---------------------------------------------------------------------

On 4/15/2012 at 1:35 PM, Dawid Weiss wrote:
> > - tests.threadspercpu seems to behave like a maximum JVM
> > count rather a core multiplier - is that right?  If so,
> > maybe this param should be renamed?
>
> You're right, I don't know how it came to be threadspercpu...
> it is actually the number of forked JVMs -- it should be
> "tests.jvms" (sounds good?).

+1

> There is no reliable way to tell if a cpu is running in
> hyperthreading mode or with real threads (multicpu
> motherboards, for example) and from my experiments it seems
> that going into hyperthreading does not speed up things
> (possibly because Java has its own concurrent threads anyway
> so everything's saturated already).

On my machine (Win7 64-bit, 4 real cores + Hyperthreading -> 8 "cores"), leaving tests.threadspercu at the default "auto" maxes out at 4 JVMs.  

Running all tests from the top level, the sweet spot for me seems to be tests.threadspercpu=5 - this is about 8-10% faster than "auto".  tests.threadspercpu=6 is also faster than "auto", but only by about 3-4%.  (I have only done a couple trials, and this is on a workstation where I'm doing other stuff...)

> > Would you like me to take a crack at modifying the RunningTests wiki page?
>
> Oh, this would be awesome if you could. I am wiki challenged...

OK, I'll work on it later today.

Steve

>> "tests.jvms" (sounds good?).
>
> +1

It'll be in in the next batch of changes.

> On my machine (Win7 64-bit, 4 real cores + Hyperthreading -> 8 "cores"), leaving tests.threadspercu at the default "auto" maxes out at 4 JVMs.

I have a similar machine. I was afraid to overspecify this tuning
though. Like if you want to do anything else in the background it does
make a difference. I guess if somebody wishes to commit more resoures
they can tune it manually?

> OK, I'll work on it later today.

Thanks.

Dawid

---------------------------------------------------------------------

On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:

> I have a similar machine. I was afraid to overspecify this tuning
> though. Like if you want to do anything else in the background it does
> make a difference. I guess if somebody wishes to commit more resoures
> they can tune it manually?

+1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

The heuristic is here if anybody wants to contribute their ideas.

https://github.com/carrotsearch/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L850-878

The maximum for "auto" mode is 4 jvms if somebody has >= 8 logical
processors. I didn't want to increase it because folks like Mike (24
cores if I remember) would quickly run out of memory with so many jvms
forked. It would be a nice touch to try to sense -Xmx options passed
to the forked JVMs and pick the number of forks (or warn the user
about memory limit) based on that but I haven't done it yet.

Dawid

On Mon, Apr 16, 2012 at 4:42 AM, Mark Miller <[EMAIL PROTECTED]> wrote:
>
> On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:
>
>> I have a similar machine. I was afraid to overspecify this tuning
>> though. Like if you want to do anything else in the background it does
>> make a difference. I guess if somebody wishes to commit more resoures
>> they can tune it manually?
>
> +1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

I ran 5 trials on my machine (Win7/64, 4 cores+HT->8 "cores") of testing under lucene/ (including the contribs there) without compilation, setting tests.jvms/threadspercpu to auto (4), 5, 6, and 7:

|| jvms || min || max || avg ||
|     4  |  92s | 109s |  97s |
|     5  |  80s |  96s |  92s |
|     6  |  90s |  98s |  98s |
|     7  |  91s | 110s | 100s |

So the benefit of overriding "auto" is barely noticeable, in both relative and absolute terms.

Steve

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 2:56 AM
To: [EMAIL PROTECTED]
Subject: Re: [Important] New test framework, differences, features, etc.

The heuristic is here if anybody wants to contribute their ideas.

https://github.com/carrotsearch/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L850-878

The maximum for "auto" mode is 4 jvms if somebody has >= 8 logical processors. I didn't want to increase it because folks like Mike (24 cores if I remember) would quickly run out of memory with so many jvms forked. It would be a nice touch to try to sense -Xmx options passed to the forked JVMs and pick the number of forks (or warn the user about memory limit) based on that but I haven't done it yet.

Dawid

On Mon, Apr 16, 2012 at 4:42 AM, Mark Miller <[EMAIL PROTECTED]> wrote:
>
> On Apr 15, 2012, at 3:04 PM, Dawid Weiss wrote:
>
>> I have a similar machine. I was afraid to overspecify this tuning
>> though. Like if you want to do anything else in the background it
>> does make a difference. I guess if somebody wishes to commit more
>> resoures they can tune it manually?
>
> +1 - out of the box should not maximize resource usage IMO. I'm always doing other things while running tests - it should be possible to max out your sys, but I think defaults should be conservative.
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For
> additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

>        || jvms || min || max || avg ||
>        |     4  |  92s | 109s |  97s |
>        |     5  |  80s |  96s |  92s |
>        |     6  |  90s |  98s |  98s |
>        |     7  |  91s | 110s | 100s |
>
> So the benefit of overriding "auto" is barely noticeable, in both relative and absolute terms.

If you go "down" from 4 you'll see the difference real cores make -- 4
is pretty much saturated I7 (I have the same). If somebody has more
than 4 physical cores they will see a difference (I think, I don't
have access to such a machine). Memory bandwidth also comes into play
here so it won't scale perfectly I bet.

Dawid

---------------------------------------------------------------------

Great stuff Dawid!
The tests.iters functionality sounds like it will be particularly useful.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

On Sun, Apr 15, 2012 at 10:45 AM, Dawid Weiss <[EMAIL PROTECTED]> wrote:
> Ah... such a relief to finally have it committed even if this means
> the work has just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene
> and make it available as a separate project for reuse in other
> projects. This kind of grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for
> randomization and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
> with an aggregation of events, balancing, logs, different (from
> standard ANT) reporting and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on
> github here: https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant
> instead of ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is
> gone. This change was motivated by the fact that the field's value was
> read from places where it shouldn't be read, passed to places where it
> shouldn't be passed, etc. Instead, the Random instance for a given
> scope (see below) can be acquired from a static method in
> LuceneTestCase called random(). In the essence, you can just add
> brackets around your previous random field references and it _should_
> work out of the box. There are differences though: Random object
> returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random
> instance in a static scope (@BeforeClass) to a field and reusing it in
> a test, passing a Random instance from one thread to another, saving a
> Random instance to a field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the
> scope (test method, hook) it was acquired for. You _can_ call random()
> from non-test threads -- they will be given their own thread-local
> Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master
> seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed
> to 'tests.iters' (you'll get a build failure and a message if you try
> to use the former one). I could add a fallback but I'd rather not
> introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a
> duplicate of a given test (or tests), including any @Before/@After
> hooks and setups. In theory, this means it is now possible to
> reiterate ANY test, no matter how complex. If it doesn't depend on
> static state, it can be repeated. This also links to how seed is used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3
> -Dtestcase=TestSubScorerFreqs test-core
>
> the above will result in 3 executions of every test in
> TestSubScorerFreqs. What you'll get on the console is:
>
>   [junit4] <JUnit4> says hello. Random seed: deadbeef
>   [junit4] Expected execution time on JVM J0:     0.02s
>   [junit4] Executing 1 suite with 1 JVM.
>   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
> Time:  0.45s
>   [junit4]
>   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>   [junit4] Execution time total: 0.98 sec.
>   [junit4] Tests summary: 1 suite, 9 tests

> Great stuff Dawid!

Thanks! This stuff is based on concepts that were already present in
Lucene so I can't take the credit for it, but I admit I really like it
being a separate project. We've been using it for some time in our own
codebase and it's worked great. In particular tests isolation, groups
and parameterized factories...

> The tests.iters functionality sounds like it will be particularly useful.

...which are the other goodies that I think will be useful as well.
I'll introduce them gradually so that it's not too overwhelming.

Dawid

---------------------------------------------------------------------

One thing I disagree:

[junit4] Suite: TestTopDocsMerge
[junit4] Completed in 3.73s, 1 test

Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Dawid Weiss [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, April 15, 2012 4:45 PM
> To: [EMAIL PROTECTED]
> Subject: [Important] New test framework, differences, features, etc.
>
> Ah... such a relief to finally have it committed even if this means the work has
> just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene and
> make it available as a separate project for reuse in other projects. This kind of
> grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for randomization
> and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an
> aggregation of events, balancing, logs, different (from standard ANT) reporting
> and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on github here:
> https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant instead of
> ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is gone. This
> change was motivated by the fact that the field's value was read from places
> where it shouldn't be read, passed to places where it shouldn't be passed, etc.
> Instead, the Random instance for a given scope (see below) can be acquired
> from a static method in LuceneTestCase called random(). In the essence, you
> can just add brackets around your previous random field references and it
> _should_ work out of the box. There are differences though: Random object
> returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random instance in
> a static scope (@BeforeClass) to a field and reusing it in a test, passing a
> Random instance from one thread to another, saving a Random instance to a
> field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the scope (test
> method, hook) it was acquired for. You _can_ call random() from non-test
> threads -- they will be given their own thread-local Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed to
> 'tests.iters' (you'll get a build failure and a message if you try to use the former
> one). I could add a fallback but I'd rather not introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a
> given test (or tests), including any @Before/@After hooks and setups. In theory,
> this means it is now possible to reiterate ANY test, no matter how complex. If it
> doesn't depend on static state, it can be repeated. This also links to how seed is
> used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-
> core
>
> the above will result in 3 executions of every test in TestSubScorerFreqs. What
> you'll get on the console is:
>
>    [junit4] <JUnit4> says hello. Random seed: deadbeef

I ordinarily run 80-column terminals.

But I agree with Uwe - package absence looks bad.

Maybe a compression scheme like people use would work here?  E.g. o.a.l.whatever.Class, or maybe even oal.whatever.Class ?

Also, does Java have access to the column width of the running terminal? If so, the output width would not have to have a fixed width.

Steve

-----Original Message-----
From: Uwe Schindler [mailto:[EMAIL PROTECTED]]
Sent: Monday, April 16, 2012 9:10 AM
To: [EMAIL PROTECTED]
Subject: RE: [Important] New test framework, differences, features, etc.

One thing I disagree:

[junit4] Suite: TestTopDocsMerge
[junit4] Completed in 3.73s, 1 test

Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Dawid Weiss [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, April 15, 2012 4:45 PM
> To: [EMAIL PROTECTED]
> Subject: [Important] New test framework, differences, features, etc.
>
> Ah... such a relief to finally have it committed even if this means
> the work has just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene
> and make it available as a separate project for reuse in other
> projects. This kind of grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for
> randomization and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
> with an aggregation of events, balancing, logs, different (from
> standard ANT) reporting and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on github here:
> https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant
> instead of ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is
> gone. This change was motivated by the fact that the field's value was
> read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc.
> Instead, the Random instance for a given scope (see below) can be
> acquired from a static method in LuceneTestCase called random(). In
> the essence, you can just add brackets around your previous random
> field references and it _should_ work out of the box. There are
> differences though: Random object returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random
> instance in a static scope (@BeforeClass) to a field and reusing it in
> a test, passing a Random instance from one thread to another, saving a
> Random instance to a field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the
> scope (test method, hook) it was acquired for. You _can_ call random()
> from non-test threads -- they will be given their own thread-local Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed
> to 'tests.iters' (you'll get a build failure and a message if you try
> to use the former one). I could add a fallback but I'd rather not introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a

> But I agree with Uwe - package absence looks bad.

Again -- play with these attributes to find what you like.

> Maybe a compression scheme like people use would work here?  E.g. o.a.l.whatever.Class, or maybe even oal.whatever.Class ?

I don't particularly like such compressions, they're misleading (some
obfuscators do emit o.a.l.... :).

> Also, does Java have access to the column width of the running terminal? If so, the output width would not have to have a fixed width.

I wish it had, but unfortunately the answer is no. This would make
several things easier...

Dawid

---------------------------------------------------------------------

I dont undertstand the problem with the column width. Whats the problem with line wrapping? Just print Class.getName() and done. I want full names for test names here, so its easier to work with (especially if you want to work with Jenkin's output). It was always fine here on windows, where terminal always wrap after 80 columns and I never had any problem with that. So please make the standard full class names and let maybe some build.properties allow it to be changed by rcmuir.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Dawid Weiss
> Sent: Monday, April 16, 2012 3:16 PM
> To: [EMAIL PROTECTED]
> Subject: Re: [Important] New test framework, differences, features, etc.
>
> > But I agree with Uwe - package absence looks bad.
>
> Again -- play with these attributes to find what you like.
>
> > Maybe a compression scheme like people use would work here?  E.g.
> o.a.l.whatever.Class, or maybe even oal.whatever.Class ?
>
> I don't particularly like such compressions, they're misleading (some
> obfuscators do emit o.a.l.... :).
>
> > Also, does Java have access to the column width of the running terminal? If
> so, the output width would not have to have a fixed width.
>
> I wish it had, but unfortunately the answer is no. This would make several
> things easier...
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

On Mon, Apr 16, 2012 at 9:51 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> I dont undertstand the problem with the column width. Whats the problem with line wrapping? Just print Class.getName() and done. I want full names for test names here, so its easier to work with (especially if you want to work with Jenkin's output). It was always fine here on windows, where terminal always wrap after 80 columns and I never had any problem with that. So please make the standard full class names and let maybe some build.properties allow it to be changed by rcmuir.
See the original issue: https://issues.apache.org/jira/browse/LUCENE-3988

My problem is only when 100% of tests line-wrap on 80-chars terminals.
This happened always previously because of padding in the output:
[junit4] Tests run:   4, Failures:   0, Errors:   0, Skipped:   0, Time:  3.97s

The shorter class name was just a suggestion, i dont care if we print
the full class name.
That way only like 1 or 2% of tests wrap. I just dont think it should be 100%.

--
lucidimagination.com

---------------------------------------------------------------------

I am indifferent about it, really. If there are identicl class names
in the code perhaps they should be made more specific though? This
will matter not only for the output but also for other things
(ambiguous imports etc).

Anyway, feel free to play with these attributes --

                    useSimpleNames="true"
                    maxClassNameColumns="70"

if you change useSimpleNames to 'false' and keep the limit there will
be an ellipsis before max number of columns is reached.

Dawid

On Mon, Apr 16, 2012 at 3:09 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> One thing I disagree:
>
> [junit4] Suite: TestTopDocsMerge
> [junit4] Completed in 3.73s, 1 test
>
> Please print the package name again, as this makes it hard to quickly lookup the test, especially if there are multiple instanges of same class name. The 80 column limit is somehow last century :-)
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
>> -----Original Message-----
>> From: Dawid Weiss [mailto:[EMAIL PROTECTED]]
>> Sent: Sunday, April 15, 2012 4:45 PM
>> To: [EMAIL PROTECTED]
>> Subject: [Important] New test framework, differences, features, etc.
>>
>> Ah... such a relief to finally have it committed even if this means the work has
>> just begun on straightening it all out. So.
>>
>> 1) WHAT'S BEEN DONE?
>>
>> The primary objective was to pull out "randomized" runner from Lucene and
>> make it available as a separate project for reuse in other projects. This kind of
>> grew into two things:
>>
>> a) RandomizedRunner - a runner that has built-in support for randomization
>> and other interesting features,
>> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an
>> aggregation of events, balancing, logs, different (from standard ANT) reporting
>> and forked JVM crash resilience.
>>
>> Everything above is heavily covered with unit tests (the code is on github here:
>> https://github.com/carrotsearch/randomizedtesting).
>>
>> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
>> RandomizedRunner. It also modifies build scripts to run junit4-ant instead of
>> ANT's default <junit>.
>>
>> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>>
>> - The most visible change is that 'random' field in LuceneTestCase is gone. This
>> change was motivated by the fact that the field's value was read from places
>> where it shouldn't be read, passed to places where it shouldn't be passed, etc.
>> Instead, the Random instance for a given scope (see below) can be acquired
>> from a static method in LuceneTestCase called random(). In the essence, you
>> can just add brackets around your previous random field references and it
>> _should_ work out of the box. There are differences though: Random object
>> returned by random() is valid only for the scope it was created for.
>> So any of the following will end up in an exception: saving a Random instance in
>> a static scope (@BeforeClass) to a field and reusing it in a test, passing a
>> Random instance from one thread to another, saving a Random instance to a
>> field in one test, using it in another, etc. In
>> short: the result of random() is per-thread and only valid for the scope (test
>> method, hook) it was acquired for. You _can_ call random() from non-test
>> threads -- they will be given their own thread-local Random instances.
>>
>> - The 'random seed' is a single HEX-encoded long. The "three seeds"
>> from before are gone. Everything is a derivative of the initial master seed.
>>
>> - I provided a 'help on syntax' for test properties. Type:
>>
>> ant test-help
>>
>> and the most common use case scenarios will be dumped to your console.
>>
>> - A notable difference is that 'tests.iter' property has been renamed to
>> 'tests.iters' (you'll get a build failure and a message if you try to use the former
>> one). I could add a fallback but I'd rather not introduce any more aliases.
>>
>> - "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a

On 4/16/2012 at 9:14 AM, Dawid Weiss wrote:
> If there are identicl class names in the code perhaps
> they should be made more specific though? This will
> matter not only for the output but also for other things
> (ambiguous imports etc).

Here's a list of java files with duplicate class names (across both Solr and Lucene):

lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/BreakIteratorBoundaryScanner.java
solr/core/src/java/org/apache/solr/highlight/BreakIteratorBoundaryScanner.java

modules/analysis/common/src/java/org/apache/lucene/analysis/util/CharArrayIterator.java
modules/analysis/icu/src/java/org/apache/lucene/analysis/icu/segmentation/CharArrayIterator.java

lucene/core/src/java/org/apache/lucene/analysis/CharStream.java
modules/queryparser/src/java/org/apache/lucene/queryparser/classic/CharStream.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/CharStream.java

modules/benchmark/src/java/org/apache/lucene/benchmark/byTask/utils/Config.java
solr/core/src/java/org/apache/solr/core/Config.java

lucene/core/src/java/org/apache/lucene/util/Constants.java
modules/benchmark/src/java/org/apache/lucene/benchmark/Constants.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/DefaultEncoder.java
solr/core/src/java/org/apache/solr/highlight/DefaultEncoder.java

modules/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/dict/Dictionary.java
modules/suggest/src/java/org/apache/lucene/search/spell/Dictionary.java

lucene/core/src/java/org/apache/lucene/document/DoubleField.java
solr/core/src/java/org/apache/solr/schema/DoubleField.java

modules/analysis/common/src/java/org/apache/lucene/analysis/nl/DutchStemmer.java
modules/analysis/common/src/java/org/tartarus/snowball/ext/DutchStemmer.java

modules/queryparser/src/java/org/apache/lucene/queryparser/classic/FastCharStream.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/FastCharStream.java

lucene/core/src/java/org/apache/lucene/document/Field.java
solr/solrj/src/java/org/apache/solr/client/solrj/beans/Field.java

lucene/core/src/java/org/apache/lucene/document/FieldType.java
solr/core/src/java/org/apache/solr/schema/FieldType.java

modules/benchmark/src/java/org/apache/lucene/benchmark/byTask/utils/FileUtils.java
solr/core/src/java/org/apache/solr/util/FileUtils.java

lucene/core/src/java/org/apache/lucene/document/FloatField.java
solr/core/src/java/org/apache/solr/schema/FloatField.java

modules/analysis/common/src/java/org/apache/lucene/analysis/fr/FrenchStemmer.java
modules/analysis/common/src/java/org/tartarus/snowball/ext/FrenchStemmer.java

modules/analysis/common/src/java/org/apache/lucene/analysis/de/GermanStemmer.java
modules/analysis/common/src/java/org/tartarus/snowball/ext/GermanStemmer.java

modules/benchmark/src/java/org/apache/lucene/benchmark/byTask/feeds/HTMLParser.java
modules/benchmark/src/java/org/apache/lucene/benchmark/byTask/feeds/demohtml/HTMLParser.java

lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java
solr/core/src/test/org/apache/solr/highlight/HighlighterTest.java

lucene/core/src/java/org/apache/lucene/document/IntField.java
solr/core/src/java/org/apache/solr/schema/IntField.java

lucene/core/src/java/org/apache/lucene/document/LongField.java
solr/core/src/java/org/apache/solr/schema/LongField.java

modules/benchmark/src/java/org/apache/lucene/benchmark/byTask/feeds/demohtml/ParseException.java
modules/queryparser/src/java/org/apache/lucene/queryparser/classic/ParseException.java
modules/queryparser/src/java/org/apache/lucene/queryparser/flexible/standard/parser/ParseException.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/ParseException.java

lucene/core/src/test/org/apache/lucene/search/payloads/PayloadHelper.java
modules/analysis/common/src/java/org/apache/lucene/analysis/payloads/PayloadHelper.java

modules/analysis/common/src/java/org/apache/lucene/analysis/en/PorterStemmer.java
modules/analysis/common/src/java/org/tartarus/snowball/ext/PorterStemmer.java

modules/analysis/common/src/java/org/apache/lucene/analysis/pt/PortugueseStemmer.java
modules/analysis/common/src/java/org/tartarus/snowball/ext/PortugueseStemmer.java

modules/queryparser/src/java/org/apache/lucene/queryparser/flexible/core/builders/QueryBuilder.java
modules/queryparser/src/java/org/apache/lucene/queryparser/xml/QueryBuilder.java

modules/queryparser/src/java/org/apache/lucene/queryparser/classic/QueryParser.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/QueryParser.java

modules/queryparser/src/java/org/apache/lucene/queryparser/classic/QueryParserConstants.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/QueryParserConstants.java

modules/queryparser/src/java/org/apache/lucene/queryparser/classic/QueryParserTokenManager.java
modules/queryparser/src/java/org/apache/lucene/queryparser/surround/parser/QueryParserTokenManager.java

lucene/test-framework/src/java/org/apache/lucene/search/QueryUtils.java
solr/core/src/java/org/apache/solr/search/QueryUtils.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/ScoreOrderFragmentsBuilder.java
solr/core/src/java/org/apache/solr/highlight/ScoreOrderFragmentsBuilder.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/highlight/Scorer.java
lucene/core/src/java/org/apache/lucene/search/Scorer.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleBoundaryScanner.java
solr/core/src/java/org/apache/solr/highlight/SimpleBoundaryScanner.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleFragListBuilder.java
solr/core/src/java/org/apache/solr/highlight/SimpleFragListBuilder.java

lucene/contrib/highlighter/src/java/org/apache/lucene/search/vectorhighlight/SimpleFragmentsBuilder.java
solr/core/src/java/org/apache/solr/highlight/Simple

Hi Dawid,

-Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Steve

-----Original Message-----
From: Dawid Weiss [mailto:[EMAIL PROTECTED]]
Sent: Sunday, April 15, 2012 10:45 AM
To: [EMAIL PROTECTED]
Subject: [Important] New test framework, differences, features, etc.

Ah... such a relief to finally have it committed even if this means the work has just begun on straightening it all out. So.

1) WHAT'S BEEN DONE?

The primary objective was to pull out "randomized" runner from Lucene and make it available as a separate project for reuse in other projects. This kind of grew into two things:

a) RandomizedRunner - a runner that has built-in support for randomization and other interesting features,
b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs, with an aggregation of events, balancing, logs, different (from standard ANT) reporting and forked JVM crash resilience.

Everything above is heavily covered with unit tests (the code is on github here: https://github.com/carrotsearch/randomizedtesting).

LUCENE-3808 removes LuceneTestCaseRunner, replacing it with RandomizedRunner. It also modifies build scripts to run junit4-ant instead of ANT's default <junit>.

2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?

- The most visible change is that 'random' field in LuceneTestCase is gone. This change was motivated by the fact that the field's value was read from places where it shouldn't be read, passed to places where it shouldn't be passed, etc. Instead, the Random instance for a given scope (see below) can be acquired from a static method in LuceneTestCase called random(). In the essence, you can just add brackets around your previous random field references and it _should_ work out of the box. There are differences though: Random object returned by random() is valid only for the scope it was created for.
So any of the following will end up in an exception: saving a Random instance in a static scope (@BeforeClass) to a field and reusing it in a test, passing a Random instance from one thread to another, saving a Random instance to a field in one test, using it in another, etc. In
short: the result of random() is per-thread and only valid for the scope (test method, hook) it was acquired for. You _can_ call random() from non-test threads -- they will be given their own thread-local Random instances.

- The 'random seed' is a single HEX-encoded long. The "three seeds"
from before are gone. Everything is a derivative of the initial master seed.

- I provided a 'help on syntax' for test properties. Type:

ant test-help

and the most common use case scenarios will be dumped to your console.

- A notable difference is that 'tests.iter' property has been renamed to 'tests.iters' (you'll get a build failure and a message if you try to use the former one). I could add a fallback but I'd rather not introduce any more aliases.

- "tests.iters" is no longer a poor-man's loop. It really re-runs a duplicate of a given test (or tests), including any @Before/@After hooks and setups. In theory, this means it is now possible to reiterate ANY test, no matter how complex. If it doesn't depend on static state, it can be repeated. This also links to how seed is used.
It's probably best explained on an example:

ant -Dtests.seed=deadbeef -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

the above will result in 3 executions of every test in TestSubScorerFreqs. What you'll get on the console is:

   [junit4] <JUnit4> says hello. Random seed: deadbeef
   [junit4] Expected execution time on JVM J0:     0.02s
   [junit4] Executing 1 suite with 1 JVM.
   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
Time:  0.45s
   [junit4]
   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
   [junit4] Execution time total: 0.98 sec.
   [junit4] Tests summary: 1 suite, 9 tests

and if you peek at the log file with test results
(build/core/test/tests-report.txt) you'll see the details of each executed test:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0
seed=[DEADBEEF:BAE943E3CC27A0F]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1
seed=[DEADBEEF:BFF828C20800B123]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2
seed=[DEADBEEF:3111BE1E59C4F9E8]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:3DDFB93BCC712A41]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#1
seed=[DEADBEEF:898905C7F8B3E16D]}
OK      0.01s | TestSubScorerFreqs.testBooleanQuery {#2
seed=[DEADBEEF:760931BA977A9A6]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#0
seed=[DEADBEEF:A7ADE8DE1DB7CA0B]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#1
seed=[DEADBEEF:13FB542229750127]}
OK      0.01s | TestSubScorerFreqs.testPhraseQuery {#2
seed=[DEADBEEF:9D12C2FE78B149EC]}
Tests run:   9, Failures:   0, Errors:   0, Skipped:   0, Time:  0.45s

Note that tests.iters=3 resulted in every test case executed three times, but they all count individually (so the reported total is 9 tests). What's also clearly seen is that the master seed is constant for all tests but each repetition gets a (predictable) derivative of the random seed. This way you can reiterate N times over a test, each time with a different seed. Now, compare this to:

ant -Dtests.seed=deadbeef:cafebabe -Dtests.iters=3 -Dtestcase=TestSubScorerFreqs test-core

The log file now will be:

Executing 1 suite with 1 JVM.
Running org.apache.lucene.search.TestSubScorerFreqs
OK      0.04s | TestSubScorerFreqs.testTermQuery {#0 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#1 seed=[DEADBEEF:CAFEBABE]}
OK      0.01s | TestSubScorerFreqs.testTermQuery {#2 seed=[DEADBEEF:CAFEBABE]}
OK      0.02s | TestSubScorerFreqs.testBooleanQuery {#0
seed=[DEADBEEF:CAFEBABE

> -Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Will this work for you (ant test-help):

     [echo] # Run all tests in a package and sub-packages
     [echo] ant test "-Dtests.class=org.apache.lucene.package.*"

This is a globbing pattern so you can also do:

ant test "-Dtests.class=*.packagename.*"

We could simulate previous options but globbing seems more versatile?
If you really need it then we can re-add it, sure.

Dawid

---------------------------------------------------------------------

I don't really need it - I was just asking about the absence of an alias for a previously advertised feature.  If you think it best not to continue to support that, it's fine with me.  (I'm updating the RunningTests wiki now, where this feature is described.)

Steve

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 9:03 AM
To: [EMAIL PROTECTED]
Subject: Re: [Important] New test framework, differences, features, etc.

> -Dtestpackage doesn't work anymore - is there some reason you didn't provide an alias, like you did for -Dtestmethod and -Dtestcase?

Will this work for you (ant test-help):

     [echo] # Run all tests in a package and sub-packages
     [echo] ant test "-Dtests.class=org.apache.lucene.package.*"

This is a globbing pattern so you can also do:

ant test "-Dtests.class=*.packagename.*"

We could simulate previous options but globbing seems more versatile?
If you really need it then we can re-add it, sure.

Dawid

---------------------------------------------------------------------
---------------------------------------------------------------------

> I don't really need it - I was just asking about the absence of an alias for a previously advertised feature.  If you think it best not to continue to support that, it's fine with me.  (I'm updating the RunningTests wiki now, where this feature is described.)

Oh, ok. I'd rather remove package and packageroot and replace it with
an example of globbing?

Dawid

---------------------------------------------------------------------

On 16/04/2012 15:08, Dawid Weiss wrote:
>> I don't really need it - I was just asking about the absence of an alias for a previously advertised feature.  If you think it best not to continue to support that, it's fine with me.  (I'm updating the RunningTests wiki now, where this feature is described.)
>
> Oh, ok. I'd rather remove package and packageroot and replace it with
> an example of globbing?

If you use a shell to invoke ant (I guess most of us do ;) ) then you
will need to escape or quote the globbing patterns, otherwise they will
be interpreted by shell, which could lead to surprises if there are
matching files other that tests...

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com
---------------------------------------------------------------------

> If you use a shell to invoke ant (I guess most of us do ;) ) then you will
> need to escape or quote the globbing patterns, otherwise they will be
> interpreted by shell, which could lead to surprises if there are matching
> files other that tests...

Yes, you need to be careful about it. I encourage you to use 'ant
test-help' frequently -- the examples there come with quotes and it's
not a coincidence :)

Dawid

---------------------------------------------------------------------

Okay, that's what I did on the RunningTests wiki page.

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Dawid Weiss
Sent: Monday, April 16, 2012 9:09 AM
To: [EMAIL PROTECTED]
Subject: Re: [Important] New test framework, differences, features, etc.

> I don't really need it - I was just asking about the absence of an
> alias for a previously advertised feature.  If you think it best not
> to continue to support that, it's fine with me.  (I'm updating the
> RunningTests wiki now, where this feature is described.)

Oh, ok. I'd rather remove package and packageroot and replace it with an example of globbing?

Dawid

---------------------------------------------------------------------

Thanks Dawid. I love it when my Junit4 says hello.

On Apr 15, 2012, at 10:45 AM, Dawid Weiss wrote:

> Ah... such a relief to finally have it committed even if this means
> the work has just begun on straightening it all out. So.
>
> 1) WHAT'S BEEN DONE?
>
> The primary objective was to pull out "randomized" runner from Lucene
> and make it available as a separate project for reuse in other
> projects. This kind of grew into two things:
>
> a) RandomizedRunner - a runner that has built-in support for
> randomization and other interesting features,
> b) junit4-ant - an ANT task for running JUnit4 tests in parallel JVMs,
> with an aggregation of events, balancing, logs, different (from
> standard ANT) reporting and forked JVM crash resilience.
>
> Everything above is heavily covered with unit tests (the code is on
> github here: https://github.com/carrotsearch/randomizedtesting).
>
> LUCENE-3808 removes LuceneTestCaseRunner, replacing it with
> RandomizedRunner. It also modifies build scripts to run junit4-ant
> instead of ANT's default <junit>.
>
> 2) HOW DOES IT AFFECT ME AS A LUCENE/ SOLR DEVELOPER?
>
> - The most visible change is that 'random' field in LuceneTestCase is
> gone. This change was motivated by the fact that the field's value was
> read from places where it shouldn't be read, passed to places where it
> shouldn't be passed, etc. Instead, the Random instance for a given
> scope (see below) can be acquired from a static method in
> LuceneTestCase called random(). In the essence, you can just add
> brackets around your previous random field references and it _should_
> work out of the box. There are differences though: Random object
> returned by random() is valid only for the scope it was created for.
> So any of the following will end up in an exception: saving a Random
> instance in a static scope (@BeforeClass) to a field and reusing it in
> a test, passing a Random instance from one thread to another, saving a
> Random instance to a field in one test, using it in another, etc. In
> short: the result of random() is per-thread and only valid for the
> scope (test method, hook) it was acquired for. You _can_ call random()
> from non-test threads -- they will be given their own thread-local
> Random instances.
>
> - The 'random seed' is a single HEX-encoded long. The "three seeds"
> from before are gone. Everything is a derivative of the initial master
> seed.
>
> - I provided a 'help on syntax' for test properties. Type:
>
> ant test-help
>
> and the most common use case scenarios will be dumped to your console.
>
> - A notable difference is that 'tests.iter' property has been renamed
> to 'tests.iters' (you'll get a build failure and a message if you try
> to use the former one). I could add a fallback but I'd rather not
> introduce any more aliases.
>
> - "tests.iters" is no longer a poor-man's loop. It really re-runs a
> duplicate of a given test (or tests), including any @Before/@After
> hooks and setups. In theory, this means it is now possible to
> reiterate ANY test, no matter how complex. If it doesn't depend on
> static state, it can be repeated. This also links to how seed is used.
> It's probably best explained on an example:
>
> ant -Dtests.seed=deadbeef -Dtests.iters=3
> -Dtestcase=TestSubScorerFreqs test-core
>
> the above will result in 3 executions of every test in
> TestSubScorerFreqs. What you'll get on the console is:
>
>   [junit4] <JUnit4> says hello. Random seed: deadbeef
>   [junit4] Expected execution time on JVM J0:     0.02s
>   [junit4] Executing 1 suite with 1 JVM.
>   [junit4] Running org.apache.lucene.search.TestSubScorerFreqs
>   [junit4] Tests run:   9, Failures:   0, Errors:   0, Skipped:   0,
> Time:  0.45s
>   [junit4]
>   [junit4] JVM J0:     0.36 ..     0.96 =     0.60s
>   [junit4] Execution time total: 0.98 sec.
>   [junit4] Tests summary: 1 suite, 9 tests
>
> and if you peek at the log file with test results
> (build/core/test/tests-report.txt) you'll see the details of each

- Mark Miller
lucidimagination.com

> Thanks Dawid. I love it when my Junit4 says hello.

Isn't it a nice, personal touch? :D

Dawid

---------------------------------------------------------------------