...after I get back from Thanksgiving dinner :-)

1. In URLFilterChecker, the cmd line tool requires URLs to be fed into it on STDIN, but
that isn't documented anywhere, even in the tool help printed to STDOUT. I'll fix that.

2. In ParseOutputFormat, I see a code block:

{code}
          // collect outlinks for subsequent db update
          Outlink[] links = parseData.getOutlinks();
          int outlinksToStore = Math.min(maxOutlinks, links.length);
          if (ignoreExternalLinks) {
            try {
              fromHost = new URL(fromUrl).getHost().toLowerCase();
            } catch (MalformedURLException e) {
              fromHost = null;
            }
          } else {
            fromHost = null;
          }
{code}

The if(ignoreExternalLinks) part then gets subsequently set and
reset in the ensuing for loop:

{code}
          int validCount = 0;
          CrawlDatum adjust = null;
          List<Entry<Text, CrawlDatum>> targets = new ArrayList<Entry<Text, CrawlDatum>>(outlinksToStore);
          List<Outlink> outlinkList = new ArrayList<Outlink>(outlinksToStore);
          for (int i = 0; i < links.length && validCount < outlinksToStore; i++) {
            String toUrl = links[i].getToUrl();
            // ignore links to self (or anchors within the page)
            if (fromUrl.equals(toUrl)) {
              continue;
            }
            if (ignoreExternalLinks) {
              try {
                toHost = new URL(toUrl).getHost().toLowerCase();
              } catch (MalformedURLException e) {
                toHost = null;
              }
              if (toHost == null || !toHost.equals(fromHost)) { // external links
                continue; // skip it
              }
            }
{code}

So, what's the point of that initial if(...) block outside of the for loop. Isn't it
redundant?

If so, I'll file an issue and fix that.

Cheers,
Chris

P.S. Happy Thanksgiving to Nutch'ers in the US!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> ...after I get back from Thanksgiving dinner :-)
>
> 1. In URLFilterChecker, the cmd line tool requires URLs to be fed into it
> on STDIN, but that isn't documented anywhere, even in the tool help
> printed to STDOUT. I'll fix that.
>
> 2. In ParseOutputFormat, I see a code block:
>
> {code}
>           // collect outlinks for subsequent db update
>           Outlink[] links = parseData.getOutlinks();
>           int outlinksToStore = Math.min(maxOutlinks, links.length);
>           if (ignoreExternalLinks) {
>             try {
>               fromHost = new URL(fromUrl).getHost().toLowerCase();
>             } catch (MalformedURLException e) {
>               fromHost = null;
>             }
>           } else {
>             fromHost = null;
>           }
> {code}
>
> The if(ignoreExternalLinks) part then gets subsequently set and
> reset in the ensuing for loop:
>
> {code}
>           int validCount = 0;
>           CrawlDatum adjust = null;
>           List<Entry<Text, CrawlDatum>> targets = new ArrayList<Entry<Text,
> CrawlDatum>>(outlinksToStore); List<Outlink> outlinkList = new
> ArrayList<Outlink>(outlinksToStore); for (int i = 0; i < links.length &&
> validCount < outlinksToStore; i++) { String toUrl = links[i].getToUrl();
>             // ignore links to self (or anchors within the page)
>             if (fromUrl.equals(toUrl)) {
>               continue;
>             }
>             if (ignoreExternalLinks) {
>               try {
>                 toHost = new URL(toUrl).getHost().toLowerCase();
>               } catch (MalformedURLException e) {
>                 toHost = null;
>               }
>               if (toHost == null || !toHost.equals(fromHost)) { // external
> links continue; // skip it
>               }
>             }
> {code}
>
> So, what's the point of that initial if(...) block outside of the for loop.
> Isn't it redundant?

This is trunk? I've been and still am working on some issues for a new feature
in this part of that source file.

https://issues.apache.org/jira/browse/NUTCH-1184
https://issues.apache.org/jira/browse/NUTCH-1174

>
> If so, I'll file an issue and fix that.
>
> Cheers,
> Chris
>
> P.S. Happy Thanksgiving to Nutch'ers in the US!
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hi Markus,

On Nov 24, 2011, at 12:03 PM, Markus Jelsma wrote:

> > So, what's the point of that initial if(...) block outside of the for loop.
> > Isn't it redundant?
> This is trunk? I've been and still am working on some issues for a new feature in this part of that source file.
> https://issues.apache.org/jira/browse/NUTCH-1184
> https://issues.apache.org/jira/browse/NUTCH-1174

Yep it's trunk alright. I'm fine with you making the update I suggested, or with me doing it.
2 questions:

1. Am I right in observing that the code is redundant and should be removed?
2. If I am right on #1, do you want me to make the update, or are you saying
that you want to make it as part of NUTCH-1184 and NUTCH-1174?

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hi

On Friday 25 November 2011 01:13:47 Mattmann, Chris A (388J) wrote:
> Hi Markus,
>
> On Nov 24, 2011, at 12:03 PM, Markus Jelsma wrote:
> > > So, what's the point of that initial if(...) block outside of the for
> > > loop. Isn't it redundant?
> >
> > This is trunk? I've been and still am working on some issues for a new
> > feature in this part of that source file.
> > https://issues.apache.org/jira/browse/NUTCH-1184
> > https://issues.apache.org/jira/browse/NUTCH-1174
>
> Yep it's trunk alright. I'm fine with you making the update I suggested, or
> with me doing it. 2 questions:
>
> 1. Am I right in observing that the code is redundant and should be
> removed?
I believe so. Ive tested the removal of that part with the code of NUTCH-1184
and all goes well.

> 2. If I am right on #1, do you want me to make the update, or are
> you saying that you want to make it as part of NUTCH-1184 and NUTCH-1174?

1174 is already committed. Ive added a patch for ParseOutputformat to 1184
incorporating your newly created patch.

cheers

>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

--
Markus Jelsma - CTO - Openindex

Hi Markus,

Super +1. Thanks for incorporating it as part of your patch.

1184 looks good -- my +1 to commit it, even if i progress.
Then we can close out 1212 at that point.

Thanks!

Cheers,
Chris

On Nov 25, 2011, at 5:16 AM, Markus Jelsma wrote:

> Hi
>
> On Friday 25 November 2011 01:13:47 Mattmann, Chris A (388J) wrote:
>> Hi Markus,
>>
>> On Nov 24, 2011, at 12:03 PM, Markus Jelsma wrote:
>>>> So, what's the point of that initial if(...) block outside of the for
>>>> loop. Isn't it redundant?
>>>
>>> This is trunk? I've been and still am working on some issues for a new
>>> feature in this part of that source file.
>>> https://issues.apache.org/jira/browse/NUTCH-1184
>>> https://issues.apache.org/jira/browse/NUTCH-1174
>>
>> Yep it's trunk alright. I'm fine with you making the update I suggested, or
>> with me doing it. 2 questions:
>>
>> 1. Am I right in observing that the code is redundant and should be
>> removed?
> I believe so. Ive tested the removal of that part with the code of NUTCH-1184
> and all goes well.
>
>> 2. If I am right on #1, do you want me to make the update, or are
>> you saying that you want to make it as part of NUTCH-1184 and NUTCH-1174?
>
> 1174 is already committed. Ive added a patch for ParseOutputformat to 1184
> incorporating your newly created patch.
>
> cheers
>
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: [EMAIL PROTECTED]
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> --
> Markus Jelsma - CTO - Openindex
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Friday 25 November 2011 16:48:44 Mattmann, Chris A (388J) wrote:
> Hi Markus,
>
> Super +1. Thanks for incorporating it as part of your patch.
>
> 1184 looks good -- my +1 to commit it, even if i progress.
> Then we can close out 1212 at that point.

Yes. Although julien asked to delay it since it's a pretty invasive patch.

btw: have you come around pushing out the RC that passed the voting round?

>
> Thanks!
>
> Cheers,
> Chris
>
> On Nov 25, 2011, at 5:16 AM, Markus Jelsma wrote:
> > Hi
> >
> > On Friday 25 November 2011 01:13:47 Mattmann, Chris A (388J) wrote:
> >> Hi Markus,
> >>
> >> On Nov 24, 2011, at 12:03 PM, Markus Jelsma wrote:
> >>>> So, what's the point of that initial if(...) block outside of the for
> >>>> loop. Isn't it redundant?
> >>>
> >>> This is trunk? I've been and still am working on some issues for a new
> >>> feature in this part of that source file.
> >>> https://issues.apache.org/jira/browse/NUTCH-1184
> >>> https://issues.apache.org/jira/browse/NUTCH-1174
> >>
> >> Yep it's trunk alright. I'm fine with you making the update I suggested,
> >> or with me doing it. 2 questions:
> >>
> >> 1. Am I right in observing that the code is redundant and should be
> >> removed?
> >
> > I believe so. Ive tested the removal of that part with the code of
> > NUTCH-1184 and all goes well.
> >
> >> 2. If I am right on #1, do you want me to make the update, or are
> >> you saying that you want to make it as part of NUTCH-1184 and
> >> NUTCH-1174?
> >
> > 1174 is already committed. Ive added a patch for ParseOutputformat to
> > 1184 incorporating your newly created patch.
> >
> > cheers
> >
> >> Cheers,
> >> Chris
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: [EMAIL PROTECTED]
> >> WWW:   http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

--
Markus Jelsma - CTO - Openindex

On Nov 25, 2011, at 8:02 AM, Markus Jelsma wrote:

>
>
> On Friday 25 November 2011 16:48:44 Mattmann, Chris A (388J) wrote:
>> Hi Markus,
>>
>> Super +1. Thanks for incorporating it as part of your patch.
>>
>> 1184 looks good -- my +1 to commit it, even if i progress.
>> Then we can close out 1212 at that point.
>
> Yes. Although julien asked to delay it since it's a pretty invasive patch.

Okey dok.

>
> btw: have you come around pushing out the RC that passed the voting round?

On my list of TODO for today now that my crawl is working properly! :-)

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++