Is there a tokenizer that will allow me to search for parts of a URL?  For
example, the search "google" would match on the data "
http://mail.google.com/dlkjadf"

This tokenizer factory doesn't seem to be sufficient:

        <fieldType name="text_standard" class="solr.TextField"
positionIncrementGap="100">
            <analyzer type="index">
                <tokenizer class="solr.WhitespaceTokenizerFactory"/>
                <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.SnowballPorterFilterFactory"
language="English" protected="protwords.txt"/>
            </analyzer>
            <analyzer type="query">
                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>

                 <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
                 <filter class="solr.LowerCaseFilterFactory"/>
                 <filter class="solr.SnowballPorterFilterFactory"
language="English" protected="protwords.txt"/>
             </analyzer>
    </fieldType>

Thanks.

Try setting generateWordParts=1 in your WDF. Also, having a WhitespaceTokenizer makes little sense for URL's, there should be no whitespace in a URL, the StandardTokenizer can tokenize a URL. Anyway, the problem is your WDF.
 
-----Original message-----
From: Max Lynch <[EMAIL PROTECTED]>
Sent: Thu 23-09-2010 23:00
To: [EMAIL PROTECTED];
Subject: Search a URL

Is there a tokenizer that will allow me to search for parts of a URL?  For
example, the search "google" would match on the data "
http://mail.google.com/dlkjadf"

This tokenizer factory doesn't seem to be sufficient:

       <fieldType name="text_standard" class="solr.TextField"
positionIncrementGap="100">
           <analyzer type="index">
               <tokenizer class="solr.WhitespaceTokenizerFactory"/>
               <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
               <filter class="solr.LowerCaseFilterFactory"/>
               <filter class="solr.SnowballPorterFilterFactory"
language="English" protected="protwords.txt"/>
           </analyzer>
           <analyzer type="query">
                <tokenizer class="solr.WhitespaceTokenizerFactory"/>

                <filter class="solr.WordDelimiterFilterFactory"
generateWordParts="0" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.SnowballPorterFilterFactory"
language="English" protected="protwords.txt"/>
            </analyzer>
   </fieldType>

Thanks.

WDF is not WTF(what I think when I see WDF), right ;-)

What is WDF?

Dennis Gearon

Signature Warning
----------------
EARTH has a Right To Life,
  otherwise we all die.

Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php
--- On Thu, 9/23/10, Markus Jelsma <[EMAIL PROTECTED]> wrote:

> From: Markus Jelsma <[EMAIL PROTECTED]>
> Subject: RE: Search a URL
> To: [EMAIL PROTECTED]
> Date: Thursday, September 23, 2010, 2:11 PM
> Try setting generateWordParts=1 in
> your WDF. Also, having a WhitespaceTokenizer makes little
> sense for URL's, there should be no whitespace in a URL, the
> StandardTokenizer can tokenize a URL. Anyway, the problem is
> your WDF.
>  
> -----Original message-----
> From: Max Lynch <[EMAIL PROTECTED]>
> Sent: Thu 23-09-2010 23:00
> To: [EMAIL PROTECTED];
>
> Subject: Search a URL
>
> Is there a tokenizer that will allow me to search for parts
> of a URL?  For
> example, the search "google" would match on the data "
> http://mail.google.com/dlkjadf"
>
> This tokenizer factory doesn't seem to be sufficient:
>
>        <fieldType name="text_standard"
> class="solr.TextField"
> positionIncrementGap="100">
>            <analyzer type="index">
>                <tokenizer
> class="solr.WhitespaceTokenizerFactory"/>
>                <filter
> class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="1"
> catenateWords="1"
> catenateNumbers="1" catenateAll="0"
> splitOnCaseChange="1"/>
>                <filter
> class="solr.LowerCaseFilterFactory"/>
>                <filter
> class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
>            </analyzer>
>            <analyzer type="query">
>                 <tokenizer
> class="solr.WhitespaceTokenizerFactory"/>
>
>                 <filter
> class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="1"
> catenateWords="1"
> catenateNumbers="1" catenateAll="0"
> splitOnCaseChange="1"/>
>                 <filter
> class="solr.LowerCaseFilterFactory"/>
>                 <filter
> class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
>             </analyzer>
>    </fieldType>
>
> Thanks.
>

WordDelimiterFilter

On Friday 24 September 2010 02:42:52 Dennis Gearon wrote:
> WDF is not WTF(what I think when I see WDF), right ;-)
>
> What is WDF?
>
> Dennis Gearon
>
> Signature Warning
> ----------------
> EARTH has a Right To Life,
>   otherwise we all die.
>
> Read 'Hot, Flat, and Crowded'
> Laugh at http://www.yert.com/film.php
>
> --- On Thu, 9/23/10, Markus Jelsma <[EMAIL PROTECTED]> wrote:
> > From: Markus Jelsma <[EMAIL PROTECTED]>
> > Subject: RE: Search a URL
> > To: [EMAIL PROTECTED]
> > Date: Thursday, September 23, 2010, 2:11 PM
> > Try setting generateWordParts=1 in
> > your WDF. Also, having a WhitespaceTokenizer makes little
> > sense for URL's, there should be no whitespace in a URL, the
> > StandardTokenizer can tokenize a URL. Anyway, the problem is
> > your WDF.
> >  
> > -----Original message-----
> > From: Max Lynch <[EMAIL PROTECTED]>
> > Sent: Thu 23-09-2010 23:00
> > To: [EMAIL PROTECTED];
> >
> > Subject: Search a URL
> >
> > Is there a tokenizer that will allow me to search for parts
> > of a URL?  For
> > example, the search "google" would match on the data "
> > http://mail.google.com/dlkjadf"
> >
> > This tokenizer factory doesn't seem to be sufficient:
> >
> >        <fieldType name="text_standard"
> > class="solr.TextField"
> > positionIncrementGap="100">
> >            <analyzer type="index">
> >                <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> >                <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="0" generateNumberParts="1"
> > catenateWords="1"
> > catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> >                <filter
> > class="solr.LowerCaseFilterFactory"/>
> >                <filter
> > class="solr.SnowballPorterFilterFactory"
> > language="English" protected="protwords.txt"/>
> >            </analyzer>
> >            <analyzer type="query">
> >                 <tokenizer
> > class="solr.WhitespaceTokenizerFactory"/>
> >
> >                 <filter
> > class="solr.WordDelimiterFilterFactory"
> > generateWordParts="0" generateNumberParts="1"
> > catenateWords="1"
> > catenateNumbers="1" catenateAll="0"
> > splitOnCaseChange="1"/>
> >                 <filter
> > class="solr.LowerCaseFilterFactory"/>
> >                 <filter
> > class="solr.SnowballPorterFilterFactory"
> > language="English" protected="protwords.txt"/>
> >             </analyzer>
> >    </fieldType>
> >
> > Thanks.
>

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

LetterTokenizerFactory will use each contiguous sequence of letters and discard the rest. http, https, com,  etc. would need to be a stopword.

Alternatively you can try PatternTokenizerFactory with a regular expression if you are looking for a specific part of the URL.

On Sep 23, 2010, at 10:59 PM, Max Lynch wrote:

> Is there a tokenizer that will allow me to search for parts of a URL?  For
> example, the search "google" would match on the data "
> http://mail.google.com/dlkjadf"
>
> This tokenizer factory doesn't seem to be sufficient:
>
>        <fieldType name="text_standard" class="solr.TextField"
> positionIncrementGap="100">
>            <analyzer type="index">
>                <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>                <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
>                <filter class="solr.LowerCaseFilterFactory"/>
>                <filter class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
>            </analyzer>
>            <analyzer type="query">
>                 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>
>                 <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="0" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
>                 <filter class="solr.LowerCaseFilterFactory"/>
>                 <filter class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
>             </analyzer>
>    </fieldType>
>
> Thanks.