Hello,
I enter Chinese chars in the admin console for searching matched documents, it does not return any though I have uploaded some documents that has Chinese chars.

I guess the Chinese characters are not indexed. Is there any configuration I need to make in Solr?
 rgds,
canal

      

> I enter Chinese chars in the admin console for searching
> matched documents, it does not return any though I have
> uploaded some documents that has Chinese chars.
Could it be URI Charset Config?
http://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config

Hi,
I am using the sample, not deploying Solr in Tomcat. Is there a place I can modify this setting ?
 thanks,
canal
________________________________
From: Ahmet Arslan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sun, June 27, 2010 4:47:15 PM
Subject: Re: Chinese chars are not indexed ?

> I enter Chinese chars in the admin console for searching
> matched documents, it does not return any though I have
> uploaded some documents that has Chinese chars.
Could it be URI Charset Config?
http://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config
      

> I am using the sample, not deploying Solr in Tomcat. Is
> there a place I can modify this setting ?
Ha, okey if you are using jetty with java -jar start.jar then it is okey.
But for Chinese you need special tokenizer since Chinese is written without spaces between words.

<tokenizer class="solr.CJKTokenizerFactory"/>
Or you can search with both leading and trailing star. q=*ChineseText* should return something.

oh yes, *...* works. thanks.

I saw tokenizer is defined in schema.xml. There are a few places that define the tokenizer. Wondering if it is enough to define one for:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
       <!--  --------  this is the only one I need to modify ? --------- -->
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
       <!-- --------------------------------------------------------- -->
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
      </analyzer>    </fieldType>

 thanks,
canal
________________________________
From: Ahmet Arslan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Mon, June 28, 2010 2:54:16 PM
Subject: Re: Chinese chars are not indexed ?

> I am using the sample, not deploying Solr in Tomcat. Is
> there a place I can modify this setting ?
Ha, okey if you are using jetty with java -jar start.jar then it is okey.
But for Chinese you need special tokenizer since Chinese is written without spaces between words.

<tokenizer class="solr.CJKTokenizerFactory"/>
Or you can search with both leading and trailing star. q=*ChineseText* should return something.
      

> oh yes, *...* works. thanks.
>
> I saw tokenizer is defined in schema.xml. There are a few
> places that define the tokenizer. Wondering if it is enough
> to define one for:

It is better to define a brand new field type specific to Chinese.

http://wiki.apache.org/solr/LanguageAnalysis?highlight=%28CJKtokenizer%29#Chinese.2C_Japanese.2C_KoreanSomething like:

at index time:
<tokenizer class="solr.CJKTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>

at query time:
<tokenizer class="solr.CJKTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PositionFilterFactory" />

What if Chinese is mixed with English?

I have text that is entered by users and it could be a mix of Chinese, English, etc.

What's the best way to handle that?

Thanks.

--- On Mon, 6/28/10, Ahmet Arslan <[EMAIL PROTECTED]> wrote:

> From: Ahmet Arslan <[EMAIL PROTECTED]>
> Subject: Re: Chinese chars are not indexed ?
> To: [EMAIL PROTECTED]
> Date: Monday, June 28, 2010, 3:44 AM
> > oh yes, *...* works. thanks.
> >
> > I saw tokenizer is defined in schema.xml. There are a
> few
> > places that define the tokenizer. Wondering if it is
> enough
> > to define one for:
>
> It is better to define a brand new field type specific to
> Chinese.
>
> http://wiki.apache.org/solr/LanguageAnalysis?highlight=%28CJKtokenizer%29#Chinese.2C_Japanese.2C_KoreanSomething
> like:
>
> at index time:
> <tokenizer class="solr.CJKTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
>
> at query time:
> <tokenizer class="solr.CJKTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.PositionFilterFactory" />
>
>
>
>      
>