Hi all,

greetings from my end. This is my first post on this mailing list. I have
few questions on multicore solr. For background we want to create a core
for each user logged in to our application. In that case it may be 50, 100,
1000, N-numbers. Each core will be used to write and search index in real
time.

1. Is this a good idea to go with?
2. What are the pros and cons of this approch?

Awaiting for your response.

Regards
AJ

Hi all,

greetings from my end. This is my first post on this mailing list. I have
few questions on multicore solr. For background we want to create a core
for each user logged in to our application. In that case it may be 50, 100,
1000, N-numbers. Each core will be used to write and search index in real
time.

1. Is this a good idea to go with?
2. What are the pros and cons of this approch?

Awaiting for your response.

Regards
AJ

Having cores per user is not good idea. The count is too high. Keep
everything in single core. You could filter the data based on user name or
user id.

Regards
Aditya
www.findbestopensource.com

On Tue, May 22, 2012 at 2:29 PM, Shanu Jha <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> greetings from my end. This is my first post on this mailing list. I have
> few questions on multicore solr. For background we want to create a core
> for each user logged in to our application. In that case it may be 50, 100,
> 1000, N-numbers. Each core will be used to write and search index in real
> time.
>
> 1. Is this a good idea to go with?
> 2. What are the pros and cons of this approch?
>
> Awaiting for your response.
>
> Regards
> AJ
>

Hi,

Could please tell me what do you mean by filter data by users? I would like
to know is there real problem creating a core for a user. ie. resource
utilization, cpu usage etc.

AJ

On Tue, May 22, 2012 at 4:39 PM, findbestopensource <
[EMAIL PROTECTED]> wrote:

> Having cores per user is not good idea. The count is too high. Keep
> everything in single core. You could filter the data based on user name or
> user id.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
>
> On Tue, May 22, 2012 at 2:29 PM, Shanu Jha <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > greetings from my end. This is my first post on this mailing list. I have
> > few questions on multicore solr. For background we want to create a core
> > for each user logged in to our application. In that case it may be 50,
> 100,
> > 1000, N-numbers. Each core will be used to write and search index in real
> > time.
> >
> > 1. Is this a good idea to go with?
> > 2. What are the pros and cons of this approch?
> >
> > Awaiting for your response.
> >
> > Regards
> > AJ
> >
>

It would help if you provide your use case. What are you indexing for each
user and why would you need a separate core for indexing each user? How do
you decide schema for each user? It might be better to describe your use
case and desired results. People on the list will be able to advice on the
best approach.

Sohail

Hi,

Thanks for your advice.
It is basically a meta search application. Users can perform a search on N number of data sources at a time. We broadcast Parallel search to each  selected data sources and write data to solr using custom build API(API and solr are deployed on separate machine API job is to perform parallel search, write data to solr ). API respond to application that some results are available then application fires  a search query to display the results(query would be q=unique_search_id). And other side API keep writing data to solr and user can fire a search to solr to view all results.

In current scenario we are using single solr server & we performing real time index and search. Performing these operations on single solr making process slow as index size increases.

So we are planning to use multi core solr and each user will have its core. All core will have the same schema.

Please suggest if this approach has any issues.

Rgds
AJ

On 22-May-2012, at 20:14, Sohail Aboobaker <[EMAIL PROTECTED]> wrote:

> It would help if you provide your use case. What are you indexing for each
> user and why would you need a separate core for indexing each user? How do
> you decide schema for each user? It might be better to describe your use
> case and desired results. People on the list will be able to advice on the
> best approach.
>
> Sohail

Awaiting for suggestions.

On Wed, May 23, 2012 at 8:04 AM, Amit Jha <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Thanks for your advice.
> It is basically a meta search application. Users can perform a search on N
> number of data sources at a time. We broadcast Parallel search to each
>  selected data sources and write data to solr using custom build API(API
> and solr are deployed on separate machine API job is to perform parallel
> search, write data to solr ). API respond to application that some results
> are available then application fires  a search query to display the
> results(query would be q=unique_search_id). And other side API keep writing
> data to solr and user can fire a search to solr to view all results.
>
> In current scenario we are using single solr server & we performing real
> time index and search. Performing these operations on single solr making
> process slow as index size increases.
>
> So we are planning to use multi core solr and each user will have its
> core. All core will have the same schema.
>
> Please suggest if this approach has any issues.
>
> Rgds
> AJ
>
> On 22-May-2012, at 20:14, Sohail Aboobaker <[EMAIL PROTECTED]> wrote:
>
> > It would help if you provide your use case. What are you indexing for
> each
> > user and why would you need a separate core for indexing each user? How
> do
> > you decide schema for each user? It might be better to describe your use
> > case and desired results. People on the list will be able to advice on
> the
> > best approach.
> >
> > Sohail
>

So are you even doing text search in Solr at all, or just using it as a
key-value store?

If the latter, do you have your schema configured so
that only the search_id field is indexed (with a keyword tokenizer) and
everything else only stored? Also, are you sure that Solr is the best
option as a key-value store?

Jens

On 05/23/2012 04:34 AM, Amit Jha wrote:
> Hi,
>
> Thanks for your advice. It is basically a meta search application.
> Users can perform a search on N number of data sources at a time. We
> broadcast Parallel search to each  selected data sources and write
> data to solr using custom build API(API and solr are deployed on
> separate machine API job is to perform parallel search, write data to
> solr ). API respond to application that some results are available
> then application fires  a search query to display the results(query
> would be q=unique_search_id). And other side API keep writing data to
> solr and user can fire a search to solr to view all results.
>
> In current scenario we are using single solr server&  we performing
> real time index and search. Performing these operations on single
> solr making process slow as index size increases.
>
> So we are planning to use multi core solr and each user will have its
> core. All core will have the same schema.
>
> Please suggest if this approach has any issues.
>
> Rgds AJ
>
> On 22-May-2012, at 20:14, Sohail Aboobaker<[EMAIL PROTECTED]>
> wrote:
>
>> It would help if you provide your use case. What are you indexing
>> for each user and why would you need a separate core for indexing
>> each user? How do you decide schema for each user? It might be
>> better to describe your use case and desired results. People on the
>> list will be able to advice on the best approach.
>>
>> Sohail
>

Jens,

Yes we are doing text search.

My question to all is, the approach of creating cores for each user is a
good idea?

AJ

On Wed, May 23, 2012 at 2:37 PM, Jens Grivolla <j+[EMAIL PROTECTED]> wrote:

> So are you even doing text search in Solr at all, or just using it as a
> key-value store?
>
> If the latter, do you have your schema configured so
> that only the search_id field is indexed (with a keyword tokenizer) and
> everything else only stored? Also, are you sure that Solr is the best
> option as a key-value store?
>
> Jens
>
>
> On 05/23/2012 04:34 AM, Amit Jha wrote:
>
>> Hi,
>>
>> Thanks for your advice. It is basically a meta search application.
>> Users can perform a search on N number of data sources at a time. We
>> broadcast Parallel search to each  selected data sources and write
>> data to solr using custom build API(API and solr are deployed on
>> separate machine API job is to perform parallel search, write data to
>> solr ). API respond to application that some results are available
>> then application fires  a search query to display the results(query
>> would be q=unique_search_id). And other side API keep writing data to
>> solr and user can fire a search to solr to view all results.
>>
>> In current scenario we are using single solr server&  we performing
>>
>> real time index and search. Performing these operations on single
>> solr making process slow as index size increases.
>>
>> So we are planning to use multi core solr and each user will have its
>> core. All core will have the same schema.
>>
>> Please suggest if this approach has any issues.
>>
>> Rgds AJ
>>
>> On 22-May-2012, at 20:14, Sohail Aboobaker<[EMAIL PROTECTED]**>
>> wrote:
>>
>>  It would help if you provide your use case. What are you indexing
>>> for each user and why would you need a separate core for indexing
>>> each user? How do you decide schema for each user? It might be
>>> better to describe your use case and desired results. People on the
>>> list will be able to advice on the best approach.
>>>
>>> Sohail
>>>
>>
>>
>
>

Please any one can help me on this

Rgds
AJ

On 23-May-2012, at 14:37, Jens Grivolla <j+[EMAIL PROTECTED]> wrote:

> So are you even doing text search in Solr at all, or just using it as a
> key-value store?
>
> If the latter, do you have your schema configured so
> that only the search_id field is indexed (with a keyword tokenizer) and everything else only stored? Also, are you sure that Solr is the best option as a key-value store?
>
> Jens
>
> On 05/23/2012 04:34 AM, Amit Jha wrote:
>> Hi,
>>
>> Thanks for your advice. It is basically a meta search application.
>> Users can perform a search on N number of data sources at a time. We
>> broadcast Parallel search to each  selected data sources and write
>> data to solr using custom build API(API and solr are deployed on
>> separate machine API job is to perform parallel search, write data to
>> solr ). API respond to application that some results are available
>> then application fires  a search query to display the results(query
>> would be q=unique_search_id). And other side API keep writing data to
>> solr and user can fire a search to solr to view all results.
>>
>> In current scenario we are using single solr server&  we performing
>> real time index and search. Performing these operations on single
>> solr making process slow as index size increases.
>>
>> So we are planning to use multi core solr and each user will have its
>> core. All core will have the same schema.
>>
>> Please suggest if this approach has any issues.
>>
>> Rgds AJ
>>
>> On 22-May-2012, at 20:14, Sohail Aboobaker<[EMAIL PROTECTED]>
>> wrote:
>>
>>> It would help if you provide your use case. What are you indexing
>>> for each user and why would you need a separate core for indexing
>>> each user? How do you decide schema for each user? It might be
>>> better to describe your use case and desired results. People on the
>>> list will be able to advice on the best approach.
>>>
>>> Sohail
>>
>
>

It is difficult to say but it does not sound very scalable to create one
core for each user. If there are only limited number of users then it might
be fine but if the users are growing fast then it seems like a lot of
overhead for each user.

Is it possible to create logical groups of users and have a core for each
set of users. You will have to add logic to identify the group and then
direct to appropriate core.