Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Lucene and all its subprojects:

Switch to Threaded View
Mahout >> mail # user >> recommender data live refresh


Copy link to this message
-
Re: recommender data live refresh
Hello guys,
we are using a mahout recommender in a multi threaded environment with an EJB singleton bean with own Concurrency Management.

e.g.:
(http://docs.oracle.com/javaee/6/tutorial/doc/gipvi.html#gipsz)
@ConcurrencyManagement(ConcurrencyManagementType.BEAN)
@Singleton
public class ExampleSingletonBean {
  private String state;

  
@Lock(LockType.READ)

  public String getState() {
    return state;
  }

  
@Lock(LockType.WRITE)

  public void setState(String newState) {
    state = newState;
  }
}

We had some locking issues with other beans not because of mahout but because of the database.

I recommend to use JVisualVM to check how different threads behave during reload.

http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html

You could use JMeter (http://jmeter.apache.org/) to simulate usage and then checkout whats happening during reload of the data model.

I would be really interested in the pictures.

Hope that helps
    Manuel

On 27.02.2012, at 13:53, Sebastian Schelter wrote:

> Great to hear that. The most critical thing to be aware of this that you
> need memory to hold the old datamodel and the newly loaded one.
>
> --sebastian
>
> On 27.02.2012 13:50, Aleksei Udatšnõi wrote:
>> Excellent. As far as I tested, it works exactly as you described. Also
>> I did extra stress testing while auto-reload is enabled and there
>> wasn't too much performance overhead.
>>
>> Fortunately the issue of having refresh() in between recommendation
>> requests is not a problem in my case. That's because in my servlet
>> recommend() is called once per HTTP request.
>>
>>
>>
>> On Sun, Feb 26, 2012 at 12:39 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>>> You got me interested :) Refreshable.refresh() is mostly done by a
>>> 'RefreshHelper' that does re-entrant locking on the operation, so if
>>> you call refresh() twice from multiple threads it only does it once.
>>> It then changes the DataModel pointer.
>>>
>>> If a recommender method does two successive calls to the datamodel,
>>> and refresh() changes the datamodel in the middle, the different calls
>>> will go against different datamodels. The Recommender object would
>>> have to implement its own synchronous controls across methods. I
>>> looked at a couple and they do not. Using a recommender across a
>>> refresh() is not part of the contract. You have to do your own
>>> synchronous controls.
>>>
>>> On Fri, Feb 24, 2012 at 11:46 PM, Sebastian Schelter <[EMAIL PROTECTED]> wrote:
>>>> This is not true. If you are in a servlet environment, the servlet
>>>> container will use the recommender in multiple threads which means the
>>>> app does not block while the recommender is refreshed.
>>>>
>>>> During the refresh process a new DataModel is built and the reference
>>>> to that is simply switched once it is fully loaded.
>>>>
>>>> --sebastian
>>>>
>>>> 2012/2/25 Lance Norskog <[EMAIL PROTECTED]>:
>>>>> You have to use two recommender objects if you want to serve
>>>>> recommendations while refreshing. I don't think any of the
>>>>> recommenders have that as part of the api contract.
>>>>>
>>>>> On Fri, Feb 24, 2012 at 8:31 AM, Aleksei Udatšnõi <[EMAIL PROTECTED]> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am using an item-based recommender inside a servlet and would like
>>>>>> to reload its data periodically (from db). Each reload takes up to 30
>>>>>> sec.
>>>>>>
>>>>>> Recommender object is Refreshable and thus can do periodic reloads by
>>>>>> using refresh() function. My only concern is the impact of reload on
>>>>>> the live servlet users.
>>>>>>
>>>>>> I wonder what happens with existing recommender.recommend() requests
>>>>>> while the recommender is being reloaded? Does it still use the old
>>>>>> data until reload is completely finished?
>>>>>>
>>>>>> Thanks
>>>>>> Aleksei
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> [EMAIL PROTECTED]
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> [EMAIL PROTECTED]
>

--
Manuel Blechschmidt
Dortustr. 57
14467 Potsdam
Mobil: 0173/6322621
Twitter: http://twitter.com/Manuel_B
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB