Home | About | Sematext search-lucene.com search-hadoop.com
 Search Lucene and all its subprojects:

Switch to Plain View
Mahout >> mail # user >> recommender data live refresh


+
Aleksei Udatšnõi 2012-02-24, 16:31
+
Lance Norskog 2012-02-25, 03:07
+
Sebastian Schelter 2012-02-25, 07:46
+
Lance Norskog 2012-02-25, 23:39
+
Aleksei Udatšnõi 2012-02-27, 12:50
+
Sebastian Schelter 2012-02-27, 12:53
Copy link to this message
-
Re: recommender data live refresh
Hello guys,
we are using a mahout recommender in a multi threaded environment with an EJB singleton bean with own Concurrency Management.

e.g.:
(http://docs.oracle.com/javaee/6/tutorial/doc/gipvi.html#gipsz)
@ConcurrencyManagement(ConcurrencyManagementType.BEAN)
@Singleton
public class ExampleSingletonBean {
  private String state;

  
@Lock(LockType.READ)

  public String getState() {
    return state;
  }

  
@Lock(LockType.WRITE)

  public void setState(String newState) {
    state = newState;
  }
}

We had some locking issues with other beans not because of mahout but because of the database.

I recommend to use JVisualVM to check how different threads behave during reload.

http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html

You could use JMeter (http://jmeter.apache.org/) to simulate usage and then checkout whats happening during reload of the data model.

I would be really interested in the pictures.

Hope that helps
    Manuel

On 27.02.2012, at 13:53, Sebastian Schelter wrote:

> Great to hear that. The most critical thing to be aware of this that you
> need memory to hold the old datamodel and the newly loaded one.
>
> --sebastian
>
> On 27.02.2012 13:50, Aleksei Udatšnõi wrote:
>> Excellent. As far as I tested, it works exactly as you described. Also
>> I did extra stress testing while auto-reload is enabled and there
>> wasn't too much performance overhead.
>>
>> Fortunately the issue of having refresh() in between recommendation
>> requests is not a problem in my case. That's because in my servlet
>> recommend() is called once per HTTP request.
>>
>>
>>
>> On Sun, Feb 26, 2012 at 12:39 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>>> You got me interested :) Refreshable.refresh() is mostly done by a
>>> 'RefreshHelper' that does re-entrant locking on the operation, so if
>>> you call refresh() twice from multiple threads it only does it once.
>>> It then changes the DataModel pointer.
>>>
>>> If a recommender method does two successive calls to the datamodel,
>>> and refresh() changes the datamodel in the middle, the different calls
>>> will go against different datamodels. The Recommender object would
>>> have to implement its own synchronous controls across methods. I
>>> looked at a couple and they do not. Using a recommender across a
>>> refresh() is not part of the contract. You have to do your own
>>> synchronous controls.
>>>
>>> On Fri, Feb 24, 2012 at 11:46 PM, Sebastian Schelter <[EMAIL PROTECTED]> wrote:
>>>> This is not true. If you are in a servlet environment, the servlet
>>>> container will use the recommender in multiple threads which means the
>>>> app does not block while the recommender is refreshed.
>>>>
>>>> During the refresh process a new DataModel is built and the reference
>>>> to that is simply switched once it is fully loaded.
>>>>
>>>> --sebastian
>>>>
>>>> 2012/2/25 Lance Norskog <[EMAIL PROTECTED]>:
>>>>> You have to use two recommender objects if you want to serve
>>>>> recommendations while refreshing. I don't think any of the
>>>>> recommenders have that as part of the api contract.
>>>>>
>>>>> On Fri, Feb 24, 2012 at 8:31 AM, Aleksei Udatšnõi <[EMAIL PROTECTED]> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am using an item-based recommender inside a servlet and would like
>>>>>> to reload its data periodically (from db). Each reload takes up to 30
>>>>>> sec.
>>>>>>
>>>>>> Recommender object is Refreshable and thus can do periodic reloads by
>>>>>> using refresh() function. My only concern is the impact of reload on
>>>>>> the live servlet users.
>>>>>>
>>>>>> I wonder what happens with existing recommender.recommend() requests
>>>>>> while the recommender is being reloaded? Does it still use the old
>>>>>> data until reload is completely finished?
>>>>>>
>>>>>> Thanks
>>>>>> Aleksei
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> [EMAIL PROTECTED]
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> [EMAIL PROTECTED]
>

--
Manuel Blechschmidt
Dortustr. 57
14467 Potsdam
Mobil: 0173/6322621
Twitter: http://twitter.com/Manuel_B