Home | About | Sematext search-lucene.com search-hadoop.com
 Search Lucene and all its subprojects:

Switch to Threaded View
Mahout >> mail # user >> recommender data live refresh

Copy link to this message
Re: recommender data live refresh
Hello guys,
we are using a mahout recommender in a multi threaded environment with an EJB singleton bean with own Concurrency Management.

public class ExampleSingletonBean {
  private String state;


  public String getState() {
    return state;


  public void setState(String newState) {
    state = newState;

We had some locking issues with other beans not because of mahout but because of the database.

I recommend to use JVisualVM to check how different threads behave during reload.


You could use JMeter (http://jmeter.apache.org/) to simulate usage and then checkout whats happening during reload of the data model.

I would be really interested in the pictures.

Hope that helps

On 27.02.2012, at 13:53, Sebastian Schelter wrote:

> Great to hear that. The most critical thing to be aware of this that you
> need memory to hold the old datamodel and the newly loaded one.
> --sebastian
> On 27.02.2012 13:50, Aleksei Udatšnõi wrote:
>> Excellent. As far as I tested, it works exactly as you described. Also
>> I did extra stress testing while auto-reload is enabled and there
>> wasn't too much performance overhead.
>> Fortunately the issue of having refresh() in between recommendation
>> requests is not a problem in my case. That's because in my servlet
>> recommend() is called once per HTTP request.
>> On Sun, Feb 26, 2012 at 12:39 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>>> You got me interested :) Refreshable.refresh() is mostly done by a
>>> 'RefreshHelper' that does re-entrant locking on the operation, so if
>>> you call refresh() twice from multiple threads it only does it once.
>>> It then changes the DataModel pointer.
>>> If a recommender method does two successive calls to the datamodel,
>>> and refresh() changes the datamodel in the middle, the different calls
>>> will go against different datamodels. The Recommender object would
>>> have to implement its own synchronous controls across methods. I
>>> looked at a couple and they do not. Using a recommender across a
>>> refresh() is not part of the contract. You have to do your own
>>> synchronous controls.
>>> On Fri, Feb 24, 2012 at 11:46 PM, Sebastian Schelter <[EMAIL PROTECTED]> wrote:
>>>> This is not true. If you are in a servlet environment, the servlet
>>>> container will use the recommender in multiple threads which means the
>>>> app does not block while the recommender is refreshed.
>>>> During the refresh process a new DataModel is built and the reference
>>>> to that is simply switched once it is fully loaded.
>>>> --sebastian
>>>> 2012/2/25 Lance Norskog <[EMAIL PROTECTED]>:
>>>>> You have to use two recommender objects if you want to serve
>>>>> recommendations while refreshing. I don't think any of the
>>>>> recommenders have that as part of the api contract.
>>>>> On Fri, Feb 24, 2012 at 8:31 AM, Aleksei Udatšnõi <[EMAIL PROTECTED]> wrote:
>>>>>> Hi,
>>>>>> I am using an item-based recommender inside a servlet and would like
>>>>>> to reload its data periodically (from db). Each reload takes up to 30
>>>>>> sec.
>>>>>> Recommender object is Refreshable and thus can do periodic reloads by
>>>>>> using refresh() function. My only concern is the impact of reload on
>>>>>> the live servlet users.
>>>>>> I wonder what happens with existing recommender.recommend() requests
>>>>>> while the recommender is being reloaded? Does it still use the old
>>>>>> data until reload is completely finished?
>>>>>> Thanks
>>>>>> Aleksei
>>>>> --
>>>>> Lance Norskog
>>> --
>>> Lance Norskog

Manuel Blechschmidt
Dortustr. 57
14467 Potsdam
Mobil: 0173/6322621
Twitter: http://twitter.com/Manuel_B