-Re: Bottleneck of my crawls: NativeCodeLoader
Sebastian Nagel 2012-03-27, 21:29
there is a description on how to install native libraries:
If installed appropriately native libs are loaded and the
warnings will disappear.
But are you sure that it's really the library loading that
takes the time and not the step run after but without an
initial log message? LinkDb may take some time because it
reads all (or all newly created) segments.
On 03/26/2012 01:35 PM, James Ford wrote:
> I am trying to optimize my crawls as much as possible. The current
> bottleneck is the step after adding segments to the linkdb, where Nutch is
> trying to load the natiive-hadoop library:
> 2012-03-26 13:20:59,089 WARN util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> This step takes about 15 minutes, compared to all other steps which takes
> about 25 minutes in total. How can I make this step faster?
> James Ford
> View this message in context: http://lucene.472066.n3.nabble.com/Bottleneck-of-my-crawls-NativeCodeLoader-tp3857929p3857929.html
> Sent from the Nutch - User mailing list archive at Nabble.com.