Hello, i hope you can help me
i am doing a crawling from a country, what i did was a script for avoid an
interrupted process configured for 20 threads. But when a certain page is
reached, the crawler behaves rarely stoping all threads (20) . Below is the
log, no one message of fetcher done is at the end and the processor is at
98% of usage. I don't know what happens, the server's page is not blocking
me (tested with wget) and i have enough disk space
* queue: http://www.frente.com.mx
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 6000
  minCrawlDelay = 0
  nextFetchTime = 1338578382163
  now           = 1338578381900
  0.
http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/
fetching
http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/
-finishing thread FetcherThread, activeThreads=19
-finishing thread FetcherThread, activeThreads=18
-finishing thread FetcherThread, activeThreads=17
-finishing thread FetcherThread, activeThreads=16
-finishing thread FetcherThread, activeThreads=15
-finishing thread FetcherThread, activeThreads=14
-finishing thread FetcherThread, activeThreads=13
-finishing thread FetcherThread, activeThreads=12
-finishing thread FetcherThread, activeThreads=11
-finishing thread FetcherThread, activeThreads=10
-finishing thread FetcherThread, activeThreads=9
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-activeThreads=7, spinWaiting=6, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0

--
View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381.html
Sent from the Nutch - User mailing list archive at Nabble.com.

This is normal and means the fetcher is finishing all it's input URL's and writing stuff to disk.
 
-----Original message-----
> From:pepe3059 <[EMAIL PROTECTED]>
> Sent: Sat 02-Jun-2012 22:15
> To: [EMAIL PROTECTED]
> Subject: threads disminution when fetching page
>
> Hello, i hope you can help me
>
>
> i am doing a crawling from a country, what i did was a script for avoid an
> interrupted process configured for 20 threads. But when a certain page is
> reached, the crawler behaves rarely stoping all threads (20) . Below is the
> log, no one message of fetcher done is at the end and the processor is at
> 98% of usage. I don't know what happens, the server's page is not blocking
> me (tested with wget) and i have enough disk space
>
>
> * queue: http://www.frente.com.mx
>   maxThreads    = 1
>   inProgress    = 0
>   crawlDelay    = 6000
>   minCrawlDelay = 0
>   nextFetchTime = 1338578382163
>   now           = 1338578381900
>   0.
> http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/
> fetching
> http://www.frente.com.mx/2012/03/01/rockabilly-radiografia-de-un-copete/
> -finishing thread FetcherThread, activeThreads=19
> -finishing thread FetcherThread, activeThreads=18
> -finishing thread FetcherThread, activeThreads=17
> -finishing thread FetcherThread, activeThreads=16
> -finishing thread FetcherThread, activeThreads=15
> -finishing thread FetcherThread, activeThreads=14
> -finishing thread FetcherThread, activeThreads=13
> -finishing thread FetcherThread, activeThreads=12
> -finishing thread FetcherThread, activeThreads=11
> -finishing thread FetcherThread, activeThreads=10
> -finishing thread FetcherThread, activeThreads=9
> -finishing thread FetcherThread, activeThreads=8
> -finishing thread FetcherThread, activeThreads=7
> -activeThreads=7, spinWaiting=6, fetchQueues.totalSize=0
> -finishing thread FetcherThread, activeThreads=6
> -finishing thread FetcherThread, activeThreads=5
> -finishing thread FetcherThread, activeThreads=4
> -finishing thread FetcherThread, activeThreads=3
> -finishing thread FetcherThread, activeThreads=2
> -finishing thread FetcherThread, activeThreads=1
> -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
> -finishing thread FetcherThread, activeThreads=0
> -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
> -activeThreads=0
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

thank you for your answer Markus

you mean, until the fetch process finishes, is information stored using hdfs
by nutch? meanwhile is in the defined tmp directory ?

no change is presented in two days (and i think is so many time for
writing). and the problem persists, could be some agent revisor from the
above mentioned site that provokes a pause?

thank you vary much for your time

José
--
View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381p3987629.html
Sent from the Nutch - User mailing list archive at Nabble.com.

-----Original message-----
> From:pepe3059 <[EMAIL PROTECTED]>
> Sent: Mon 04-Jun-2012 20:42
> To: [EMAIL PROTECTED]
> Subject: RE: threads disminution when fetching page
>
> thank you for your answer Markus

Hi

>
> you mean, until the fetch process finishes, is information stored using hdfs
> by nutch? meanwhile is in the defined tmp directory ?

Yes. Hadoop MapReduce stores intermediate files in the tmp directory and then writes the reduce output to HDFS.

>
> no change is presented in two days (and i think is so many time for
> writing). and the problem persists,

You mean it hasn't finished the fetch job in two days? Thats odd and shouldnt happen, at least an error should be thrown.

> could be some agent revisor from the
> above mentioned site that provokes a pause?

No. If a site doesnt respond it times out gracefully. If a site has a too large crawl delay it is skipped as well.

You can test with $nutch indexchecker <url> to see if a page processes correctly.
>
>
>
> thank you vary much for your time
>
> José
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381p3987629.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

me again :)

at the end of fetch process, is the regex-urlfilter considered? because i
have 16+ regular expressions but update process is not reached yet. is
considered in updatedb?
another datum; disk usage from the last segment: 21Gb
are 20 threads heavy load for a six core xeon? (i assign the rule "-threads
20")
sorry for my bad english :P and thank you for your answers

--
View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381p3987887.html
Sent from the Nutch - User mailing list archive at Nabble.com.

-----Original message-----
> From:pepe3059 <[EMAIL PROTECTED]>
> Sent: Wed 06-Jun-2012 02:58
> To: [EMAIL PROTECTED]
> Subject: RE: threads disminution when fetching page
>
> me again :)
>
> at the end of fetch process, is the regex-urlfilter considered?

No. At the end of the fetch the mapper output is writting to intermediate files, they get merged and sorted and are sent to the reducer which consumes all output and then finally writes it to disk.

> because i have 16+ regular expressions but update process is not reached yet. is
> considered in updatedb?
> another datum; disk usage from the last segment: 21Gb

You should make your segments smaller by generating less records, consider the -topn parameter for the generator.

> are 20 threads heavy load for a six core xeon? (i assign the rule "-threads
> 20")

No, not too heavy, it can be much higher.

>
>
>
>
> sorry for my bad english :P and thank you for your answers
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/threads-disminution-when-fetching-page-tp3987381p3987887.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>