ytthet 2010-11-12, 18:27
Alexander Aristov 2010-11-12, 19:33
-Re: When and how(properly) to delete segments directory (Nutch 1.0)
Andrzej Bialecki 2010-11-12, 22:19
On 2010-11-12 20:33, Alexander Aristov wrote:
> why are you so afraid of segment merger? It appears to be the only
> "official" way to get rid of excessive folders. of course it's time/resource
> consuming but is your system so high loaded?
The truth is that in 99% cases SegmentMerger is not needed, and it is
indeed very resource intensive. Merging segments is completely optional
- NutchBean and Nutch search server will work just fine without it, and
the performance degradation related to using many segments is small. Of
course up to a point - if you have hundreds of tiny segments you should
merge them (at least some of them into larger segments), but if you have
just 30 or so then it should not be a big deal.
> Also I might be wrong but if you are not planning to return summaries and
> content from nutch when you can remove folders by rm.
> And you can completely get rid of segments by using the solr indexer. After
> that you perform indexing you can delete fetched segments. I presume this is
> what you saw in other threads.
If you keep up with re-fetching, i.e. new fetch cycles can keep up with
the amount of pages that become outdated, then you can remove all
segments older than db.max.fetch.interval, because Nutch forces all
pages older than that to become ready for re-fetching, so they will be
likely found in some newer segment.
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Ye T Thet 2010-11-13, 02:06