-Re: Getting seed url
remi tassing 2012-06-11, 22:45
Segments have a field called 'outlinks', could this help?
On Tuesday, June 12, 2012, Sebastian Nagel wrote:
> Hi Sandeep,
> tracking the seed(s) for a document could be done by a scoring filter.
> The seed URL must be passed:
> 0 into CrawlDatum's meta by injectedScore()
> (alternatively, use additional fields in the seed file:
> <url> <tab> seed=<url>
> see Injector Javadoc)
> 1 in passScoreBeforeParsing():
> from CrawlDatum to Content
> 2 in passScoreAfterParsing():
> from Content to ParseData
> 3 in distributeScoreToOutlinks():
> from source ParseData to all target/outlink CrawlDatum objects
> 4 in updateDbScore():
> resolve inlinks from multiple seeds
> Point 4 shows a little problem: a page may be reachable from multiple
> The web is a graph not a forest of trees each with one seed as root!
> Finally: amazon.com is definitely linked from apache.org
> but it is not a "project" site.
> Wouldn't a mapping <domain name> -> <meta data> be more reliable
> (though notoriously incomplete)?
> On 06/11/2012 08:09 PM, Sandeep C R wrote:
> > Hello,
> > I am trying to find a way in which I can get the seed url of current url
> > being parsed. I have many URL's in seed.txt. I am trying to add
> > metadata for each URL crawled. The metadata depends on the seed URL of
> > current URL. This metadata will be later picked by the indexer. I have
> > written a custom plugin for this purpose. However I am unable to get the
> > seed url of the current url being parsed.
> > Ex: This is my seed.txt
> > http://apache.org
> > http://amazon.com
> > http://w3.org
> > For all URL's crawled for every seed URL, I want to add metadata. The
> > of metadata will depend on seed URL. I have a properties file which will
> > map seed url to metadata value. If seed url is http://apache.org then my
> > metadata will be something like "project". If it is http://amazon.comthen
> > it will be "estore". I have written a plugin which will add metadata.
> > plugin extends HtmlParserFilter. However I am not able find a way to get
> > the seed url of current url. If http://nutch.apache.org is being parsed
> > currently, then how do we know the seed url(http:/apache.org) of this
> > Is there any API which I could use in my plugin? Or is there any better
> > to achieve this?
> > Regards,
> > Sandeep