Hello again,

I want to extract specific meta tag from HTML pages, like:

<meta name="uniks-fb" value="fb16" />

But it seems that they aren't extracted by the parser. I dumped the
segment of a page (Since the readseg doesn't work for me :-/ ) and
inspected the values for this example page:

http://cms.uni-kassel.de/unicms/index.php?id=29216&L=0

This page contains these metatags:
<meta name="uniks-fb" content="default" />
<meta name="keywords"
content="Universitt,Kassel,Forschung,Lehre,Wissenschaft" />
<meta name="robots" content="index" />
<meta name="DC.Description" content="Der Internetauftritt der
Universitï¿œt Kassel" />
<meta name="DC.Subject"
content="Universitt,Kassel,Forschung,Lehre,Wissenschaft" />
<meta name="generator" content="TYPO3 4.2 CMS" />

But these tags don't appear in the segment as shown above. I thought
I'll find them in "Parse Metadata" but there are only this two values:
"CharEncodingForConversion=utf-8" "OriginalCharEncoding=utf-8"

I use the value parse-(html|tika) in my plugin.includes as well as urlmeta.

Any suggestions what I am doing wrong?

THANK YOU!

Snippet from segment dump:

Recno:: 97
URL:: http://cms.uni-kassel.de/unicms/index.php?id=29216&L=0

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:44:49 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:42:04 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:42:04 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:42:04 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 65 (signature)
Fetch time: Mon Dec 19 12:42:04 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 0.0
Signature: 7260839eaf4927f64b03dd86dcd0918a
Metadata:

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:42:04 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 67 (linked)
Fetch time: Mon Dec 19 12:42:51 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata:

CrawlDatum::
Version: 7
Status: 1 (db_unfetched)
Fetch time: Sat Dec 17 14:45:49 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 1
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata: _ngt_: 1324289811219_pst_: exception(16), lastModified=0:
java.net.SocketTimeoutException: Read timed out

CrawlDatum::
Version: 7
Status: 33 (fetch_success)
Fetch time: Mon Dec 19 12:25:59 CET 2011
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 1
Retry interval: 603450 seconds (6 days)
Score: 0.0
Signature: null
Metadata: _ngt_: 1324289811219_pst_: success(1), lastModified=0

Content::
Version: -1
url: http://cms.uni-kassel.de/unicms/index.php?id=29216&L=0
base: http://cms.uni-kassel.de/unicms/index.php?id=29216&L=0
contentType: application/xhtml+xml
metadata: Date=Mon, 19 Dec 2011 10:24:09 GMT Vary=Accept-Encoding
Content-Length=3886 Content-Encoding=gzip Via=1.0 cms.uni-kassel.de
_fst_=33 Set-Cookie=fe_typo_user=cb42ebddb40df7e8a04b0183f79c41cb;
path=/unicms/ nutch.segment.name=20111219111925
Content-Type=text/html;charset=utf-8 Connection=close
Server=Apache/2.2.3 (Debian) mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By=PHP/5.2.0-8+etch16
Content:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html
      PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="de">
(...)
</html>

ParseData::
Version: 5
Status: success(1,0)
Title: 2004 - Universitï¿œt Kassel
Outlinks: 35
   outlink: toUrl:
http://cms.uni-kassel.de/unicms/index.php?id=29216&L=0#navigation
anchor: Zur Hauptnavigation (Nutzergruppen-Navigation)
(...)
Content Metadata: Content-Length=3886 _fst_=33
Set-Cookie=fe_typo_user=cb42ebddb40df7e8a04b0183f79c41cb; path=/unicms/
nutch.segment.name=20111219111925 Connection=close Server=Apache/2.2.3
(Debian) mod_ssl/2.2.3 OpenSSL/0.9.8c X-Powered-By=PHP/5.2.0-8+etch16
nutch.content.digest=7260839eaf4927f64b03dd86dcd0918a Date=Mon, 19 Dec
2011 10:24:09 GMT Vary=Accept-Encoding Content-Encoding=gzip Via=1.0
cms.uni-kassel.de Content-Type=text/html;charset=utf-8
Parse Metadata: CharEncodingForConversion=utf-8 OriginalCharEncoding=utf-8

ParseText::
2004 - Universitï¿œt Kassel Zur Hauptnavigation (Nutzergruppen-Navigation)
. Zur Unternavigation . Zum Inhalt . Zu verwandten Links und
Informationen . Infos fï¿œr: Universitï¿œt Studium Forschung Fachbereiche
Einrichtungen International students and scholars     Sie befinden sich
hier:  HFK  >  Ehemalige Mitarbeiter  >  Frï¿œchting  >  Liste der
Verï¿œffentlichungen  >  2004 Verï¿œffentlichungen im Fachgebiet
Hochfrequenztechnik/Kommunikationssysteme 2004: [113] Semmelrodt, S.;
Kattenbach, R.; Frï¿œchting, H.: Toolbox for Spectral Analysis and Linear
Prediction of Stationary and Non-Stationary Signals, COST 273 TD(04)019,
Athen, Greece, January 26-28, 2004 [114] Semmelrodt, S.: Maximum
Likelihood Based Parameter Estimation of Stationary and Non-Stationary
Multi-Component Signals, FREQUENZ 58 (2004) 1-2, S. 20-24. [115]
Semmelrodt, S.: Methoden zur prï¿œdiktiven Kan

Hi Marek,
Thanks for sharing the useful info.
Actually I am facing the same issue which you have faced and solved :)
But after fetching meta tags information from the html documents, I want to
store them in mysql-database.
I don't want to create any index.

I am beginner in using nutch So don't know much of the stuff.
Can you please tell me where should i modify the code or can I use some king
of plugin functionality so that i need not recompile the code.

Also, Can i extract some other tag information using the above mentioned
idea.

I am using apache 1.4

Thanks

--
View this message in context: http://lucene.472066.n3.nabble.com/Meta-Tags-tp3598549p3838746.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi,

On Mon, Mar 19, 2012 at 11:23 AM, blunderboy <[EMAIL PROTECTED]>wrote:

> But after fetching meta tags information from the html documents, I want to
> store them in mysql-database.
> I don't want to create any index.
>

You should use Nutchgora branch, which you can check out from here

http://svn.apache.org/repos/asf/nutch/branches/nutchgora/

There are some patches which need to be applied so please google Nutchgora,
try Salmon Run for a recent series of blog posts on the topic. N.B You need
to use Gora 0.1.1-incubating release to use with MySql
>
> I am beginner in using nutch So don't know much of the stuff.
> Can you please tell me where should i modify the code or can I use some
> king
> of plugin functionality so that i need not recompile the code.
>

I'm afraid that in order to do what you wish you need to get clued up with
the Nutchgora branch and be able to edit code and recompile the project
with dependencies etc.

>
> Also, Can i extract some other tag information using the above mentioned
> idea.
>

I don't see why not!!!

hth

Hi Lewis,
Thanks for your solution. I have another idea.
We fetch meta tags and create Solr index.
Now can we write a separate java program which makes use of lucene library
which query the index and store the result in mysql database.
I know i am going to do more stuff than actually required but it is apt for
my need.
So is it possible ?

--
View this message in context: http://lucene.472066.n3.nabble.com/Meta-Tags-tp3598549p3841495.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi,

CC'ing in dev@gora

This second part of your question is prime Gora territory.

On Tue, Mar 20, 2012 at 6:30 AM, blunderboy <[EMAIL PROTECTED]> wrote:

> Hi Lewis,
> Thanks for your solution. I have another idea.
> We fetch meta tags and create Solr index.
> Now can we write a separate java program which makes use of lucene library
> which query the index and store the result in mysql database.
>

We have a pending patch [0] which would allow you to do a whole bunch of
analytics and processing of your Solr data. There are some problems but
watch this space. Currently what Gora supports is an API which enables you
to persist your data in MySQL or HSQL, HBase and Cassandra then run power
ful queries and processing jobs against the store, you could then persist
the results in MySQL. If you were going to write the Lucene code you would
maybe be best looking into the Gora API.
> I know i am going to do more stuff than actually required but it is apt for
> my need.
> So is it possible ?
>

hth

Lewis
[0]  https://issues.apache.org/jira/browse/GORA-9

--
*Lewis*

Hi lewis,
After lots of efforts, finally i was able to index up the meta tag in solR.
Thanks a lot. Is there any other plugin by which I can extract div tags also
?
just like index-metatags

Searching on google gave me some hint to use productdiv plugin.
Is it the right option ?

--
View this message in context: http://lucene.472066.n3.nabble.com/Meta-Tags-tp3598549p3845285.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi,

Well I would urge you to have a look at NUTCH-978
https://issues.apache.org/jira/browse/NUTCH-978

This issue can probably be adapted to suit your needs? It's open as a
Google Summer of Code project if you are interested in applying. 'Most' of
the code is alrwady there by the looks of the patch that Ammar has
attached, however I think there is some considerable updating and refining
to be done with it before it can even be considered for integration into
current development.

What do you think?

Lewis

On Wed, Mar 21, 2012 at 11:04 AM, blunderboy <[EMAIL PROTECTED]>wrote:

> Hi lewis,
> After lots of efforts, finally i was able to index up the meta tag in solR.
> Thanks a lot. Is there any other plugin by which I can extract div tags
> also
> ?
> just like index-metatags
>
> Searching on google gave me some hint to use productdiv plugin.
> Is it the right option ?
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Meta-Tags-tp3598549p3845285.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

--
*Lewis*