Hi,
i get the following error message when I try to parse a csv file: Can't
retrieve Tika parser for mime-type text/csv...
I use nutch 1.4 and solr 3.6...

The parsechecker gives the same message:
bin/nutch parsechecker http://dsiwikis/documents/forms/open_source_decls.csv
fetching: http://dsiwikis/documents/forms/open_source_decls.csv
parsing: http://dsiwikis/documents/forms/open_source_decls.csv
contentType: text/csv
---------
Url
---------------
http://dsiwikis/documents/forms/open_source_decls.csv---------
ParseData
---------
Version: 5
Status: failed(2,0): Can't retrieve Tika parser for mime-type text/csv
Title:
Outlinks: 0
Content Metadata:
Parse Metadata:

My parse-plugins.xml file contains:
        <mimeType name="text/csv">
                <plugin id="parse-tika" />
        </mimeType>

and my nutch-default.xml contains:
<property>
  <name>plugin.includes</name>
 
<value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|sc
oring-opic|urlnormalizer-(pass|regex|basic)</value>
  <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,
  and basic indexing and search plugins. In order to use HTTPS please enable
  protocol-httpclient, but be aware of possible intermittent problems with
the
  underlying commons-httpclient library.
  </description>
</property>

I searched the list and found something about this error but the thread
changed from direction and provided no answer to the original problem...
Any idea?
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-retrieve-Tika-parser-for-mime-type-text-csv-tp3990071.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Up, please

--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-retrieve-Tika-parser-for-mime-type-text-csv-tp3990071p3990521.html
Sent from the Nutch - User mailing list archive at Nabble.com.

I can't get the URL with the parserchecker

Have you tried application/csv instead?

I would have thought parse-tika would have dealt with this anyway...

On Wed, Jun 20, 2012 at 1:59 PM, Olivier LEVILLAIN
<[EMAIL PROTECTED]> wrote:
> Up, please
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Can-t-retrieve-Tika-parser-for-mime-type-text-csv-tp3990071p3990521.html
> Sent from the Nutch - User mailing list archive at Nabble.com.

--
Lewis

Well, I didn't choose text/csv.

Actually, I do not understand how nutch chooses the mime type. For instance,
for an rtf file, sometimes, it takes text/rtf and sometimes
application/rtf.... (in the same context exactly)

Is there a way to manually map file termination to mime types with Nutch?
I saw that tika was providing a tika-mimetypes.xml file for this but I can't
find it on my (stanadrd) installation...

--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-retrieve-Tika-parser-for-mime-type-text-csv-tp3990071p3990527.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi Oliver,

On Wed, Jun 20, 2012 at 2:29 PM, Olivier LEVILLAIN
<[EMAIL PROTECTED]> wrote:
>
> Actually, I do not understand how nutch chooses the mime type. For instance,
> for an rtf file, sometimes, it takes text/rtf and sometimes
> application/rtf.... (in the same context exactly)

Do you have an example of this? I have not looked into any Nutch
parsing plugins being inconsistent with mimeTypes but if some given
page's mimeType is being interpreted inconsistently by our parsing
plugins then this is not good... it would be great to see some
evidence of this...

>
> Is there a way to manually map file termination to mime types with Nutch?
> I saw that tika was providing a tika-mimetypes.xml file for this but I can't
> find it on my (stanadrd) installation...

In Nutch we do things slightly differently than they do over in Tika
w.r.t these config files.
As you correctly pointed out the parse-plugins.xml allows us to
explicitly order preference for parsers --> mimeTypes. I am confused
that there is no dedicated Tika parser for
org.apache.tika.parser.csv... my initial guess would be that it is
maybe implemented somewhere as an extension of plain text but I would
really need someone to back me up on this one.
I suppose the other option is to implement a custom parse-csv plugin
using something like the Tika-CSV-Parser [0], it seems to look very
easy to use. Then simply replace the generated tika artifact with your
new library...

hth

[0] https://github.com/msalgado/Tika-CSV-Parser

Hi Lewis and thanks for your answers,
Unfortunatley, the logs contening the traces of the double rtf mime types
have been erased.
Nevertheless, my parse-plugins.xml contains:
       <mimeType name="text/rtf">
                <plugin id="parse-tika" />
        </mimeType>
        <mimeType name="application/rtf">
                <plugin id="parse-tika" />
        </mimeType>
becquse nutch complained that first it didn't find text/rtf and the second
time it didn't find application/rtf

I'll take a look at your solution with Tika-CSV-parser but it's a shame it
does not come out of the box with tika/nutch, but it seems to imply
recompiling things and so on...
In the meantime, what would be the simple trick to parse csv as plain text?
Which *existing* parser should I use?
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-retrieve-Tika-parser-for-mime-type-text-csv-tp3990071p3990658.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi Oliver,

I think if you need more info on the Tika parser implementation for
csv you should head over to user@tika.a.o and see what folks there can
offer you.

On Thu, Jun 21, 2012 at 8:42 AM, Olivier LEVILLAIN
<[EMAIL PROTECTED]> wrote:
> Nevertheless, my parse-plugins.xml contains:
>       <mimeType name="text/rtf">
>                <plugin id="parse-tika" />
>        </mimeType>
>        <mimeType name="application/rtf">
>                <plugin id="parse-tika" />
>        </mimeType>

I thought we were talking about csv? The mime type definition for csv
and rtf is different...
This being said, the above config looks fine... if however it was not
there parse-tika would still automatically pick this up due to
wildcard default settings.

> becquse nutch complained that first it didn't find text/rtf and the second
> time it didn't find application/rtf

Strange. Tika certainly has an rtf parser implementation

>
> I'll take a look at your solution with Tika-CSV-parser but it's a shame it
> does not come out of the box with tika/nutch, but it seems to imply
> recompiling things and so on

AFAIK its a simple case of obtaining tika source making the necessary
changes, compiling the jar then putting this on your Nutch class path.
I don't envisage it to be too much hassle.

> In the meantime, what would be the simple trick to parse csv as plain text?
> Which *existing* parser should I use?

As you have highlighted it appears parse-tika is not working out tf
the box with this mimeType. I suggest you invest half an hour or so
and get the github csv parser working.

hth