Hi guys,

Just wondering about the best way to make the language detection pluggable
instead of having it hard-wired as it is now. We now that the resources
that are currently in Tika are both slow and inaccurate [1] and there are
other libraries that we could leverage. Why not having the option to select
a different implementation just like we do for parsers? Obviously we'd need
a common interface for the parsers etc...

What do you think?

Julien

[1]
http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html

--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

On Mar 21, 2012, at 8:51am, Julien Nioche wrote:

> Hi guys,
>
> Just wondering about the best way to make the language detection pluggable
> instead of having it hard-wired as it is now. We now that the resources
> that are currently in Tika are both slow and inaccurate [1] and there are
> other libraries that we could leverage. Why not having the option to select
> a different implementation just like we do for parsers? Obviously we'd need
> a common interface for the parsers etc...
>
> What do you think?

I'd be more in favor of using that time to integrate a better language detector into Tika, so that everybody wins from the work :)

-- Ken
> [1]
> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

On Wed, Mar 21, 2012 at 12:55 PM, Ken Krugler
<[EMAIL PROTECTED]> wrote:
>
> On Mar 21, 2012, at 8:51am, Julien Nioche wrote:
>
>> Hi guys,
>>
>> Just wondering about the best way to make the language detection pluggable
>> instead of having it hard-wired as it is now. We now that the resources
>> that are currently in Tika are both slow and inaccurate [1] and there are
>> other libraries that we could leverage. Why not having the option to select
>> a different implementation just like we do for parsers? Obviously we'd need
>> a common interface for the parsers etc...
>>
>> What do you think?
>
> I'd be more in favor of using that time to integrate a better language detector into Tika, so that everybody wins from the work :)

+1

Mike McCandless

http://blog.mikemccandless.com

If you mean integrating a better third-party detector - that's exactly my
point. We don't develop and maintain our own parsers, why should we follow
a different logic when it comes to language identification? There are other
resource around why don't we just use them? I assume that by default our
existing detector (improved or not) could still be used, all we need is
just a mechanism to be able to select an alternative implementation and a
common interface. That's probably not a big deal to implement. Any thoughts
on how to do it? Are there any things we should reuse from the way we deal
with the parsers?

Thanks for your comments

Julien
On 21 March 2012 16:55, Ken Krugler <[EMAIL PROTECTED]> wrote:

>
> On Mar 21, 2012, at 8:51am, Julien Nioche wrote:
>
> > Hi guys,
> >
> > Just wondering about the best way to make the language detection
> pluggable
> > instead of having it hard-wired as it is now. We now that the resources
> > that are currently in Tika are both slow and inaccurate [1] and there are
> > other libraries that we could leverage. Why not having the option to
> select
> > a different implementation just like we do for parsers? Obviously we'd
> need
> > a common interface for the parsers etc...
> >
> > What do you think?
>
> I'd be more in favor of using that time to integrate a better language
> detector into Tika, so that everybody wins from the work :)
>
> -- Ken
>
>
> > [1]
> >
> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>
>
--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

In Solr, we made support for pluggable lang detectors, one being Tika's. See http://svn.apache.org/viewvc/lucene/dev/trunk/solr/contrib/langid/
The detectLanguage() method returns a list of DetectedLanguage objects with a normalized certainty between 0.0 and 1.0. Think it's a step in right direction.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com

On 22. mars 2012, at 11:22, Julien Nioche wrote:

> If you mean integrating a better third-party detector - that's exactly my
> point. We don't develop and maintain our own parsers, why should we follow
> a different logic when it comes to language identification? There are other
> resource around why don't we just use them? I assume that by default our
> existing detector (improved or not) could still be used, all we need is
> just a mechanism to be able to select an alternative implementation and a
> common interface. That's probably not a big deal to implement. Any thoughts
> on how to do it? Are there any things we should reuse from the way we deal
> with the parsers?
>
> Thanks for your comments
>
> Julien
>
>
> On 21 March 2012 16:55, Ken Krugler <[EMAIL PROTECTED]> wrote:
>
>>
>> On Mar 21, 2012, at 8:51am, Julien Nioche wrote:
>>
>>> Hi guys,
>>>
>>> Just wondering about the best way to make the language detection
>> pluggable
>>> instead of having it hard-wired as it is now. We now that the resources
>>> that are currently in Tika are both slow and inaccurate [1] and there are
>>> other libraries that we could leverage. Why not having the option to
>> select
>>> a different implementation just like we do for parsers? Obviously we'd
>> need
>>> a common interface for the parsers etc...
>>>
>>> What do you think?
>>
>> I'd be more in favor of using that time to integrate a better language
>> detector into Tika, so that everybody wins from the work :)
>>
>> -- Ken
>>
>>
>>> [1]
>>>
>> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>>>
>>> --
>>> *
>>> *Open Source Solutions for Text Engineering
>>>
>>> http://digitalpebble.blogspot.com/
>>> http://www.digitalpebble.com
>>> http://twitter.com/digitalpebble
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble

Hi Jan,

It probably makes sense to provide pluggable language detection in Tika, since it's the lower level library,
so I am +1 for figuring out a solution to implement it in Tika ville.

If no one has started on this in the next few weeks I'll give it a go.

Cheers,
Chris

On Apr 8, 2012, at 4:16 PM, Jan Høydahl wrote:

> In Solr, we made support for pluggable lang detectors, one being Tika's. See http://svn.apache.org/viewvc/lucene/dev/trunk/solr/contrib/langid/
> The detectLanguage() method returns a list of DetectedLanguage objects with a normalized certainty between 0.0 and 1.0. Think it's a step in right direction.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
[...snip...]

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hey Juls,

I'd be super +1 to make it pluggable and willing to help.

Cheers,
Chris

On Mar 21, 2012, at 4:51 PM, Julien Nioche wrote:

> Hi guys,
>
> Just wondering about the best way to make the language detection pluggable
> instead of having it hard-wired as it is now. We now that the resources
> that are currently in Tika are both slow and inaccurate [1] and there are
> other libraries that we could leverage. Why not having the option to select
> a different implementation just like we do for parsers? Obviously we'd need
> a common interface for the parsers etc...
>
> What do you think?
>
> Julien
>
> [1]
> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hello!

21.03.2012 19:51, Julien Nioche пишет:
> Just wondering about the best way to make the language detection pluggable
> instead of having it hard-wired as it is now. We now that the resources
> that are currently in Tika are both slow and inaccurate [1] and there are
> other libraries that we could leverage. Why not having the option to select
> a different implementation just like we do for parsers? Obviously we'd need
> a common interface for the parsers etc...
>
> What do you think?
>

I think I would be nice to have this API since currently we are using patched Tika
with our own charset/language detector.

best wishes, Max