I've run into a number of documents that cause Tika to hang. This is  
especially true when I'm only fetching the first 8K bytes, so the  
parsers are often dealing with an incomplete document.

Unfortunately when Tika hangs this causes my Hadoop job to hang, with  
an eventual timeout/retry, and finally a failed job.

My solution has been to create a TikaCallable that wraps the Tika  
parser, and then use this with a FutureTask so that I can call  
task.get() with a timeout:

             Callable<ParsedDatum> c = new TikaCallable(_parser,  
inputstream, metadata);
             FutureTask<ParsedDatum> task = new  
FutureTask<ParsedDatum>(c);
             Thread t = new Thread(task);
             t.start();

             ParsedDatum result = task.get(MAX_PARSE_DURATION,  
TimeUnit.SECONDS);

This seems like it would be generally useful, as I doubt that we'd  
ever be able to guarantee that none of the parsers being wrapped by  
Tika could ever hang.

One idea is to create a TimeoutParser that wraps a regular Tika  
Parser. E.g. something like:

  Parser p = new TimeoutParser(new AutodetectParser(), 20,  
TimeUnit.SECONDS);

Then the call to p.parse(...) would create a Callable (siimilar to the  
code above) and use the specified timeout when calling task.get().

Any thoughts on the value of this support, and input on the approach?

Thanks,

-- Ken

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g

Hi,

On Mon, Jan 25, 2010 at 10:57 PM, Ken Krugler
<[EMAIL PROTECTED]> wrote:
> Any thoughts on the value of this support, and input on the approach?

This would certainly be useful and something I've been hoping to focus
more on (see the "security" point in
http://markmail.org/message/ggihw2cns53t6ayl from 2007 :-).

The biggest problem with the FutureTask approach is that it's quite
difficult if not impossible to support proper streaming with it. One
possible approach that would avoid this problem is to modify the
ParsingReader class to include timeout support in the pipe it uses.
Modifying that approach to support the full SAX event stream should be
doable, though not necessarily trivial.

BR,

Jukka Zitting

Hi Jukka,

>> Any thoughts on the value of this support, and input on the approach?
>
> This would certainly be useful and something I've been hoping to focus
> more on (see the "security" point in
> http://markmail.org/message/ggihw2cns53t6ayl from 2007 :-).
>
> The biggest problem with the FutureTask approach is that it's quite
> difficult if not impossible to support proper streaming with it.

Could you provide more details about the problems here?

E.g. correctly interrupting the read requests when a timeout happens?  
Or something else.

Thanks,

-- Ken

> One
> possible approach that would avoid this problem is to modify the
> ParsingReader class to include timeout support in the pipe it uses.
> Modifying that approach to support the full SAX event stream should be
> doable, though not necessarily trivial.
>
> BR,
>
> Jukka Zitting

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g

Hi,

On Tue, Jan 26, 2010 at 10:56 PM, Ken Krugler
<[EMAIL PROTECTED]> wrote:
>> The biggest problem with the FutureTask approach is that it's quite
>> difficult if not impossible to support proper streaming with it.
>
> Could you provide more details about the problems here?

The main problem is that in the streaming case there is no single
ParsedDatum instance that would be the result of the parsing process.
Instead you have a sequence of SAX events that you should be able to
deliver to the client application as soon as they become available
without waiting for all the rest of the events to arrive first.

Instead of a FutureTask, I'd envision us using a BlockingQueue or
something similar to insert timeout handling between the parser that
generates the SAX events from the client that consumes them.

BR,

Jukka Zitting