Hi all

We have been using Tika to parse iWork files, but have found many
limitations and potentially bugs. Here is a sampling:
* Things like header and footer text and embedded text boxes are not
parsed.
* Pages docs created in Layout mode are not parsed at all. Only the
metadata is extracted.
* Text box text in Keynote slides is extracted, but all of the text of all
the boxes is lumped together without any spaces.
* Password protected files throw an NPE.

Is there any work in progress or planned to improve the parsing of iWork
files? Or only as defects are opened?
--
Gabriel Valencia
Software Development for IBM Content Integrator, IBM Content Analytics, and
IBM Content and Predictive Analytics
[EMAIL PROTECTED]
Tel: 408-463-4133 TL: 543-4133

Hi Gabriel,

Thanks for bringing these issues to light.

We would appreciate you filing issues in our JIRA issue tracker:

http://issues.apache.org/jira/browse/TIKA

And if you are able to attach sample files that we can use to reproduce
what you are seeing that would be great. The Tika version you're using
would also be helpful.

Thank you!

Cheers,
Chris

On Apr 24, 2012, at 4:52 PM, Gabriel Valencia wrote:

> Hi all
>
> We have been using Tika to parse iWork files, but have found many limitations and potentially bugs. Here is a sampling:
> * Things like header and footer text and embedded text boxes are not parsed.
> * Pages docs created in Layout mode are not parsed at all. Only the metadata is extracted.
> * Text box text in Keynote slides is extracted, but all of the text of all the boxes is lumped together without any spaces.
> * Password protected files throw an NPE.
>
> Is there any work in progress or planned to improve the parsing of iWork files? Or only as defects are opened?
> --
> Gabriel Valencia
> Software Development for IBM Content Integrator, IBM Content Analytics, and IBM Content and Predictive Analytics
> [EMAIL PROTECTED]
> Tel: 408-463-4133 TL: 543-4133
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Tue, 24 Apr 2012, Gabriel Valencia wrote:
> We have been using Tika to parse iWork files, but have found many
> limitations and potentially bugs.

At the moment, there aren't any suitable iWorks file format libraries in
Java for us to use. So, what we do have has come from community
contributions. There's a very strong chance that it'll only handle the
kinds of files that the people who volunteered to work on it had!

(The parsers all work by doing SAX parsing of the main content of the
iWorks file, mapping from parts of the file's XML to XHTML as possible)

> Here is a sampling:
> * Things like header and footer text and embedded text boxes are not
> parsed.
> * Pages docs created in Layout mode are not parsed at all. Only the
> metadata is extracted.
> * Text box text in Keynote slides is extracted, but all of the text of all
> the boxes is lumped together without any spaces.

As a first step, are you able to produce some simple sample files that
show these different document features? If so, please create new bugs in
JIRA and upload them.

Step two is probably to write some (initially failing) unit tests to show
what text you'd expect to get out

Step three is to unzip your sample files (iWorks are a zip of xml +
images), and hunt around for your missing text. For each case, try to work
out where in the file structure the text is held, and if it's inline with
the surrounding content or stored off elsewhere with a pointer

Finally, step four is to update the parsers based on this information,
and upload the patch!

> * Password protected files throw an NPE.

Are you able to check if these use Zip based password protection, or
something else? A sample file that's small and has a known password would
be handy

> Is there any work in progress or planned to improve the parsing of iWork
> files?

I think you may have just volunteered yourself ;-)

Cheers
Nick