Tika, mail # user - Problem extracting date from Outlook 2007 .msg file

Problem extracting date from Outlook 2007 .msg file

Joe Wicentowski 2012-06-25, 20:14

Hi all,

Hello! This is my message to the list. I'm building an application
that uses Tika to extract text from Outlook 2007 .msg files, among
other things. While experimenting with some sample .msg files, I
noticed that Tika is failing not returning the date of most messages.
For example, Outlook indicates that the following message was sent on
"Fri 6/22/2012 8:11 AM", but no date appears in the HTML head or in
the early portion of the body of the Tika output [1]. I retrieved
this using Tika 1.1 on Windows XP using the following command:

  java -jar tika-app-1.1.jar "C:\Documents and
Settings\wicentowskijc\Desktop\portal\outlook\RE Inquiry.msg" >
inquiry.html

If anyone has suggestions for ensuring that the date can be preserved
in Tika's output, I'd be grateful.

Thanks,
Joe
[1] Tika output showing no date

<html xmlns="http://www.w3.org/1999/xhtml">
   <head>
   <meta name="Message-Bcc" content="" />
   <meta name="subject" content="Inquiry" />
   <meta name="Content-Length" content="40960" />
   <meta name="Message-Recipient-Address" content="[EMAIL PROTECTED]" />
   <meta name="Message-From" content="History Mailbox" />
   <meta name="Author" content="History Mailbox" />
   <meta name="Message-To" content="'Snip'" />
   <meta name="Message-Cc" content="" />
   <meta name="Content-Type" content="application/vnd.ms-outlook" />
   <meta name="resourceName" content="RE Inquiry.msg" />
   </head>
   <body>
   <h1>RE: Inquiry</h1>
   <dl>
   <dt>From</dt>
   <dd>History Mailbox</dd>
   <dt>To</dt>
   <dd>'Snip'</dd>
   <dt>Recipients</dt>
   <dd>[EMAIL PROTECTED]</dd>
   </dl>
   <p>Dear Snip</p>
...

Re: Problem extracting date from Outlook 2007 .msg file

Nick Burch 2012-06-26, 23:29

On 25/06/12 21:14, Joe Wicentowski wrote:
> Hello! This is my message to the list. I'm building an application
> that uses Tika to extract text from Outlook 2007 .msg files, among
> other things. While experimenting with some sample .msg files, I
> noticed that Tika is failing not returning the date of most messages.
> For example, Outlook indicates that the following message was sent on
> "Fri 6/22/2012 8:11 AM", but no date appears in the HTML head or in
> the early portion of the body of the Tika output [1]. I retrieved
> this using Tika 1.1 on Windows XP using the following command:

Did you try with --metadata?

Also, are you sure that the messages contain the dates? Some kinds of
outlook files don't...

Nick

Re: Problem extracting date from Outlook 2007 .msg file

Joe Wicentowski 2012-06-27, 00:19

Hi Nick,

Thanks so much for your reply.

>> While experimenting with some sample .msg files, I
>> noticed that Tika is failing not returning the date of most messages.
>> For example, Outlook indicates that the following message was sent on
>> "Fri 6/22/2012 8:11 AM", but no date appears in the HTML head or in
>> the early portion of the body of the Tika output [1]. I retrieved
>> this using Tika 1.1 on Windows XP using the following command:
>
> Did you try with --metadata?

I ran tika with --metadata on the same message I mentioned in my first
email, and tika didn't output the message's date this way either.
Here are the results:

Author: PA History Mailbox
Content-Length: 40960
Content-Type: application/vnd.ms-outlook
Message-Bcc:
Message-Cc:
Message-From: History Mailbox
Message-Recipient-Address: [EMAIL PROTECTED]
Message-To: 'Snip'
resourceName: RE Inquiry.msg
subject: Inquiry
title: RE: Inquiry

> Also, are you sure that the messages contain the dates? Some kinds of
> outlook files don't...

This same message does show a date in Outlook ("Fri 6/22/2012 8:11
AM"). Do you know of some way to tell whether the date that appears
in Outlook is actually inside the message (versus stored elsewhere in
some sort of Outlook database)? (In other mail clients I would think
to look at the "mail headers" mode, but I don't recall seeing such a
mode in Outlook. Do you happen to know under what circumstances
Outlook would not include a date?

Tika does recognize dates in some of my sample messages, but
definitely this is the minority. In fact, tika only retrieved dates
for 3 of 47 messages. (Specifically, those 3 messages have the
following fields: date, Creation-Date, and Last-Save-Date.

Thanks for any suggestions,
Joe

Re: Problem extracting date from Outlook 2007 .msg file

Joe Wicentowski 2012-06-28, 17:03

Hi again,

I have an update regarding my report about Tika not recognizing the
date in an Outlook .msg files [1]. I tried using a different tool,
ruby-msg (http://code.google.com/p/ruby-msg/), to process the same
message as in my earlier email, and ruby-msg did pull out the date [2]
This experiment shows that the email *is* in the .msg file, and that
Tika is failing to pick it up.

Can anyone suggest the best way to proceed to improve Tika's handling
of dates in Outlook .msg files? I'll be happy to file a bug report,
but I'm just not sure whether this is an issue in Tika itself or in
one of Tika's dependencies.

Thanks,
Joe
[1] The Tika output, quoting from my last email:

> Author: PA History Mailbox
> Content-Length: 40960
> Content-Type: application/vnd.ms-outlook
> Message-Bcc:
> Message-Cc:
> Message-From: History Mailbox
> Message-Recipient-Address: [EMAIL PROTECTED]
> Message-To: 'Snip'
> resourceName: RE Inquiry.msg
> subject: Inquiry
> title: RE: Inquiry

[2] The ruby-msg output -- notice the "Date:" line:

From: "History Mailbox" <[EMAIL PROTECTED]>
To: "Snip" <[EMAIL PROTECTED]>
Subject: RE: Inquiry
Date: Fri, 22 Jun 2012 12:11:00 -0000
Message-ID: <[EMAIL PROTECTED]>
In-Reply-To: <CAJ4nNe1FPo7Q=10dbK8sdzPRaRzYKJV6SKV3nyg5L2Li13b+[EMAIL PROTECTED]>
Priority: 0
Thread-Topic: Inquiry
Content-Type: multipart/alternative;
boundary="----_=_NextPart_001_8149ed38.4fec8c61"

Re: Problem extracting date from Outlook 2007 .msg file

Joe Wicentowski 2012-06-28, 17:10

Hi again,

Incidentally, when I run the output of the ruby-msg through Tika, Tika
does get the date:

<meta name="date" content="2012-06-22T00:11:00Z"/>

I guess I could use ruby-msg to pre-process Tika, but that defeats the
purpose of an all-in-one tool like Tika.

Joe

> I have an update regarding my report about Tika not recognizing the
> date in an Outlook .msg files [1]. I tried using a different tool,
> ruby-msg (http://code.google.com/p/ruby-msg/), to process the same
> message as in my earlier email, and ruby-msg did pull out the date [2]
> This experiment shows that the email *is* in the .msg file, and that
> Tika is failing to pick it up.

Re: Problem extracting date from Outlook 2007 .msg file

Nick Burch 2012-06-28, 18:51

On Thu, 28 Jun 2012, Joe Wicentowski wrote:
> I have an update regarding my report about Tika not recognizing the date
> in an Outlook .msg files [1]. I tried using a different tool, ruby-msg
> (http://code.google.com/p/ruby-msg/), to process the same message as in
> my earlier email, and ruby-msg did pull out the date [2] This experiment
> shows that the email *is* in the .msg file, and that Tika is failing to
> pick it up.

That suggests that it's stored in a different bit of the file (a different
stream) to the ones we're expecting to find it in. The file format is
documented, so you can look up what each different bit means, but there
are a lot of duplicate fields for historical reasons. What we lack is a
guide saying "outlook 200x stores the sent date as MAPI_???_DATE, while
200y uses OUTLOOK_DATE_MAPI_???_V3"

What'd be great is if you could use org.apache.poi.hsmf.dev.HSMFDump
(contained within the poi-scratchpad jar, dependency on the main poi jar
but I don't think anything else) to try to track down which chunk contains
the date. You might need to combine that with a little bit of hacking of
your ruby script, to have it print some debug logging of what fields it's
printing from

Once we know the field, we can look up the details on how it's stored,
then add a fallback check of that field/chunk too

Nick

Re: Problem extracting date from Outlook 2007 .msg file

Joe Wicentowski 2012-06-28, 19:37

Hi Nick,

> That suggests that it's stored in a different bit of the file (a different
> stream) to the ones we're expecting to find it in. The file format is
> documented, so you can look up what each different bit means, but there are
> a lot of duplicate fields for historical reasons. What we lack is a guide
> saying "outlook 200x stores the sent date as MAPI_???_DATE, while 200y uses
> OUTLOOK_DATE_MAPI_???_V3"

I see. This makes sense.

> What'd be great is if you could use org.apache.poi.hsmf.dev.HSMFDump
> (contained within the poi-scratchpad jar, dependency on the main poi jar but
> I don't think anything else) to try to track down which chunk contains the
> date. You might need to combine that with a little bit of hacking of your
> ruby script, to have it print some debug logging of what fields it's
> printing from
>
> Once we know the field, we can look up the details on how it's stored, then
> add a fallback check of that field/chunk too

Sounds promising. Is there a command line way to run the HSMFDump
component of the poi-scratchpad.jar on my .msg file? If so, could you
give me a pointer? If not, I fear this may be getting beyond my
abilities, as I'm not a java programmer. (I'm generally
unix-literate, work with Mac OS X; in terms of programming, I use
XQuery with eXist-db, which has incorporates Tika as an extension
module.) I'm happy to provide the .msg file in question off list, if
it would help, but I'd understand if you aren't able to help to that
extent.

Thanks,
Joe

Re: Problem extracting date from Outlook 2007 .msg file

Nick Burch 2012-06-28, 20:19

On Thu, 28 Jun 2012, Joe Wicentowski wrote:
> Sounds promising. Is there a command line way to run the HSMFDump
> component of the poi-scratchpad.jar on my .msg file? If so, could you
> give me a pointer?

Something like this should work:
java -classpath poi-3.8-FINAL.jar:poi-scratchpad-3.8-FINAL.jar \
org.apache.poi.hsmf.dev.HSMFDump <problem.msg>

Nick