hi...

i'm looking at a problem and i can't figure out how to "easily" solve it...

basically, i'm trying to figure out if there's a way to use lucene/nutch
with some form of pattern matching to extract course information from a
College/Registrar's course section...

Assume I can point to a Regiatrar's section of a College site.
Assume I can then crawl through the section, and capture
 all the underlying information, including the Course
 information...
Is there a way to somehow use pattern matching/recognition
 to somehow interpret the DOM to pull out the class schedule
 information. I'm pretty sure there's no vanilla approach,
 so I'd even consider some kind of solution where I might
 have to intially evaluate/analyze the site, to tell it
 what DOM elements are "important"...

anyone done any work/projects like this...
any research/papers/sample apps i could look at...
any thoughts/comments/etc....

i could brute force this by writing a bunch of perl
scripts, with each script tied to a given registrar site,
but i'd like a more generalizable solution if one exists..

thanks

-bruce

---------------------------------------------------------------------

Check out Andrew McCallum's paper:

http://www.cs.umass.edu/~mccallum/papers/acm-queue-ie.pdf

It mentions this very problem.  There are
also some more technical presentations around.

He was part of the Whiz-Bang team that took
on the problem.  The fact that the company's
out of business is a testament to how hard
this problem is in general.

- Bob Carpenter
   Alias-i

>
> i'm looking at a problem and i can't figure out how to "easily" solve it...
>
> basically, i'm trying to figure out if there's a way to use lucene/nutch
> with some form of pattern matching to extract course information from a
> College/Registrar's course section...
>
> Assume I can point to a Regiatrar's section of a College site.
> Assume I can then crawl through the section, and capture
>  all the underlying information, including the Course
>  information...
> Is there a way to somehow use pattern matching/recognition
>  to somehow interpret the DOM to pull out the class schedule
>  information. I'm pretty sure there's no vanilla approach,
>  so I'd even consider some kind of solution where I might
>  have to intially evaluate/analyze the site, to tell it
>  what DOM elements are "important"...
>
> anyone done any work/projects like this...
> any research/papers/sample apps i could look at...
> any thoughts/comments/etc....
>
> i could brute force this by writing a bunch of perl
> scripts, with each script tied to a given registrar site,
> but i'd like a more generalizable solution if one exists..
>
> thanks
>
> -bruce
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

You might also check out an old paper by Kruger, Giles, Lawrence et al. on
a search engine called Deadliner (see here at
http://clgiles.ist.psu.edu/papers/CIKM-2000-deadliner.pdf).
Deadliner crawled for Calls for Papers for conferences, using Support
Vector Machines trained to
recognise relevant pages, and then applying sets of regular expressions
to extract information
from the CFP pages.  Lawrence is now with Google, I believe.

Hope this helps,

Simon
Bob Carpenter wrote:
> Check out Andrew McCallum's paper:
>
> http://www.cs.umass.edu/~mccallum/papers/acm-queue-ie.pdf
>
> It mentions this very problem.  There are
> also some more technical presentations around.
>
> He was part of the Whiz-Bang team that took
> on the problem.  The fact that the company's
> out of business is a testament to how hard
> this problem is in general.
>
> - Bob Carpenter
>   Alias-i
>
>>
>> i'm looking at a problem and i can't figure out how to "easily" solve
>> it...
>>
>> basically, i'm trying to figure out if there's a way to use lucene/nutch
>> with some form of pattern matching to extract course information from a
>> College/Registrar's course section...
>>
>> Assume I can point to a Regiatrar's section of a College site.
>> Assume I can then crawl through the section, and capture
>>  all the underlying information, including the Course
>>  information...
>> Is there a way to somehow use pattern matching/recognition
>>  to somehow interpret the DOM to pull out the class schedule
>>  information. I'm pretty sure there's no vanilla approach,
>>  so I'd even consider some kind of solution where I might
>>  have to intially evaluate/analyze the site, to tell it
>>  what DOM elements are "important"...
>>
>> anyone done any work/projects like this...
>> any research/papers/sample apps i could look at...
>> any thoughts/comments/etc....
>>
>> i could brute force this by writing a bunch of perl
>> scripts, with each script tied to a given registrar site,
>> but i'd like a more generalizable solution if one exists..
>>
>> thanks
>>
>> -bruce
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Dr. Simon Courtenage
Software Systems Engineering Research Group
Dept. of Software Engineering, Cavendish School of Computer Science
University of Westminster, London, UK
Email: [EMAIL PROTECTED]   Web: http://users.cscs.wmin.ac.uk/~courtes | http://www.sse.wmin.ac.uk
---------------------------------------------------------------------

hi simon....

like a hole in my head!!!!

what i really need is a way to recursively iterate through a site, and to to
be able to selectively iterate through the 'form' elements on a given page.
ie, if i visually analyze a site and determine that the 1st level (page) has
a form, and i need to set the 1st two elements of the form, i'd like to be
able to somehow perform this function when i crawl through the site as
opposed to every potential combination of the form, or as opposed to only
using the default for the form...

i'd also like to be able to handle forms at lower levels in a similar
manner. this would/should allow the app to return all the required pages.

i could then create the parser(s) to extract information from each page.
although it would be better to actually have the ability to somehow manage
the data extraction from the DOM within the crawling application...

-bruce
-----Original Message-----
From: Simon Courtenage [mailto:[EMAIL PROTECTED]]
Sent: Thursday, June 22, 2006 2:08 PM
To: [EMAIL PROTECTED]
Subject: Re: lucene in combination with pattern recognition...
You might also check out an old paper by Kruger, Giles, Lawrence et al. on
a search engine called Deadliner (see here at
http://clgiles.ist.psu.edu/papers/CIKM-2000-deadliner.pdf).
Deadliner crawled for Calls for Papers for conferences, using Support
Vector Machines trained to
recognise relevant pages, and then applying sets of regular expressions
to extract information
from the CFP pages.  Lawrence is now with Google, I believe.

Hope this helps,

Simon
Bob Carpenter wrote:
> Check out Andrew McCallum's paper:
>
> http://www.cs.umass.edu/~mccallum/papers/acm-queue-ie.pdf
>
> It mentions this very problem.  There are
> also some more technical presentations around.
>
> He was part of the Whiz-Bang team that took
> on the problem.  The fact that the company's
> out of business is a testament to how hard
> this problem is in general.
>
> - Bob Carpenter
>   Alias-i
>
>>
>> i'm looking at a problem and i can't figure out how to "easily" solve
>> it...
>>
>> basically, i'm trying to figure out if there's a way to use lucene/nutch
>> with some form of pattern matching to extract course information from a
>> College/Registrar's course section...
>>
>> Assume I can point to a Regiatrar's section of a College site.
>> Assume I can then crawl through the section, and capture
>>  all the underlying information, including the Course
>>  information...
>> Is there a way to somehow use pattern matching/recognition
>>  to somehow interpret the DOM to pull out the class schedule
>>  information. I'm pretty sure there's no vanilla approach,
>>  so I'd even consider some kind of solution where I might
>>  have to intially evaluate/analyze the site, to tell it
>>  what DOM elements are "important"...
>>
>> anyone done any work/projects like this...
>> any research/papers/sample apps i could look at...
>> any thoughts/comments/etc....
>>
>> i could brute force this by writing a bunch of perl
>> scripts, with each script tied to a given registrar site,
>> but i'd like a more generalizable solution if one exists..
>>
>> thanks
>>
>> -bruce
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Dr. Simon Courtenage
Software Systems Engineering Research Group
Dept. of Software Engineering, Cavendish School of Computer Science
University of Westminster, London, UK
Email: [EMAIL PROTECTED]   Web: http://users.cscs.wmin.ac.uk/~courtes |
http://www.sse.wmin.ac.uk
---------------------------------------------------------------------
--------------------------------------------