Hi,

I am crawling a SharePoint 2010 library named Documents. This library contains folders. These folders contains files.

I see that MCF treats folders ( and its underlying files) as single output documents. In other words MCF does not traverse folders.

It seems that ows_FSObjType="1" indicates folders and ows_FSObjType="0" indicates files.

How can I traverse folders in a SP library? Or it is a bad idea to put folders under a library?

Thanks,
Ahmet

Hello,

I am crawling a SharePoint 2010 library named Pages. This library contains *.aspx files only.

There is an if check in SPSProxyHelper.java that ignores aspx files.

if (!relPath.endsWith(".aspx")) {
     fileStream.addFile( relPath );
}

What is the motivation of this if clause?

If I disable this if check, I can successfully get content of *.aspx files and metadata (field values) that I selected for this Pages library.

Thanks,
Ahmet

This clause came directly from the patch you submitted.  I did not
understand the reasoning for it.  I'm happy to remove it.

Karl

On Thu, Aug 2, 2012 at 9:19 AM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I am crawling a SharePoint 2010 library named Pages. This library contains *.aspx files only.
>
> There is an if check in SPSProxyHelper.java that ignores aspx files.
>
> if (!relPath.endsWith(".aspx")) {
>      fileStream.addFile( relPath );
> }
>
> What is the motivation of this if clause?
>
> If I disable this if check, I can successfully get content of *.aspx files and metadata (field values) that I selected for this Pages library.
>
> Thanks,
> Ahmet

Hi Ahmet,

The DspSts service lists all the files in a library, including those
that have a folder path.  I believe the lists service does the same.
So you should see all the files crawled including those that are
within folders.  Please let me know if this seems not to be the case.

Karl

On Thu, Aug 2, 2012 at 9:14 AM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am crawling a SharePoint 2010 library named Documents. This library contains folders. These folders contains files.
>
> I see that MCF treats folders ( and its underlying files) as single output documents. In other words MCF does not traverse folders.
>
> It seems that ows_FSObjType="1" indicates folders and ows_FSObjType="0" indicates files.
>
> How can I traverse folders in a SP library? Or it is a bad idea to put folders under a library?
>
> Thanks,
> Ahmet

I removed this clause on trunk, from both the Dspsts result processing
and the Lists result processing.  If you encounter any problems with
this please let me know.

Karl

On Thu, Aug 2, 2012 at 9:24 AM, Karl Wright <[EMAIL PROTECTED]> wrote:
> This clause came directly from the patch you submitted.  I did not
> understand the reasoning for it.  I'm happy to remove it.
>
> Karl
>
> On Thu, Aug 2, 2012 at 9:19 AM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>> I am crawling a SharePoint 2010 library named Pages. This library contains *.aspx files only.
>>
>> There is an if check in SPSProxyHelper.java that ignores aspx files.
>>
>> if (!relPath.endsWith(".aspx")) {
>>      fileStream.addFile( relPath );
>> }
>>
>> What is the motivation of this if clause?
>>
>> If I disable this if check, I can successfully get content of *.aspx files and metadata (field values) that I selected for this Pages library.
>>
>> Thanks,
>> Ahmet

> The DspSts service lists all the files in a library,
> including those
> that have a folder path.  I believe the lists service
> does the same.
> So you should see all the files crawled including those that
> are
> within folders.  Please let me know if this seems not
> to be the case.

It seems that lists service does not lists files that have a folder path. Instead it lists folder paths. Currently MCF injects one SolrDocument per folder, and sends all files under it it to extracting update handler.

I was reading this : http://sympmarc.com/2011/03/28/listing-folders-in-a-sharepoint-list-or-library-with-spservices/

I am attaching a response example that contains 9 items ( 1 file and 8 folders)

> I removed this clause on trunk, from
> both the Dspsts result processing
> and the Lists result processing.  If you encounter any
> problems with
> this please let me know.

I believe I copied that if clause from Dspsts result processing section. I will continue with the mcf-trunk and will let you know if I encounter any problems.

Thanks!

In that case, you will need to wait until CONNECTORS-492 is resolved.
Because of SharePoint's lack of support for accessing large libraries
via the Lists service, we're having to write our own.  But this is not
yet ready, although we are getting closer to trying it out soon.

Karl

On Thu, Aug 2, 2012 at 9:57 AM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>> The DspSts service lists all the files in a library,
>> including those
>> that have a folder path.  I believe the lists service
>> does the same.
>> So you should see all the files crawled including those that
>> are
>> within folders.  Please let me know if this seems not
>> to be the case.
>
> It seems that lists service does not lists files that have a folder path. Instead it lists folder paths. Currently MCF injects one SolrDocument per folder, and sends all files under it it to extracting update handler.
>
> I was reading this : http://sympmarc.com/2011/03/28/listing-folders-in-a-sharepoint-list-or-library-with-spservices/
>
> I am attaching a response example that contains 9 items ( 1 file and 8 folders)

> In that case, you will need to wait
> until CONNECTORS-492 is resolved.
> Because of SharePoint's lack of support for accessing large
> libraries
> via the Lists service, we're having to write our own. 
> But this is not
> yet ready, although we are getting closer to trying it out
> soon.

How is this related to CONNECTORS-492? This is something alternative to your pagination solution (while loop)?

This replaces the getlistitems call in spsproxyhelper with a custom
method call.

Karl

Sent from my Windows Phone
From: Ahmet Arslan
Sent: 8/2/2012 3:25 PM
To: [EMAIL PROTECTED]
Subject: Re: SharePoint Library consist of folders
> In that case, you will need to wait
> until CONNECTORS-492 is resolved.
> Because of SharePoint's lack of support for accessing large
> libraries
> via the Lists service, we're having to write our own.
> But this is not
> yet ready, although we are getting closer to trying it out
> soon.

How is this related to CONNECTORS-492? This is something alternative
to your pagination solution (while loop)?

> This replaces the getlistitems call
> in spsproxyhelper with a custom
> method call.

Once 492 in place is it going to list files that have folder path too? Without checking value of ows_FSObjType attirbute?

Hello,

I found that there is an queryOptions parameter for this. <ViewAttributes Scope="Recursive" />.

http://msdn.microsoft.com/en-us/library/lists.lists.getlistitems.aspx

If I add these three lines to SPSProxyHelper#buildPagingQueryOptions()

MessageElement viewAttributesNode = new MessageElement((String)null,"ViewAttributes");
queryOptionsNode.addChild(viewAttributesNode);
viewAttributesNode.addAttribute(null,"Scope","Recursive");

return rval;

SPSProxyHelper#getDocuments() returns expected results:

/Documents/Vekaletname.pdf, /Documents/ik_docs/diger/diger_dilekceler/aile_yardimi_almaz_dilecesi.doc, /Documents/ik_docs/diger/fonksiyonel_ekipman_talep_formu.doc, ...

But SPSProxyHelper#getFieldValues() method works for only docId=/Documents/Vekaletname.pdf returns empty map for others. Therefore only this one injected.

--- On Thu, 8/2/12, Ahmet Arslan <[EMAIL PROTECTED]> wrote:

> From: Ahmet Arslan <[EMAIL PROTECTED]>
> Subject: RE: SharePoint Library consist of folders
> To: [EMAIL PROTECTED]
> Date: Thursday, August 2, 2012, 10:38 PM
> > This replaces the getlistitems
> call
> > in spsproxyhelper with a custom
> > method call.
>
> Once 492 in place is it going to list files that have folder
> path too? Without checking value of ows_FSObjType
> attirbute?
>
>
>

I checked this change into trunk, and added also corresponding code in
the place where fields and metadata are fetched.  This may work for
you in the interim while we're finishing up CONNECTORS-492.

Karl

On Fri, Aug 3, 2012 at 8:11 AM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I found that there is an queryOptions parameter for this. <ViewAttributes Scope="Recursive" />.
>
> http://msdn.microsoft.com/en-us/library/lists.lists.getlistitems.aspx
>
> If I add these three lines to SPSProxyHelper#buildPagingQueryOptions()
>
> MessageElement viewAttributesNode = new MessageElement((String)null,"ViewAttributes");
> queryOptionsNode.addChild(viewAttributesNode);
> viewAttributesNode.addAttribute(null,"Scope","Recursive");
>
> return rval;
>
> SPSProxyHelper#getDocuments() returns expected results:
>
> /Documents/Vekaletname.pdf, /Documents/ik_docs/diger/diger_dilekceler/aile_yardimi_almaz_dilecesi.doc, /Documents/ik_docs/diger/fonksiyonel_ekipman_talep_formu.doc, ...
>
> But SPSProxyHelper#getFieldValues() method works for only docId=/Documents/Vekaletname.pdf returns empty map for others. Therefore only this one injected.
>
> --- On Thu, 8/2/12, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>
>> From: Ahmet Arslan <[EMAIL PROTECTED]>
>> Subject: RE: SharePoint Library consist of folders
>> To: [EMAIL PROTECTED]
>> Date: Thursday, August 2, 2012, 10:38 PM
>> > This replaces the getlistitems
>> call
>> > in spsproxyhelper with a custom
>> > method call.
>>
>> Once 492 in place is it going to list files that have folder
>> path too? Without checking value of ows_FSObjType
>> attirbute?
>>
>>
>>

> I checked this change into trunk, and
> added also corresponding code in
> the place where fields and metadata are fetched.  This
> may work for
> you in the interim while we're finishing up CONNECTORS-492.

Awesome, I confirm that folder docs are indexed along with their metadata nicely. (From a SharePoint 2010 instance)

Thanks!