Droids seems like a very interesting project upon which I'd like to  
base some of my custom web crawlers - but I'm running into this error  
when I try to run the examples.   The file droids-core/src/test/java/
org/apache/droids/examples/SimpleRuntime.java, for example, throws  
this exception.   I've spent awhile trying to figure out what the  
problem is, to no avail.

I've verified with TCPDump that running the example never initiates  
any HTTP activity.   I'm running droids-trunk, svn revision 754973, on  
Mac OSX 10.5.6.

Can anyone point me in the right direction?

2009-03-16 14:24:04,107 INFO [main] - <Number of concurrent workers: 5>
2009-03-16 14:24:04,113 INFO [main] - <Finshed invocation, waiting for  
workers to finish.>
2009-03-16 14:24:04,113 INFO [pool-1-thread-2] - <SHUTTING DOWN>
2009-03-16 14:24:04,114 INFO [pool-1-thread-2] - <All threads have  
finished. (elapsed: 8)>
2009-03-16 14:24:04,114 INFO [pool-1-thread-2] - <FINISHED!!!>
2009-03-16 14:24:04,760 WARN [pool-1-thread-1] - <Connection pool shut  
down.>
java.lang.IllegalStateException: Connection pool shut down.
at  
org
.apache
.http
.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:
284)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
$1.getPoolEntry(ConnPoolByRoute.java:238)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
$1.getConnection(ThreadSafeClientConnManager.java:175)
at  
org
.apache
.http
.impl
.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
at  
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
555)
at  
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
487)
at  
org
.apache
.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:
465)
at  
org
.apache
.droids
.protocol
.http.HttpClientContentLoader.exists(HttpClientContentLoader.java:58)
at org.apache.droids.norobots.NoRobotClient.parse(NoRobotClient.java:
98)
at  
org
.apache.droids.protocol.http.HttpProtocol.isAllowed(HttpProtocol.java:
102)
at  
org
.apache
.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:53)
at  
org
.apache
.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:36)
at org.apache.droids.impl.MultiThreadedTaskMaster
$WorkerRunner.run(MultiThreadedTaskMaster.java:280)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
2009-03-16 14:24:04,764 INFO [pool-1-thread-1] - <SHUTTING DOWN>
2009-03-16 14:24:04,764 INFO [pool-1-thread-1] - <All threads have  
finished. (elapsed: 658)>
2009-03-16 14:24:04,764 INFO [pool-1-thread-1] - <FINISHED!!!>

  

On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
> Droids seems like a very interesting project upon which I'd like to  
> base some of my custom web crawlers - but I'm running into this error  
> when I try to run the examples.   The file droids-core/src/test/java/
> org/apache/droids/examples/SimpleRuntime.java, for example, throws  
> this exception.   I've spent awhile trying to figure out what the  
> problem is, to no avail.
>
> I've verified with TCPDump that running the example never initiates  
> any HTTP activity.   I'm running droids-trunk, svn revision 754973, on  
> Mac OSX 10.5.6.

I just tried and I have a similar problem. Not sure ATM why but I can
reproduce the problem.

I do not get an exception like you but rather:  
<Finshed invocation, waiting for workers to finish.>
2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
(org.apache.http.NoHttpResponseException) caught when processing
request: The target server failed to respond>

salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>

On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
> > Droids seems like a very interesting project upon which I'd like to  
> > base some of my custom web crawlers - but I'm running into this error  
> > when I try to run the examples.   The file droids-core/src/test/java/
> > org/apache/droids/examples/SimpleRuntime.java, for example, throws  
> > this exception.   I've spent awhile trying to figure out what the  
> > problem is, to no avail.
> >
> > I've verified with TCPDump that running the example never initiates  
> > any HTTP activity.   I'm running droids-trunk, svn revision 754973, on  
> > Mac OSX 10.5.6.
>
> I just tried and I have a similar problem. Not sure ATM why but I can
> reproduce the problem.
>
> I do not get an exception like you but rather:  
> <Finshed invocation, waiting for workers to finish.>
> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
> (org.apache.http.NoHttpResponseException) caught when processing
> request: The target server failed to respond>

If I do
Index:
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
==================================================================---
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
(revision 755424)
+++
droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
(working copy)
@@ -98,6 +98,7 @@
     DroidsHttpClient httpclient = new DroidsHttpClient(params);
    
     HttpProtocol httpProtocol = new HttpProtocol(httpclient);
+    httpProtocol.setForceAllow(true);
     protocolFactory.setMap(new HashMap<String, Object>());

I as well get. Seems that the http pool is not working.

Oleg do you have an idea?

java.lang.IllegalStateException: Connection pool shut down.
at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
$1.getPoolEntry(ConnPoolByRoute.java:238)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
$1.getConnection(ThreadSafeClientConnManager.java:175)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
at
org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
at
org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
at
org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
at org.apache.droids.impl.MultiThreadedTaskMaster
$WorkerRunner.run(MultiThreadedTaskMaster.java:280)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

> salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>

Thorsten Scherler wrote:
> On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
>> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
>>> Droids seems like a very interesting project upon which I'd like to  
>>> base some of my custom web crawlers - but I'm running into this error  
>>> when I try to run the examples.   The file droids-core/src/test/java/
>>> org/apache/droids/examples/SimpleRuntime.java, for example, throws  
>>> this exception.   I've spent awhile trying to figure out what the  
>>> problem is, to no avail.
>>>
>>> I've verified with TCPDump that running the example never initiates  
>>> any HTTP activity.   I'm running droids-trunk, svn revision 754973, on  
>>> Mac OSX 10.5.6.
>> I just tried and I have a similar problem. Not sure ATM why but I can
>> reproduce the problem.
>>
>> I do not get an exception like you but rather:  
>> <Finshed invocation, waiting for workers to finish.>
>> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
>> (org.apache.http.NoHttpResponseException) caught when processing
>> request: The target server failed to respond>
>
> If I do
> Index:
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> ==================================================================> ---
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> (revision 755424)
> +++
> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
> (working copy)
> @@ -98,6 +98,7 @@
>      DroidsHttpClient httpclient = new DroidsHttpClient(params);
>      
>      HttpProtocol httpProtocol = new HttpProtocol(httpclient);
> +    httpProtocol.setForceAllow(true);
>      protocolFactory.setMap(new HashMap<String, Object>());
>
> I as well get. Seems that the http pool is not working.
>
> Oleg do you have an idea?
>

Thorsten,

I'll look into it.

Oleg
> java.lang.IllegalStateException: Connection pool shut down.
> at
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
> $1.getPoolEntry(ConnPoolByRoute.java:238)
> at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> $1.getConnection(ThreadSafeClientConnManager.java:175)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
> at
> org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
> at
> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
> at
> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
> at org.apache.droids.impl.MultiThreadedTaskMaster
> $WorkerRunner.run(MultiThreadedTaskMaster.java:280)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
>> salu2

Oleg Kalnichevski wrote:
> Thorsten Scherler wrote:
>> On Tue, 2009-03-17 at 23:31 +0100, Thorsten Scherler wrote:
>>> On Mon, 2009-03-16 at 14:25 -0400, Andy Gross wrote:
>>>> Droids seems like a very interesting project upon which I'd like to  
>>>> base some of my custom web crawlers - but I'm running into this
>>>> error  when I try to run the examples.   The file
>>>> droids-core/src/test/java/
>>>> org/apache/droids/examples/SimpleRuntime.java, for example, throws  
>>>> this exception.   I've spent awhile trying to figure out what the  
>>>> problem is, to no avail.
>>>>
>>>> I've verified with TCPDump that running the example never initiates  
>>>> any HTTP activity.   I'm running droids-trunk, svn revision 754973,
>>>> on  Mac OSX 10.5.6.
>>> I just tried and I have a similar problem. Not sure ATM why but I can
>>> reproduce the problem.
>>>
>>> I do not get an exception like you but rather:  <Finshed invocation,
>>> waiting for workers to finish.>
>>> 2009-03-17 23:25:17,670 INFO [pool-1-thread-1] - <I/O exception
>>> (org.apache.http.NoHttpResponseException) caught when processing
>>> request: The target server failed to respond>
>>
>> If I do Index:
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> ==================================================================>> ---
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> (revision 755424)
>> +++
>> droids-core/src/test/java/org/apache/droids/examples/SimpleRuntime.java
>> (working copy)
>> @@ -98,6 +98,7 @@
>>      DroidsHttpClient httpclient = new DroidsHttpClient(params);
>>           HttpProtocol httpProtocol = new HttpProtocol(httpclient);
>> +    httpProtocol.setForceAllow(true);
>>      protocolFactory.setMap(new HashMap<String, Object>());
>>
>> I as well get. Seems that the http pool is not working.
>> Oleg do you have an idea?
>>
>
> Thorsten,
>
> I'll look into it.
>
> Oleg
>

The problem is not caused by the HTTP transport. The
MultiThreadedTaskMaster appears to have synchronization issues and
should also be reviewed for thread-safety.

Oleg
>
>> java.lang.IllegalStateException: Connection pool shut down.
>>     at
>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:284)
>>
>>     at org.apache.http.impl.conn.tsccm.ConnPoolByRoute
>> $1.getPoolEntry(ConnPoolByRoute.java:238)
>>     at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
>> $1.getConnection(ThreadSafeClientConnManager.java:175)
>>     at
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
>>
>>     at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
>>
>>     at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
>>
>>     at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
>>
>>     at
>> org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:63)
>>     at
>> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:57)
>>
>>     at
>> org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
>>
>>     at org.apache.droids.impl.MultiThreadedTaskMaster
>> $WorkerRunner.run(MultiThreadedTaskMaster.java:280)
>>     at java.util.concurrent.ThreadPoolExecutor
>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>     at java.util.concurrent.ThreadPoolExecutor
>> $Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>>> salu2
>

On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
> Oleg Kalnichevski wrote:
...
> >
>
> The problem is not caused by the HTTP transport. The
> MultiThreadedTaskMaster appears to have synchronization issues and
> should also be reviewed for thread-safety.

Ok, do you have any method in special as possible candidates? Will now
look into it and hopefully find the problem.

salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>

Thorsten Scherler wrote:
> On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
>> Oleg Kalnichevski wrote:
> ...
>> The problem is not caused by the HTTP transport. The
>> MultiThreadedTaskMaster appears to have synchronization issues and
>> should also be reviewed for thread-safety.
>
> Ok, do you have any method in special as possible candidates? Will now
> look into it and hopefully find the problem.
>
> salu2

I have looked at MultiThreadedTaskMaster quite extensively and got an
impression that it simply did not attempt to synchronize access to its
instance variables at all despite the fact they could be accessed from
multiple threads of execution. I'll try to find time to propose a fix
for the issue, but cannot really tell when I can manage to get around to
doing that.

Oleg

On Mon, 2009-03-23 at 20:52 +0100, Oleg Kalnichevski wrote:
> Thorsten Scherler wrote:
> > On Sun, 2009-03-22 at 00:07 +0100, Oleg Kalnichevski wrote:
> >> Oleg Kalnichevski wrote:
> > ...
> >> The problem is not caused by the HTTP transport. The
> >> MultiThreadedTaskMaster appears to have synchronization issues and
> >> should also be reviewed for thread-safety.
> >
> > Ok, do you have any method in special as possible candidates? Will now
> > look into it and hopefully find the problem.
> >
> > salu2
>
> I have looked at MultiThreadedTaskMaster quite extensively and got an
> impression that it simply did not attempt to synchronize access to its
> instance variables at all despite the fact they could be accessed from
> multiple threads of execution. I'll try to find time to propose a fix
> for the issue, but cannot really tell when I can manage to get around to
> doing that.
>

Thank you very much Oleg, looking forward.

I ATM preparing some small example for ApacheCon and your recent change
shows the sequential capacity so that is good and brings back a running
example.

salu2
--
Thorsten Scherler <thorsten.at.apache.org>
Open Source <consulting, training and solutions>
--
Thorsten Scherler <scherler.at.gmail.com>
Open Source <consulting, training and solutions>