I changed about ten days ago and the jobs were running correctly. I could do 2 passages without problems since the introduction of the trunk version. I have a doubt that they are old documents. I restart indexing and if it happens again, I'll tell you.

 

Maxence,

 

De : Karl Wright [mailto:[EMAIL PROTECTED]]
Envoyé : lundi 18 juin 2018 14:25
À : [EMAIL PROTECTED]
Objet : Re: Documents blocked sometimes without errors

 

My concern is that you upgraded the code but DID NOT do the pause/resume after you did that.  If that was was the sequence, you were left with old, un-updated records.

 

 

 

On Mon, Jun 18, 2018 at 8:18 AM msaunier <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

Yes my solution is paused the job and resume it.

 

With the trunk version, I feel it's less common but the problem is still here.

 

 

De : Karl Wright [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ]
Envoyé : lundi 18 juin 2018 12:14
À : [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
Objet : Re: Documents blocked sometimes without errors

 

Just so it is clear, the fix only will address documents that are in the "ACTIVE" state.  Documents that are already blocked will not be fixed.  The way you fix the blocked documents is by pausing and resuming the job that the documents are part of -- and then, if you are running the patched version of MCF, you should not see blocked documents again.

 

Thanks,

Karl

 

 

On Mon, Jun 18, 2018 at 5:15 AM msaunier <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:
Forget that. My ln-s is good on this server. I confused the servers. So I have a similar problem with trunk. I continu the tests.

 

 

 

De : msaunier [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ]
Envoyé : lundi 18 juin 2018 11:13
À : '[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ' <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> >
Objet : RE: Documents blocked sometimes without errors

 

Ok I have miss my ln –s so my link go to 2.9.1. Sorry for this error. Your corrections are okay.

 

 

De : Karl Wright [mailto:[EMAIL PROTECTED]]
Envoyé : lundi 18 juin 2018 10:43
À : [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
Objet : Re: Documents blocked sometimes without errors

 

If there's any chance these were leftover from before the patch was applied, we should try to eliminate that.  To do that:

 

- pause the job

- restart the job
Then, either wait for the script-based agents process shutdown, or shut down the agents process manually and restart.  Do this a number of times and see if any documents become stuck.

 

Thanks,

Karl

 

 

On Mon, Jun 18, 2018 at 4:35 AM Karl Wright <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

These are still indeed blocked.

 

Unfortunately I don't see any pathway for documents to wind up in such a state.  I'll have to look in more depth and get back to you later.

 

Karl

 

On Mon, Jun 18, 2018 at 4:07 AM msaunier <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

CSV joined.

 

Thanks,

Maxence,

 

 

 

De : Karl Wright [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ]
Envoyé : lundi 18 juin 2018 10:02
À : [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
Objet : Re: Documents blocked sometimes without errors

 

The only way to know if these are truly blocked is to find the document records in the database and include them here.

 

Thanks,

Karl

 

 

On Mon, Jun 18, 2018 at 3:55 AM msaunier <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

Hello Karl,

 

Today, I have 2 documents blocked on the new trunk version (I think). Can I verify my trunk vertion after the build?

 

Thanks,

Maxence ,

 

 

De : msaunier [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ]
Envoyé : mardi 5 juin 2018 14:54
À : '[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> ' <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> >
Objet : RE: Documents blocked sometimes without errors

 

Ok. I have build and deploy.

 

The tests are in progress.

 

Thanks,

Maxence

 

 

De : Karl Wright [mailto:[EMAIL PROTECTED]]
Envoyé : lundi 4 juin 2018 19:55
À : [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
Objet : Re: Documents blocked sometimes without errors

 

I attached a patch to the ticket that is a tentative fix.  Please let me know if you still see this problem after applying it.  Thanks!

 

Karl

 

 

On Mon, Jun 4, 2018 at 12:56 PM Karl Wright <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

CONNECTORS-1507 created.

Karl

 

 

On Mon, Jun 4, 2018 at 12:51 PM Karl Wright <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

I think I found the issue.

Basically, when the agents process is restarted, it doesn't reprioritize the documents that were active when the service was brought down, but it should because when the documents became active they lost their document priority.  This should be trivial to fix.

 

Karl

 

 

On Mon, Jun 4, 2018 at 12:35 PM Karl Wright <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote:

Hi Maxence,

 

The docpriority values for these stuck documents show that they are "null":

 

  public static final double noDocPriorityValue = 1e9;

  public static final Double nullDocPriority = new Double(noDocPriorityValue + 1.0);

 

The document status is "G", which is STATUS_PENDINGPURGATORY, so the documents are awaiting being queued, which they will never be with a docpriority set to nullDocPriority.

 

It isn't supposed to be possible for a document to wind up in this state.  Documents that are pending are always supposed to set a document priority.  I will need to review the code to see how this could happen.

It is also possible that you're seeing a database bug.  I presume that you are running Postgresql?

 

Karl

 

 

On Mon, Jun 4, 2018 at 8:43 AM msaunier <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB