I agree that 46 million docs later you'd expect things to have settled
out. However, I do note that you have
"add-unknown-fields-to-the-schema" in your error stack which means
you're using "field guessing", sometimes called data_driven. I would
recommend you do _not_ use this for production as, while it does the
best job it can it has to make assumptions about what the data looks
like based on the first document it sees which may later be violated.
Getting "possible analysis error" is one of the messages that happens
when this occurs.

The simple example is that if the first time data_driven sees "1"
it'll guess integer. If sometime later there's a doc with "1.0" it'll
generate a parse error.

I totally agree that 46 million docs later you'd expect all of this
kind of thing to have flushed out, but the "possible analysis error"
seems to be pointing that direction. If this is, indeed, the problem
you'll see better evidence on the Solr instance that's actually having
the problem. Unfortunately you'll just to look at one Solr log from
each shard to see whether this is an issue.


On Mon, Jul 17, 2017 at 7:23 AM, Joe Obernberger
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB