I tried some new things. I changed the number of shards to 27, since I saw the size of some of my indices to be over 400GB in size. After I this change, I saw the heap usage increase to around 80GB. I will have to collect some data in the upcoming days to find out whether this actually helps. I removed one replica, and only do 1 now.

I also tried _forcemerge'ing a few indices of days that have passed. The data in these indices will not change anymore, so the number of segments can be minimized to increase searching performance.

I performed a simple '*' discovery query over the last 7 days today, and the query takes so long to complete, that it times out instead. The query returns at most 2 billion records. Why does Kibana take so long to complete a simple query? Also, why does Kibana query all indices that are present in Elasticsearch?

Another side note: My data nodes run on three different physical machines. These machines are 10GE connected, but are not located in the same datacenter. Does the slightly increased latency matter much?

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB