I tried some new things. I changed the number of shards to 27, since I saw the size of some of my indices to be over 400GB in size. After I this change, I saw the heap usage increase to around 80GB. I will have to collect some data in the upcoming days to find out whether this actually helps. I removed one replica, and only do 1 now.
I also tried _forcemerge'ing a few indices of days that have passed. The data in these indices will not change anymore, so the number of segments can be minimized to increase searching performance.
I performed a simple '*' discovery query over the last 7 days today, and the query takes so long to complete, that it times out instead. The query returns at most 2 billion records. Why does Kibana take so long to complete a simple query? Also, why does Kibana query all indices that are present in Elasticsearch?
Another side note: My data nodes run on three different physical machines. These machines are 10GE connected, but are not located in the same datacenter. Does the slightly increased latency matter much?