On 3/17/2017 11:14 AM, Imad Qureshi wrote:
What's going to matter is how fast the data can be retrieved. With
standard local filesystems, the operating system will use unallocated
memory to cache the data, so if you have enough available memory for
that caching to be effective, access is lightning fast -- the most
requested index data will be in memory, and pulled directly from there
into the application. If the disk has to be read to obtain the needed
data, it will be slow. If data has to be transferred over a network
that's gigabit or slower, that is also slow. Faster network
technologies are available for a price premium, but if a disk has to be
read to get the data, the network speed won't matter. Good performance
means avoiding going to the disk or transferring over the network.
SSD storage is faster than regular disks, but still not as fast as main
memory, and increased storage speed probably won't matter if the network
can't keep up.
If I'm not mistaken, I think an HDFS client can allocate system memory
for caching purposes to avoid the slow transfer for frequently requested
data. If my understanding is correct, then enough memory allocated to
the HDFS client MIGHT avoid network/disk transfer for the important data
in the index ... but whether this works in practice is a question I
Unless your 16TB of index data is being utilized by MANY Solr servers
that each use a very small part of the data and have the ability to
cache a significant percentage of the data they're using, it's highly
unlikely that you're going to have enough memory for good caching.
Indexes that large are typically slow unless you can afford a LOT of
hardware, which means a lot of money.