Thanks for the question. We have warmed up the cluster - running a slew of queries before running our test. We've also run tests where we have run the exact same queries over and over to watch the change in performance. Running the exact same query over and over several times does result in it going a bit faster... but instead of being 10X slower, it might be 8 or 9X slower than 1.7.4. Also - running a variety queries to warm up the cluster has minimal impact on our test queries unless the warm up queries were exactly the same.
We've also ran tests where we explicitly clear caches on the 1.7.4 cluster, run test queries there. Even with a cleared cache on 1.7.4, its much faster than a warmed up 5.4 cluster.