Have you tried using the gradle plugin provided by Elastic to build ES plugins? This automatically sets up a test task which uses the ES test framework.

Aside from that, I see a couple things likely to cause problems. One is using compileOnly here. While that is correct, you need to make your testCompile extend from it (gradle does not do this). Something like this should work:

configurations.testCompile.extendsFrom configurations.compileOnly

Your issue is likely how the test is run. We don't use the gradle test runner, but instead the same test runner that Lucene uses: randomized testing framework. In order to use it, you need to have your tests either extend RandomizedTest, or annotate your tests to use the RandomizedRunner (see https://github.com/randomizedtesting/randomizedtesting/wiki/Randomized-Runner).

But even better would be to use the gradle plugin. Unfortunately we don't have any guide for this yet, but a simple build.gradle using it should look something like this:

buildscript {
    repositories {
    dependencies {
        classpath "org.elasticsearch.gradle:build-tools:$elasticsearchVersion"

apply plugin: 'elasticsearch.esplugin'

esplugin {
    name 'my-plugin-name'
    description 'a description of my plugin'
    classname 'pkg.MyClassExtendingPlugin'

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB