wild guess: The ergonomic choice of the JVM for the maximum allowed direct memory is too high.
Unfortunately, it is a bit hard to find out which value the JVM has chosen (you can write a small Java program which reads `sun.misc.VM.maxDirectMemory()` and invoke it in the Docker container).
You can restrict direct memory explicitly e.g. to 2G with `-XX:MaxDirectMemory=2G`. Be aware that you will see more frequent garbage collections if the value you choose is too small. Also, ensure that `-XX:+DisableExplicitGC` is **not** set in your `jvm.options` (it was set by default before Elasticsearch 5.5.2).