user3231055 user3231055 - 9 days ago 6
Java Question

Jetty server unexpectedly trades cpu to memory and vise versa

I have a rest service based on Spark Java 2.5, which uses Jetty server under the hood.
My problem is that it doesn't work on a constant performance and suddenly "decides to trade" cpu to memory and vise versa some time later. Here are the plots

Plots are created using Java melody.
As you can see - at about 18:00 performance plots abruptly changed. Memory consumption began to grow and processor load goes down. At the same time, request latencies didn't change as well as request per|second and request types. Additional parameters also changed - especially used buffered memory and number of opened files.

A week later everything will change back and maybe two weeks or a month later the cycle repeats, I saw this cycle for the last three months.

I tried to use profiler, but didn't find anything useful.

I'm pretty sure such change is not provoked by business logic, because nothing is changed in user's interaction with web server and no background tasks are active, so probably it's jetty's or java's internals or misconfiguration.

Server runs on a Java 8 in a Docker container in AWS EC2 (we use AWS ECS for docker autoscaling). There is about 50 requests per second load. The api itself uses spring and hibernate with postgresql 9.4 driver. For hibernate second level ehcache is used. Some of api requests are multipart with size about 100kb and they are uniformly distributed on a request timeline. Java is started with parameters: -server -Xmx6000m -XX:+UseG1GC. If more details needed, please ask me.

What I want is a constant performance. If my problem resonates with your experience, please reply.

ps: on ~23:30 change is not related to the problem, so don't analyze it.

Answer

Finally the solution was simple. Increasing -Xms to 500mb helped, after it behavior became stationary.

Comments