I noticed that a mysql server is with CPU at 100%, and the "kernel time" (I'm not sure what it means) is unusually high, about 70%.
There are many connections on this server (around 400) and some active queries (about 40). Would that explain this behavior? Is there something wrong or this is expected?
As suggested by a comment, I checked the 'handler_read%' variables:
show global status like 'handler_read%'
The high values for handler_read_rnd* indicate that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have.
Due to syscall overhead and context switches table scans use more CPU.
Before changing parameters or invest money in hardware, I would suggest to optimize your database: