I'm having a very weird issue, I have an Apache server running with mod_wsgi. The website runs fine but every once in a while I get the
IOError: failed to write data
sudo service mysqld restart
(2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")
160610 10:51:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160610 10:51:53 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 7658 ...
160617 14:35:47 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 32054 ...
160617 14:35:47 [Note] Plugin 'FEDERATED' is disabled.
160617 14:35:47 InnoDB: The InnoDB memory heap is disabled
160617 14:35:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160617 14:35:47 InnoDB: Compressed tables use zlib 1.2.8
160617 14:35:47 InnoDB: Using Linux native AIO
160617 14:35:47 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160617 14:35:47 InnoDB: Completed initialization of buffer pool
160617 14:35:47 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160617 14:35:47 [ERROR] Plugin 'InnoDB' init function returned error.
160617 14:35:47 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160617 14:35:47 [ERROR] Unknown/unsupported storage engine: InnoDB
160617 14:35:47 [ERROR] Aborting
This seems to be the culprit
160617 14:35:47 InnoDB: Initializing buffer pool, size = 128.0M InnoDB: mmap(137363456 bytes) failed; errno 12
This condition is caused by the server running out of physical memory. That's what errno 12 stands for in the kernel errors.
This topic has been discussed at both stackoverflow and dba.stackexchange. If you want to simulate low memory situations or try to manually reproduce the error try some of these tools: http://unix.stackexchange.com/questions/99334/how-to-fill-90-of-the-free-memory
If you are able to upgrade memory you can try that. If not, you can try creating a large swap file. It's possible that you don't have a swap at all. Some AWS EC2 instanes don't have one by default. You can find out by typing
top in the shell. If you don't see swap near the top, that means you don't have one.
A swap file would make the queries a lot slower but at least it's better than the site getting offline.
You might be tempted to try to modify the
systemd files to make mysql auto start. Update: @PeterBrittain points out that mysql is auto restarting anyway as shown by the logs. Sometimes though databases can take a bit of time to restart and if the data get's corrupted it will refuse to restart.
If you don't have any other server running on it, 2GB will be more than enough to host a site that serves 6000 pages per day. It could be that you have some rather heavy queries that put an unnecessary load on the db. There are some remedial actions that can be taken