运行中的 mysql 突然关闭了

[guangzhouwuyanzu]

2018-11-15 10:30:35 3452 [Note] Plugin 'FEDERATED' is disabled.
2018-11-15 10:30:35 3452 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-11-15 10:30:35 3452 [Note] InnoDB: The InnoDB memory heap is disabled
2018-11-15 10:30:35 3452 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-11-15 10:30:35 3452 [Note] InnoDB: Memory barrier is not used
2018-11-15 10:30:35 3452 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-11-15 10:30:35 3452 [Note] InnoDB: Using CPU crc32 instructions
2018-11-15 10:30:35 3452 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2018-11-15 10:30:35 3452 [Note] InnoDB: Completed initialization of buffer pool
2018-11-15 10:30:35 3452 [Note] InnoDB: Highest supported file format is Barracuda.
2018-11-15 10:30:36 3452 [Note] InnoDB: The log sequence numbers 329132395 and 329132395 in ibdata files do not match the log sequence number 341178818 in the ib_logfiles!
2018-11-15 10:30:36 3452 [Note] InnoDB: Database was not shutdown normally!
2018-11-15 10:30:36 3452 [Note] InnoDB: Starting crash recovery.
2018-11-15 10:30:36 3452 [Note] InnoDB: Reading tablespace information from the .ibd files...
2018-11-15 10:30:39 3452 [Note] InnoDB: Restoring possible half-written data pages 
2018-11-15 10:30:39 3452 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 8014580, file name mysql-bin.000044
2018-11-15 10:30:41 3452 [Note] InnoDB: 128 rollback segment(s) are active.
2018-11-15 10:30:41 3452 [Note] InnoDB: Waiting for purge to start
2018-11-15 10:30:41 3452 [Note] InnoDB: 5.6.36 started; log sequence number 341178818
2018-11-15 10:30:41 3452 [Note] Recovering after a crash using mysql-bin
2018-11-15 10:30:41 3452 [Note] Starting crash recovery...
2018-11-15 10:30:41 3452 [Note] Crash recovery finished.
2018-11-15 10:30:42 3452 [Note] Server hostname (bind-address): '*'; port: 4580
2018-11-15 10:30:42 3452 [Note] IPv6 is available.
2018-11-15 10:30:42 3452 [Note]   - '::' resolves to '::';
2018-11-15 10:30:42 3452 [Note] Server socket created on IP: '::'.
2018-11-15 10:30:42 3452 [Note] Event Scheduler: Loaded 0 events
2018-11-15 10:30:42 3452 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.6.36-log'  socket: '/tmp/mysql.sock'  port: 4580  Source distribution
2018-11-15 10:40:16 5810 [Note] Plugin 'FEDERATED' is disabled.
2018-11-15 10:40:16 5810 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-11-15 10:40:16 5810 [Note] InnoDB: The InnoDB memory heap is disabled
2018-11-15 10:40:16 5810 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-11-15 10:40:16 5810 [Note] InnoDB: Memory barrier is not used
2018-11-15 10:40:16 5810 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-11-15 10:40:16 5810 [Note] InnoDB: Using CPU crc32 instructions
2018-11-15 10:40:16 5810 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2018-11-15 10:40:16 5810 [Note] InnoDB: Completed initialization of buffer pool
2018-11-15 10:40:16 5810 [Note] InnoDB: Highest supported file format is Barracuda.
2018-11-15 10:40:16 5810 [Note] InnoDB: The log sequence numbers 329132395 and 329132395 in ibdata files do not match the log sequence number 341499308 in the ib_logfiles!
2018-11-15 10:40:16 5810 [Note] InnoDB: Database was not shutdown normally!
2018-11-15 10:40:16 5810 [Note] InnoDB: Starting crash recovery.
2018-11-15 10:40:16 5810 [Note] InnoDB: Reading tablespace information from the .ibd files...
2018-11-15 10:40:18 5810 [Note] InnoDB: Restoring possible half-written data pages 
2018-11-15 10:40:18 5810 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 272993, file name mysql-bin.000045
2018-11-15 10:40:19 5810 [Note] InnoDB: 128 rollback segment(s) are active.
2018-11-15 10:40:19 5810 [Note] InnoDB: Waiting for purge to start
2018-11-15 10:40:19 5810 [Note] InnoDB: 5.6.36 started; log sequence number 341499308
2018-11-15 10:40:19 5810 [Note] Recovering after a crash using mysql-bin
2018-11-15 10:40:19 5810 [Note] Starting crash recovery...
2018-11-15 10:40:19 5810 [Note] Crash recovery finished.
2018-11-15 10:40:20 5810 [Note] Server hostname (bind-address): '*'; port: 4580
2018-11-15 10:40:20 5810 [Note] IPv6 is available.
2018-11-15 10:40:20 5810 [Note]   - '::' resolves to '::';
2018-11-15 10:40:20 5810 [Note] Server socket created on IP: '::'.
2018-11-15 10:40:20 5810 [Note] Event Scheduler: Loaded 0 events
2018-11-15 10:40:20 5810 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.6.36-log'  socket: '/tmp/mysql.sock'  port: 4580  Source distribution
2018-11-15 10:50:15 7647 [Note] Plugin 'FEDERATED' is disabled.
2018-11-15 10:50:15 7647 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-11-15 10:50:15 7647 [Note] InnoDB: The InnoDB memory heap is disabled
2018-11-15 10:50:15 7647 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-11-15 10:50:15 7647 [Note] InnoDB: Memory barrier is not used
2018-11-15 10:50:15 7647 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-11-15 10:50:15 7647 [Note] InnoDB: Using CPU crc32 instructions
2018-11-15 10:50:15 7647 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2018-11-15 10:50:15 7647 [Note] InnoDB: Completed initialization of buffer pool
2018-11-15 10:50:15 7647 [Note] InnoDB: Highest supported file format is Barracuda.
2018-11-15 10:50:15 7647 [Note] InnoDB: The log sequence numbers 329132395 and 329132395 in ibdata files do not match the log sequence number 341872567 in the ib_logfiles!
2018-11-15 10:50:15 7647 [Note] InnoDB: Database was not shutdown normally!
2018-11-15 10:50:15 7647 [Note] InnoDB: Starting crash recovery.
2018-11-15 10:50:15 7647 [Note] InnoDB: Reading tablespace information from the .ibd files...
2018-11-15 10:50:18 7647 [Note] InnoDB: Restoring possible half-written data pages 
2018-11-15 10:50:18 7647 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 349361, file name mysql-bin.000046

错误日志如上 zabbix 监控到内存 Lack of available memory

然后数据库就自动关闭了，是因为内存的不足，导致 do not match the log sequence 吗？

messages 里面还真有OOM，，很大一段，，内存不足 数据库被kill了，，谢谢2位大佬

Nov 15 10:50:14 VM_125_204_centos kernel: [ 7601]     0  7601    14708       62      26        0             0 php
Nov 15 10:50:14 VM_125_204_centos kernel: [ 7602]     0  7602    28280       52      12        0             0 bash
Nov 15 10:50:14 VM_125_204_centos kernel: [ 7603]     0  7603    30829      685      16        0             0 python
Nov 15 10:50:14 VM_125_204_centos kernel: [ 7604]     0  7604    14708       62      24        0             0 php
Nov 15 10:50:14 VM_125_204_centos kernel: [ 7605]     0  7605    14708       63      26        0             0 php
Nov 15 10:50:14 VM_125_204_centos kernel: [ 7620]     0  7620    14708       62      25        0             0 php
Nov 15 10:50:14 VM_125_204_centos kernel: Out of memory: Kill process 5810 (mysqld) score 164 or sacrifice child
Nov 15 10:50:14 VM_125_204_centos kernel: Killed process 5810 (mysqld) total-vm:4259884kB, anon-rss:1316636kB, file-rss:0kB, shmem-rss:0kB

[msg7086]

是因为内存不足，导致数据库被内核结束进程。

[Aliencn]

看 message 日志里面有没有 OOM 的记录

[guangzhouwuyanzu]

@msg7086
@Aliencn

看了 messages，还真是因为内存不足，，导致数据库进程被 kill 了。

[julyclyde]

@msg7086 赞经验丰富！