My MySQL Server went down today!! Troubleshooting.


Panic!!!

My MySQL Server went down today, which meant that my website was not working as well. When I checked, it clearly replied:

[root@sdb ~]# service mysqld status
mysqld is stopped

I automatically checked for the uptime of the server to see that it was a few hours ago. A chkconfig revealed that I forgot to add mysqld to autostart in run levels 2-5. Serious mistake, but since I reinstalled the server a few days ago, comprehensible.. i guess!

mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off

I then switched it to autorun and started the service via the following commands:

 

chkconfig mysqld on

 

service mysqld start

Solved! But not proper..

..The big question why did the server reboot?

I checked the logs at /var/log/messages to see the reason of the reboot.

less /var/log/messages

which revealed

Jun  1 03:08:06 sdb init: tty (/dev/tty1) main process (1972) killed by TERM signal
Jun  1 03:08:06 sdb init: serial (hvc0) main process (1974) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty2) main process (1975) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty3) main process (1977) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty4) main process (1980) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty5) main process (1982) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty6) main process (1984) killed by TERM signal
Jun  1 03:08:12 sdb abrtd: Got signal 15, exiting
Jun  1 03:09:12 sdb init: Disconnected from system bus
Jun  1 03:09:12 sdb auditd[1824]: The audit daemon is exiting.
Jun  1 03:09:14 sdb kernel: Kernel logging (proc) stopped.
Jun  1 03:09:14 sdb rsyslogd: [origin software=”rsyslogd” swVersion=”4.6.2″ x-pid=”1839″ x-info=”http://www.rsyslog.com”] exiting on signal 15.
Jun  1 03:18:53 sdb kernel: imklog 4.6.2, log source = /proc/kmsg started.

What does this mean?

The system was shut down by an external person, Linode. I forgot they said that they would be having migration. Here’s the extract of the mail:

 * sdb – (7864940) System Shutdown – Completed Fri, 01 Jun 2012 03:08:07 GMT
* sdb – (7864941) Linode Initial Configuration – Completed Fri, 01 Jun 2012 03:09:28 GMT
* sdb – (7864942) Migrate Filesystem – swap from london62.linode.com – Completed Fri, 01 Jun 2012 03:09:29 GMT
* sdb – (7864943) Migrate Filesystem – Data Disk from london62.linode.com – Completed Fri, 01 Jun 2012 03:10:00 GMT
* sdb – (7864944) Migrate Filesystem – Backup Disk from london62.linode.com – Completed Fri, 01 Jun 2012 03:12:36 GMT
* sdb – (7864945) Migrate Filesystem – CentOS 6.2 64bit Disk Image from london62.linode.com – Completed Fri, 01 Jun 2012 03:15:02 GMT
* sdb – (7864946) Linode Migration Cleanup – Completed Fri, 01 Jun 2012 03:18:29 GMT
* sdb – (7864947) System Boot – SDB Config – Completed Fri, 01 Jun 2012 03:18:36 GMT

Happy as ever, I wrote a script to auto start a service if they have been found as not running, even if the server did not reboot. View it here.

Gulshan Yashveer Beejan

Gulshan holds a BSc(Hons) Computer Science with Distributed systems from the City University of London. He is currently working at DSOGroup(Mauritius) as a Linux Administrator on SaaS Solutions. More about Gulshan

Leave a Reply

Your email address will not be published. Required fields are marked *