My MySQL Server went down today!! Troubleshooting.


Panic!!!

My MySQL Server went down today, which meant that my website was not working as well. When I checked, it clearly replied:

[root@sdb ~]# service mysqld status
mysqld is stopped

I automatically checked for the uptime of the server to see that it was a few hours ago. A chkconfig revealed that I forgot to add mysqld to autostart in run levels 2-5. Serious mistake, but since I reinstalled the server a few days ago, comprehensible.. i guess!

mysqld          0:off   1:off   2:off   3:off   4:off   5:off   6:off

I then switched it to autorun and started the service via the following commands:

 

chkconfig mysqld on

 

service mysqld start

Solved! But not proper..

..The big question why did the server reboot?

I checked the logs at /var/log/messages to see the reason of the reboot.

less /var/log/messages

which revealed

Jun  1 03:08:06 sdb init: tty (/dev/tty1) main process (1972) killed by TERM signal
Jun  1 03:08:06 sdb init: serial (hvc0) main process (1974) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty2) main process (1975) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty3) main process (1977) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty4) main process (1980) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty5) main process (1982) killed by TERM signal
Jun  1 03:08:06 sdb init: tty (/dev/tty6) main process (1984) killed by TERM signal
Jun  1 03:08:12 sdb abrtd: Got signal 15, exiting
Jun  1 03:09:12 sdb init: Disconnected from system bus
Jun  1 03:09:12 sdb auditd[1824]: The audit daemon is exiting.
Jun  1 03:09:14 sdb kernel: Kernel logging (proc) stopped.
Jun  1 03:09:14 sdb rsyslogd: [origin software=”rsyslogd” swVersion=”4.6.2″ x-pid=”1839″ x-info=”http://www.rsyslog.com”] exiting on signal 15.
Jun  1 03:18:53 sdb kernel: imklog 4.6.2, log source = /proc/kmsg started.

What does this mean?

The system was shut down by an external person, Linode. I forgot they said that they would be having migration. Here’s the extract of the mail:

 * sdb – (7864940) System Shutdown – Completed Fri, 01 Jun 2012 03:08:07 GMT
* sdb – (7864941) Linode Initial Configuration – Completed Fri, 01 Jun 2012 03:09:28 GMT
* sdb – (7864942) Migrate Filesystem – swap from london62.linode.com – Completed Fri, 01 Jun 2012 03:09:29 GMT
* sdb – (7864943) Migrate Filesystem – Data Disk from london62.linode.com – Completed Fri, 01 Jun 2012 03:10:00 GMT
* sdb – (7864944) Migrate Filesystem – Backup Disk from london62.linode.com – Completed Fri, 01 Jun 2012 03:12:36 GMT
* sdb – (7864945) Migrate Filesystem – CentOS 6.2 64bit Disk Image from london62.linode.com – Completed Fri, 01 Jun 2012 03:15:02 GMT
* sdb – (7864946) Linode Migration Cleanup – Completed Fri, 01 Jun 2012 03:18:29 GMT
* sdb – (7864947) System Boot – SDB Config – Completed Fri, 01 Jun 2012 03:18:36 GMT

Happy as ever, I wrote a script to auto start a service if they have been found as not running, even if the server did not reboot. View it here.

Read More

Bash Script: Check service and auto start if needed


If you are running a service and for some reason the service gets broken, you will want it to automatically restart. For that reason, I have created a script as below. Feel free to amend the array autoservice as needed because in my case it is monitoring mysqld, cherokee and squid.

#!/bin/bash
#Add values to the array autoservice seperated by spaces
autoservice=(mysqld cherokee squid)

#the code
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
for i in “${autoservice[@]}”
do
:
service $i status # > /dev/null
if [ $? -eq 0 ]; then
echo $i “Running..”
else
service $i restart
fi
done

Created in /scripts/checkService.sh, you need to chmod 777 the file and add it to crontab to run every 5minutes via the following:

 

chmod 777 /scripts/checkService.sh

 

crontab -e

add this line to run it every 5 mintues

*/5     *       *       *       *       sh /scripts/checkService.sh

Read More

Backup/Restore MySQL Database Linux


Initiating a backup from phpMyAdmin is not always the best solution when are you running a medium to massive web database. If you are running your mysql server on a vps or a dedicated server on Linux, those commands will definitely be useful to you.

Backup:

mkdir -p /backup

cd /backup

mysqldump -u root -p database_name > database_name.dmp

Restore:

cd /backup

mysql -u root -p database_name < database_name.dmp

Read More