Debian server crashing after a lot of data transfer

I’m running 2.6.32-5-xen-amd64 version of Debian….

Recently I’ve had quite of a surprise when all of a sudden my DNS server crashed – to make matters even worse this happened during a presentation of a newly created web page, making the presentation impossible.

I rushed to connect to the server only to realize that the VM that is running and everything seems to be just fine.
Before further examination I’ve decided to restart the whole machine (running a couple of VMs).

After restart and successfully booting of all the services / VMs – everything seemed to be in order. All of our online web services started running and it looked like DNS started working again.

I was completely confused…and started to check logs (one by one) in order to find out of what was going on.

This is what caught my eye in /var/log/syslog

Jun 17 03:00:02 **YourServerName** kernel: [10600465.191348] EXT4-fs (dm-15): ext4_orphan_cleanup: deleting unreferenced inode 131315
Jun 18 03:00:05 **YourServerName** kernel: [10686867.920018] EXT4-fs (dm-15): ext4_orphan_cleanup: deleting unreferenced inode 131315
Jun 19 03:00:03 **YourServerName** kernel: [10773266.248593] EXT4-fs (dm-15): ext4_orphan_cleanup: deleting unreferenced inode 131315
Jun 20 03:00:02 **YourServerName** kernel: [10859664.573971] EXT4-fs (dm-15): ext4_orphan_cleanup: deleting unreferenced inode 131315

After googling it…I found out it is a issue for a lot of IT folk out there

http://ubuntuforums.org/showthread.php?t=1861588

Some are suggesting that it is hardware related (HDD bad sectors)

https://bbs.archlinux.org/viewtopic.php?id=95683

It seems like a lot of data transfer locks the HDD into read only mode!

The only thing that I found in common is that it happens when a lot of data transfer(heavy hard disk activity) is being done 12h / 24 h before on Unix machines.
The night before I have scp a DB from my computer to the server and then later on to another server that acted as a host.

Read More