What To Do If You Think Your Linux Server Was Hacked
There are a number of things you can do if you think your Linux box was hacked. A common myth is to simply and quickly reinstall the OS, however that is the exact opposite of what you want to do, at least initially. What you want to do ASAP is take the box offline. Before you do that, you have an option, you can get some data on what’s running and what IPs are currently connected. For example by running these commands: lsof, netstat -anpe, ps aux. If you are already logged in, it would be a good idea to run those, if you are not logged in you may want to just pull the plug on the machine. This is one case where you want to pull the plug, or if its on a remote rebooter
turn off that rebooter port, rather than running halt or shutdown from the command line. Now if you do decide to run the commands to see whats running, you should send the output to another server, for example by using netcat. You should always have an unblocked outgoing port, to be used for netcat and then further secure that, by adding an ACL to only allow traffic to your netcat server on that port. In order to preserve a compromised system, you don’t want to write any new data to the drive. You also don’t want to remove anything at this point, even if you see files you think were placed by a hacker.
At this point you should notify your customers and anyone else who may have been affected that there may have been a security breach. Notify your hosting provider if appropriate, any security professionals you work with, and
if you are meeting compliance, for example PCI compliance for credit cards, you will want to notify your merchant account provider and/or the payment card companies, such as Visa.
Also notify law enforcement if appropriate. If needed, wait for information from the other parties on how you should proceed. Yes, this does mean taking the box offline, but its an
In the event your cannot take the box offline, you need to at the very least want to close all ports except SSH, allow SSH only from your IP, allow SSH via key only, and not via password authentication. This
method is not recommended and any evidence which could be used later will likely now be considered contaminated. The other ports should not be reopened until you are confident the
vulnerability has been fixed, if root access was gained, the other ports should never be reopened until you have a new server up with a fresh install of the OS.
Next make an image of the drive. Keep the original drive in a secure location, and maintain a chain of custody on it. The copy of the drive will be what you analyse. And you will
want to do so with a Live or Rescue version of Linux, that runs from a CD or DVD. Now, gather all the logs from the software you have monitoring your server, ex. RKhunter, Snort,
AIDE, Samhain, Tripwire, Osiris, Integrit, Chrootkit, LogWatch, etc. You should have at least several of those in an offsite location and be able to produce them quickly if needed.
In addition look through the logs on the image of the compromised drive.
Additionally on the image you want to look for version of the software and packages which are installed, for example, was the kernel up to date? Check the version on all the packages, php, perl, etc. Look in the
world writeable directories, such as /tmp, /var/tmp, /dev/shm. Analyse the output from the commands you ran above using netcat, if you ran them. Also look at shell histories, set uid files, etc.
Now document everything you found, including any hints, hunches, or gut instincts you may have about how the hacker got in. Write down what files you think he put there, or what processes running you thing were
his. Write down if the hacker got root access or not. Normally you have a good idea of how they got in.
Only after you are certain you know how the hacker got in, and certain the vulnerability will be patched, then you can bring that server back online. If you think the hacker may
have gotten root access, you will need to reinstall the OS, all packages should be reinstalled/recompiled, any files copied from your image, must be inspected to ensure they were
After completing the above steps you will see the importance of the thoroughness of taking them time to complete them all. And you will have several good reasons why you shouldn’t immediately restore the OS, and why you
should have the server offline as soon as possible, and don’t bring it back online until you have completed your investigation and applied the appropriate fixes to ensure it is safe to do so.