Portal Home > Knowledgebase > Articles Database > Server is running a Dos attack
Server is running a Dos attack
| Posted by lesley, 01-14-2014, 07:09 AM |
| Hi
I have a client whose hosting company complain his server is running a DoS attack - the evidence given indicates a DoS on an IBM IP but presumably the target could change?
(a) anyone aware of this specific DoS attack?
(b) client had allowed root access via ssh and there are a number of attempts to access the root account over ssh. This has now been stopped.
(c) client has not installed rkhunter, tripwire or any other intrusion detection software. I installed rkhunter and ran w/out initialising it's db to find out if any rootkits around but it didn't detect any.
So I'm left with running wireshark on the interface.
It's a Centos 5.5 box so what other methods can I use to detect what is controlling/initiating/running the DoS attack?
Do I have to consider this server compromised and re-image?
To me that's the safest bet but there's a lot on there and I would prefer to know where that attack is originating from on the server e.g. at least tie it down to a script or scripts.
Looking forward to your replies.
(Sorry if this is the wrong place to discuss this.)
Kind regards
Lesley
|
| Posted by whmxtra, 01-15-2014, 03:26 AM |
| This is in the wrong forum, should have been in dedicated I think, Mods will most likely move it.
To answer your question, yes if the server was compromised enough to DoS then it's probably too damaged to fix so your safest bet is a reinstall and restore backups (inspect the backups for compromised files too) and then to secure the server with a proper firewall and security to prevent it happening again.
SSH access in itself isn't bad if it's jailed properly, preferably cloud linux with cagefs and just keep an eye on who has access and what they are doing with it.
|
| Posted by foobic, 01-15-2014, 04:03 AM |
| Moved to Hosting Security and Technology
I wouldn't consider it a lost cause yet - from what you say it could be just a compromised website. Look for perl scripts in /tmp and any upload directories, or any changes to .php files.
If you need expert help you may want to put in a request on the Systems Management Requests forum.
That's normal for any machine running SSH on port 22. Provided they didn't use a weak password such attempts are unlikely to succeed.
|
| Posted by kevincheri, 01-15-2014, 02:56 PM |
| yep, it should be a site (or multiple) hacked and being used as a source of DOS. You would need to check the network logs live (/var/log/messages if Cpanel and CSF) and the processes currently running to see which user causing that. Do a 'lsof ' and find the folder/files responsible and check from there.
|
| Posted by grapenut, 01-15-2014, 03:09 PM |
| that's lsof -p (i like -np though)
but do follow what this guy is saying. it's literally as simple as just finding the process and any breadcrumbs it leaves behind. typically they will curl/wget a file and then delete it after it's launched as a proc. but it should give you some insight with timestamps and a case of the "well that just doesn't look right". match up the malicious content with timestamps in your access logs and of course check any outdated CMS software and assocaited plugins, themes, and other addons.
kill the processes and directly follow it up with a ps faux to see if it continues to come back. usually you'll see the code used to download the content to your server. if not, there is always our good friend grep and digging through eval(base64_decode and {eval(stripslashes returns. these are what i typically see as for how DoS content makes it way onto your server. other than the obvious vulnerabilities or compromised accounts that allowed the initial files to make their way in. but those are typically easy enough to find unless the logs have been rotated or wiped.
Last edited by grapenut; 01-15-2014 at 03:14 PM.
|
| Posted by kevincheri, 01-15-2014, 03:53 PM |
| yep exactly, my bad.
|
| Posted by CircuitoX, 01-15-2014, 08:11 PM |
| Hi Lesley you can see this Thread:
http://www.webhostingtalk.com/showth...stmenu_8973985
|
| Posted by AttackerNET, 01-15-2014, 09:22 PM |
| Can you make sure that their DNS server is configured properly and recursion is disabled or restricted? Any control panel being used on this server? What about kernel and php versions on this server? I would recommend to review the cronjob logs and search for all .bash_history files and review it all as well as checking all running processes.
|
| Posted by lesley, 01-24-2014, 10:02 AM |
| Hi
I've made *some* progress but I have yet to clear the system entirely.
I found a screen running as root and a php script exploitx.php running testing for phpmyadmin vulnerabilities. I killed off all the exploitx.php processes and the screen.
1. The screen was working in the homedir /var/html as root.
2. The environment was modified to show a reduced history of a few lines.
3. A file, pma5.tgz, has been uploaded and extracted to the homedir. This contains the exploitx.php script along with a script that modifies the cron on the system.
4. A file update is called via the script a.
update existed on the system in these places
/etc/cron.hourly/update
/usr/lib/mailman/bin/update
/root/update
/update
of which only the mailman candidate looked okay.
5. I've discovered that a number of system files have been altered via the script mcelog also existing in /etc/cron.hourly and I now believe the ssh environment to be compromised. The script mcelog isn't deletable and any attempts to chattr result in the word killed being displayed and the attributes not being changed at all.
e.g.
lsattr mcelog
su--ia------- mcelog
chattr -suia mcelog
Killed
lsattr mcelog
su--ia------- mcelog
chattr isn't one of the scripts shown as replaced in mcelog.
There may be lots of compromises on this system and I really do not want to re-image.
If anyone can comment on this exploit and particularly any advice in handling it I'd be glad to hear it.
Kind regards
lesley
|
| Posted by foobic, 01-24-2014, 10:52 AM |
| As you've now established that it's a full root compromise you really have no choice but to re-image. You've found several issues but there may be many more you haven't found. The only question is whether you want to continue investigations to try and find the original entry point.
|
| Posted by simon_enzu, 01-24-2014, 04:26 PM |
| Best option would be reload your server and restore he accounts from backups. Other option is to find out the source of DDOS.
I would suggest you to install iftop command which would list traffic through each interface and the amount of traffic inbound/outbound from/to specific IP's. Once you get a high traffic IP, you can check the logs to see how they are generated. If the connections are through specific ports, we can block that ports using firewall. Also it is good to scan your machine for any malwares or rootkits using maldet, chkrootkit or rkhunter.
|
| Posted by lesley, 01-27-2014, 08:40 AM |
| Hi everyone
Thank you for your help and all the suggestions.
I have yet to hear back from the client since I told him the server was root compromised and it needed a re-image.
The problem with using anything from that box at the moment is that it looks like he has changed chattr, ssh, scp, sftp, /bin/bash and /bin/sh and made his replacements undeletable.
My only option would be to use a tsh or other shell scripting language and upload new copy of chattr to get everything else out the way. I still don't have a clue where the entry took place and I'm assuming in the minimum the hacker is now aware I am on the box.
The client doesn't have any server maintenance schedule in place or backup schedule in place which doesn't help matters.
I feel there would be so many security holes to fix a re-image is the only way out. An update might close most holes up but not if the hacker is still able to get in through anything he has put on the system.
So I think a re-image, immediate intrusion detection and rootkit checks in place plus other hardenings and an extremely careful re-install of data from back-up is the only way of out this.
Simon: thanks for the tips about iftop and maldet. I'll look at those shortly. I typically use firewalls via iptables, intrusion detection systems and rootkit hunters but wasn't aware of maldet or iftop.
The hacker is using port 80, shutting that port off is a good idea if nothing more than to get the attention of the client, stop the attacks going out and protect anyone using the e-commerce on there.
Thanks all once again.
Kind Regards
Lesley
|
Add to Favourites 
Print this Article
Also Read
Mochahost (Views: 581)