RATs - they are following me!

I'm guessign there's your problem:

@mrmat:

When I migrated the site, I took a backup of the SQL database and site files, and built a clean installation of ModX onto the virtual host and imported the CMS database.
so you created a clean setup, and then copied the infected website over? :roll:

deleted

I'm not sure what you have is RATs unless they have enough access to actually administer your server.

Find out how these files are getting in your web root and stop that happening. Read your web logs for clues. Do you allow users to update files of any kind or allow them to enter data that could be interpreted as PHP when it's displayed later?

Post the URL here and someone here will have a look at it.

PHP Shells and stuffs can be made pretty much useless if you disable these functions. With PHP-FPM:

php_admin_value[disable_functions] = apache_child_terminate, apache_setenv, define_syslog_variables, escapeshellarg, escapeshellcmd, eval, exec, fp, fput, ftp_connect, ftp_exec, ftp_get, ftp_login, ftp_nb_fput, ftp_put, ftp_raw, ftp_rawlist, highlight_file, ini_alter, ini_get_all, ini_restore, inject_code, mysql_pconnect, openlog, passthru, php_uname, phpAds_remoteInfo, phpAds_XmlRpc, phpAds_xmlrpcDecode, phpAds_xmlrpcEncode, popen, posix_getpwuid, posix_kill, posix_mkfifo, posix_setpgid, posix_setsid, posix_setuid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, syslog, system, xmlrpc_entity_decode, mail

Note that the last value here is mail, this will prevent the mass mailing but it is a bit tricky if you also send out legitimate mail through PHP. You may be able to reconfigure your application to send mails through an external mail server instead. Either way, the above should help, even if you exclude the mail function :)

GL

@zunzun:

1) Run a cron job that counts the number of files in the subdirectory in question, say once per minute or so. When the file count changes, it can email you a time near which to look through log files.

Or something like this:

http://www.cyberciti.biz/faq/linux-inot … rectories/">http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/

and cp all relevant logs to a safe location, dump netstats and whatever else…

I hope/presume that you keep the online applications up-to-date, in this case ModX and whatever eles you maybe running. Open source applications have the problem that if you do not patch them amateur hackers can gain acces through known security holes.

@saman007uk:

I'm guessign there's your problem:

@mrmat:

When I migrated the site, I took a backup of the SQL database and site files, and built a clean installation of ModX onto the virtual host and imported the CMS database.
so you created a clean setup, and then copied the infected website over? :roll:

I don't think there's my problem. I may be wrong, but the infections in question took the form of some base 64 encoded php files in various directories under the web root of the given vhost.

When I inspected these naughty php files (FTP to my laptop), they were flagged up in my virus scanner (Avast for Win 7). I ran the sql backup file through the same virus scanner and it looked clean. Also I checked it for any base 64 encoded crap and there wasn't any.

But yes - I've no doubt it would be better to re-author all my sites from scratch. Something that would take weeks…

Thanks for the helpful words, all.

Here's what I did:

• installed incrontab, this is awesome as it detects file changes in a given directory then runs a script accordingly which emails me about any changes

• I keep postfix stopped, then login periodically to check the mailq is clean then start postfix when I'm happy, then stop it again (not ideal, right?)

• When I notice huge traffic spikes, I try to log in and run iftop -n to see where connections are coming from and how much bandwith they use, then ban them using e.g. iptables -A INPUT -s 65.23.85.110 -j DROP

• I installed and ran phpsecinfo to help secure my php installation

The above has basically made the problem go away for the last 4 weeks now. But it's not ideal, right?

I've looked through apache logs and cross-referenced them with the datetime of the infected file to see what IP address it came from. I can check this is when the huge traffic spikes start.

But my expertise stops there.

Can someone help me to determine exactly how the files got there? Through what process? What insecurity?

Any help greatly appreciated.

mrmat,

It's almost impossible to determine what happened at this point, even if we had access to the server and all of the logs. I assume you've changed all the passwords for accessing the site, including root, FTP, email users, etc.

If all of your sites are running of the same CMS and only one is getting infected it might help you find the issue by focusing on the differences between the sites (file & directory permissions ownership, CMS settings, software versions, etc).

Have you searched for security issues and/or other ModX sites being infected in this way?

MSJ

What version of modx are you using?

There was an execute arbitrary code thing until version 0.9.6.2.

http://www.cvedetails.com/cve/CVE-2008-5938/

Thanks all

I'm running version 1.0.9 (from only a month ago, I think there's a new version now, I'll update), either way it's well past v0.9.6.2

@Main Street James, yep - I've changed every single credential on the server and replaced them with ridiculously strong passwords. Also I have standardised on a file/dir permissioning model across all sites and researched permissions quite heavily, not to mention standardising on CMS settings etc.

I think I will go through once again with a fine toothcomb all the scripts, modules, plugins, etc. to check their versions.

Thanks, I'll keep battling. Problem is, whenever there is a compromise, I have to set aside more than a few hours to rebuild a second server from scratch then migrate sites one by one - argh!

mrmat,

If you are still getting compromised on that one site:

• set up a brand new site.

~~ use a subdomain or even a throwaway domain (still cheap based on the amount of time you're losing).

• clean up the infected files on the problem site.

• port only the data from the problem site to the new site.

• sit and wait for the problem site (or both sites) to get compromised.

If you only bring over the data from the problem site and the new site gets compromised then you know it is something in the data. If the new site doesn't get compromised then you copy the current data from the problem site over to the new site and move/rename the new site's code to replace the problem site (be sure to have a backup of the problem site just in case). You may have to chown the files so the newly copied files match their new location/owner.

If the problem is hidden in a post, comment, etc of a specific page of the infected site then the new site would eventually end up compromised when the remote system requests that page/post (it wouldn't happen under the test domain/subdomain because it wouldn't get requested on that unknown domain/subdomain).

Have you checked the logs to see what page on the problem domain was accessed around the same time as the timestamp of the infected files that end up on your site?

BTW, have you checked the .htaccess file of the problem site? The shenanigans could be hidden in there. Have you checked all your cron jobs? Are there any you don't recognize? If the infection happens very regularly or predictably it may be a cron job.

MSJ

@MSJ

Really great post; some very sensible ideas I hadn't thought of.

One question, when you say "only bring over the data", do you mean, copy the files but not the website database?

That strikes me as a really good idea, one I hadn't thought of.

Thanks again,

Mat

Only bring over the database, but don't bring over the executable code. (This assumes you've inspected the contents of the database for, say, unauthorized users and the like.) Re-deploy the executable code from trusted sources.

mrmat,

hoopycat is correct, I was suggesting only bringing over the database and the supporting content (images, etc), but not the code.

It's hard to go through all of the DB content looking for an infected entry, but if you can check the logs to see if there is a correlation between the timestamps of the infected files and the access of an older post/page then you might be able to figure out which entry it might be (if it is a DB entry).

We host several hundred websites with various CMS's. We've been hacked several times.

There were only two methods used:

1) File manager in CPanel, some exploit or root password (probably exploit, I dont think anyone would find out root password)

2) insecure CMS and their plugins - they were all Wordpress, Joomla and Zencart. In Joomla it was an exploit in the older version of the plugin JCE Editor for the Wysiwyg editor. Don't know about Wordpress or Zencart.

We don't use these CMS's anymore, they're older sites not upgraded for some time.

For 1) we now use CSF firewall to block all ports except http/https/email and only allow on approved IP addresses and hostnames access to CPanel, SSH, FTP etc. This has a massive benefit on securing the servers. We don't use CPanel on new servers either, but use Virtualmin.

For 2) we upgraded the offending systems/plugins and not seen hacks since

If we do get hacked now its likely because of 2) on old systems not upgraded for sometime. We have also put htaccess directives to disable PHP in the folders they upload to (always images and tmp) in case they do get in they can't do anything.

So could be an exploit in ModX. The software is the most common way they got in with us.