Posted: Sat Apr 25, 2009 10:17 pm Post subject: Security breached on Webserver - What went wrong?
First of all a small introduction on my side so you know who you're dealing with. I'm a 21 year old support engineer working at a rather large company. In my spare time I run a forum and some other small websites. For this I have bought a virtual private server which has been running quite smooth for about a year now. I'm quite familiar with the Microsoft Windows operating system to the extend where I know more about Windows, than i know about my own mother. And I have some mild experience with Linux i.e I (think I know) know what I'm doing.
The VPS I mentioned earlier has sadly enough been hacked. After looking trough the logs I have found some interesting records which pretty much do my head in. I have no idea where the leak is.
What is running on the VPS?
All of this is managed via ye' olde Plesk 9.0 installation
What exactly did the exploit/attacker do?
It modified a lot of index*.* files. From HTML to PHP files. None was left unharmed. The attacker basically added some extra text to the file, linking to an iframe that contained a page which is using a vulnerability in Adobe PDF. It basically installs malware on your PC which is quite hard to to remove. (Trust me, I've been there). The line of text added to files is as following:
What is the scale of the exploit?
As I mentioned before it left no index*.* file unharmed. From index_test.php to oldindex.html. All was tainted.
How does the exploit work?
After carefully analyzing file changedates and accesslogs I figured out it's using FTP access to modify files. The attack took place on April 24th around and about 7:38pm GMT+1. Looking at the Last modified property I found this date. After I found these dates I skimmed trough my logs to discover these lines:
Apr 24 19:36:15 h1369581 proftpd: removed_hostname (22.214.171.124[126.96.36.199]) - FTP session closed.
Apr 24 19:36:16 h1369581 proftpd: pam_unix(proftpd:session): session closed for user user1
Apr 24 19:36:16 h1369581 proftpd: removed_hostname (188.8.131.52[184.108.40.206]) - FTP session closed.
Apr 24 19:36:31 h1369581 proftpd: pam_unix(proftpd:session): session opened for user user1 by (uid=0)
Apr 24 19:36:31 h1369581 proftpd: removed_hostname (220.127.116.11[18.104.22.168]) - USER user1: Login successful.
Apr 24 19:36:31 h1369581 proftpd: removed_hostname (22.214.171.124[126.96.36.199]) - Preparing to chroot to directory '/var/www/vhosts/user1'
Apr 24 19:36:47 h1369581 proftpd: pam_unix(proftpd:session): session closed for user user1
Apr 24 19:36:47 h1369581 proftpd: removed_hostname (188.8.131.52[184.108.40.206]) - FTP session closed.
Apr 24 19:36:49 h1369581 proftpd: pam_unix(proftpd:session): session opened for user user1 by (uid=0)
Apr 24 19:36:49 h1369581 proftpd: removed_hostname (220.127.116.11[18.104.22.168]) - USER user2: Login successful.
Apr 24 19:36:49 h1369581 proftpd: removed_hostname (22.214.171.124[126.96.36.199]) - Preparing to chroot to directory '/var/www/vhosts/user2'
Apr 24 19:36:52 h1369581 proftpd: pam_unix(proftpd:session): session closed for user user2
Please note that I removed the hostname, and the usernames for my own peace of mind. Also please note that on the system itself username and URL have little to NO relation.
For example if one of the domains was www.google.com, the username would not be google but for example 'Bob'. Although I do not use the most strongest passwords one can think of; it does contain special characters. Seeing as this attack took place within a matter of minutes the passwords where not brute forced. Which brings up my question:
Where would the breach in security be!
I am absolutely clueless. As far as I know Linux/Plesk/ProFTPD does not store user names and passwords unhashed. Nor should any FTP account be able to peek outside their var/www/vhost/url/ folders. There is just one shell account for the root user, whose password contains upper- and lowercase letters, numbers, and special characters.
Prior to the attack, starting on April 7th, the logs does show this:
Apr 7 04:40:20 h1369581 sshd: reverse mapping checking getaddrinfo for 103.hosting-5.xtream.co.il [188.8.131.52] failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 7 04:41:52 h1369581 sshd: reverse mapping checking getaddrinfo for web.fusionity.com [184.108.40.206] failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 7 04:43:19 h1369581 sshd: reverse mapping checking getaddrinfo for server1.mirror-reflections.com [220.127.116.11] failed - POSSIBLE BREAK-IN ATTEMPT!
Apr 8 06:51:09 h1369581 sshd: Invalid user admin from 18.104.22.168
Apr 8 06:51:35 h1369581 sshd: Invalid user admin from 22.214.171.124
Apr 8 06:51:57 h1369581 sshd: Invalid user admin from 126.96.36.199
Apr 8 06:52:40 h1369581 sshd: Invalid user admin from 188.8.131.52
Then some more days later the attack changed:
Apr 17 04:18:38 h1369581 sshd: Invalid user borna from 184.108.40.206
Apr 17 04:28:31 h1369581 sshd: Invalid user botan from 220.127.116.11
Apr 17 04:42:07 h1369581 sshd: Invalid user bowen from 18.104.22.168
Apr 17 04:42:15 h1369581 sshd: Invalid user fluffy from 22.214.171.124
I am guessing that at this point it's using a dictionary or wordlist to try to find a valid user account.
However shortly after that it changes yet again
Apr 17 04:42:44 h1369581 sshd: pam_unix(sshd:auth): check pass; user unknown
Apr 17 04:42:44 h1369581 sshd: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=desdemona.fs.witz-inc.co.jp
Apr 17 04:42:46 h1369581 sshd: Failed password for invalid user library from 126.96.36.199 port 37872 ssh2
Apr 17 04:42:48 h1369581 sshd: Invalid user info from 188.8.131.52
Small note: I am only posting small samples, right now my logauth.log contains over 100.000 entries.
Then some minutes later it changes to
Apr 17 04:44:49 h1369581 sshd: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=desdemona.fs.witz-inc.co.jp user=root
Apr 17 04:44:51 h1369581 sshd: Failed password for root from 184.108.40.206 port 43066 ssh2
So by now it's trying to brute force itself into the root account.
Then out of nowhere my log becomes like this
Apr 20 03:24:01 h1369581 CRON: pam_unix(cron:session): session closed for user root
Apr 20 03:30:01 h1369581 CRON: pam_unix(cron:session): session opened for user www-data by (uid=0)
Apr 20 03:30:01 h1369581 CRON: pam_unix(cron:session): session closed for user www-data
Apr 20 03:39:01 h1369581 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
As far as I can tell the root password has not been guessed so far. I don't know why this happening. The previous shell logins stopped, but now it's trying to attack via my cronjobs, or my server is just being silly.
It's probably the latter.I haven't found any jobs that are out of the ordinary.
It continues on like this untill it "guesses" FTP usernames and their correct passwords out of the blue.
I am puzzled as to where the leak is as I can't find any traces of unauthorized access.
if I sit back and re-upload the corrupted scripts it is bound to happen again, as it looks like a simple script compromised my server.
Not so much a hacker who actually invested time into breaching trough my security measurements.
So my question to you guys is; where is the security leak? Or in which direction should I look? I've never had this happen to me before so I want to make this as much as a learning experience as possible. If you require more information or complete logs please do not hesitate to ask as I am more then willing to send you them if it will solve this situation
I thank you greatly in advance, and I apologize for bluntly intruding your forums asking -hopefully not to- annoying questions.
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum