Server is black

When you as an independent developer always when faced with this kind of problem, previously considered a small probability event is always a point in time flocked to test your patience, before the waves just experienced a Trojan Cry (See article cheetah master cleanup we can trust it?  ), and this time met server is black.

Deploy server configuration management and general services for people who write code for a natural mention , but relatively professional programmer operation and maintenance personnel is really less a sense of security, always thought to be attacked server is a small probability event. It used to be so considered "host on the Internet, but why so many hackers eyeing your host is? This is not the same as what a major prize, so lucky I have it?" Although some time ago as their products are already in the Trojan a jackpot. So the psychological luck, be able to save it the easy way, it can be used to deploy first service on the line, which is actually in line with my minimalist acting style. Only experience in order to grow, the remark was right, this is not thanks to clever hackers give me a lesson, from doing any server deployment and management of time in my head can have a string children. " safety consciousness". The way it is. . .

A few days ago, as usual, visit my website products  http://xbrowser.me  , suddenly appeared in page 404 page not found. Strange, how could this be? I have checked the domain name-check the input, yes indeed own website. Then I suddenly surprised psychological, gasped, Could that be because a few days ago published an article commemorating I want to waste product features - a key over the wall  so quickly GFW noticed? Give me a DNS hijacking? To test this idea I immediately opened the computer's VPN program, visit my website via VPN again (to avoid the GFW DNS hijacking). Presentation of the web page is still "404 page not found", indicating that the problem is not DNS hijacking, GFW did not notice me. At this time I do not know hi or worry, but this is exactly what causes it? To find the truth I immediately logged into your server.

First I went to see the webserver root directory, and she saw the site files were deleted. Fortunately, I have a backup program, this loss I can afford, what key the hacker's intent is to it? Home tampering? It does not make sense ah, for me this is not a well-known two did not influence the small site does not make sense ah. Well, I do not guess the reason, this guy so much action I would be glad, or alone, my little safety awareness is as broiler certainly notice it. This time I quickly see who has logged server

1. $ last | more

2. chengkai pts/0 111.199.208.96 Mon May 18 14:41 still logged in

3. chengkai pts/4 192.154.200.61 Mon May 18 12:39 still logged in

4. chengkai pts/4 111.199.208.96 Sun May 10 02:40 - 14:39 (1+11:59)

5. chengkai pts/4 111.199.208.96 Sat May 9 14:02 - 14:03 (00:00)

6. chengkai pts/9 111.199.208.96 Sat May 9 00:15 - 02:31 (02:15)

From the point of view logs, are my own user name, but it is clear that there is a different source IP address, I quickly checked the ip address of the source, the Beijing Netcom's IP processing another different IP to the Taiwanese. It is clear that hackers have broken my root password in order to deceive the public and then use my user name for online operations, but even more amazed that at this time this man actually is online.

chengkai pts/4        192.154.200.61   Mon May 18 12:39   still logged in

Nima This is really a big girl married the first time, ah, this is the spot to duel ah. The first crossed my mind I have to put the guy next to Tixia Qu, but another thought no, this guy has got my root privileges, I Tixia Qu is likely to re-login system will angrily destroy, or to change the root password and see if there is anything left to say back door, take a look at what the other special service process does not start.

$ netstat -nl

These ports are 80,22,1723 I know webserver, ssh and vpn. The few remaining large port number in particular is questionable.

1. Active Internet connections (only servers)

2. Proto Recv-Q Send-Q Local Address Foreign Address State

3. tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN

4. tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

5. tcp 0 0 0.0.0.0:1723 0.0.0.0:* LISTEN

6. tcp 0 0 127.0.0.1:48988 0.0.0.0:* LISTEN

7. tcp 0 0 127.0.0.1:52732 0.0.0.0:* LISTEN

So then I use "lsof" command to see these big port number is what the process started?

1. lsof -i :48988

2. ruby 13727 chengkai 9u IPv4 98307068 0t0 TCP localhost:58185 (LISTEN)

3. lsof -i :52732

4. ruby 18258 chengkai 9u IPv4 97802878 0t0 TCP localhost:52732 (LISTEN)

Fortunately, all ruby ​​process, I guess http request should be forwarded to the passenger rails listening port, in order to prove I guess I stopped the nginx service, really ruby ​​port number occupied by the process gone.

After removing the suspect hackers listening port, the next will take a look at my server when the "chicken" to use, the best way to detect this problem is to look at the flow outlet outside, this time I thought iftop command.

1. $ sudo iftop -np

2.  

3. 195kb 391kb 586kb 781kb 977kb

4. └──────────────────────────────────┴───────────────────────────────────┴──────────────────────────────────┴───────────────────────────────────┴───────────────────────────────────

5. 96.126.127.11:7000 => 101.219.21.98:50726 30.1kb 6.03kb 1.51kb

6. <= 1.12kb 230b 57b

7. 96.126.127.11:ssh => 111.199.216.183:63446 5.09kb 5.91kb 6.13kb

8. <= 416b 291b 250b

9. 96.126.127.11:7000 => 101.219.21.98:54488 25.3kb 5.06kb 1.27kb

10. <= 1.12kb 230b 58b

11. 96.126.127.11:7000 => 101.219.21.98:36312 24.3kb 4.87kb 1.22kb

12. <= 1.12kb 230b 58b

13. 96.126.127.11:ssh => 222.186.21.250:56628 5.20kb 2.61kb 668b

14. <= 3.14kb 1.44kb 368b

15. 96.126.127.11 => 187.184.246.118 8.39kb 3.65kb 3.65kb

16. <= 0b 0b 0b

17. 96.126.127.11:7000 => 101.219.21.98:34993 15.8kb 3.17kb 811b

18. <= 1.13kb 231b 58b

19. 96.126.127.11:7000 => 103.254.203.177:58825 8.54kb 1.71kb 437b

In it, and did not find anything unusual exit out of traffic and connections, but also carried out some routine examination, also found no particularly unusual problem.

1. # 查看有无异常进程

2. $ ps aux

3. ......

4. # 查看系统资源占用有无异常

5. $ top

6. ......

7. # 有没有新增异常用户

8. $ cat /etc/passwd

9. ......

10. #查看了root用户的命令历史记录,当然这个对稍有经验家伙是没有意义的，拿到了root权限后可以清理任何痕迹

11. # history

12. ......

I know two things it needs to do next, change the username and password for the root of my current user and the intruder Tixia Qu. After modifying the password for the server, I entered the following two commands in the terminal.

1. $ write chengkai pts/4

2. what are you fucking about

3. #结束消息

4. ctrl+d

5. # 踢掉在线终端

6. $ pkill -kill -t pts/4

The first command is to send a message to the intruder "what are you fucking about", while the second command followed by the intruder kicked down. This time can not help but have a trace of satisfaction, it is conceivable they'll get a look at the news and I kicked terminal, of course, I think he might not be so easy to let go . (In fact, I know this is very dangerous provocation, the other side had got the root user privileges, if you set up somewhere a few holes will be very difficult to find, the system can easily be broken again). I opened the authentication log to log in again quietly waiting to crack each other's, and she soon came the other side, but launched a different host ssh brute force.

1. $ sudo tail -f /var/log/auth.log

2.  

3. Received disconnect from 222.186.21.243: 11: [preauth]

4. May 18 14:36:52 localhost sshd[16428]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.243 user=root

5. May 18 14:36:54 localhost sshd[16432]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.243 user=root

6. May 18 14:36:56 localhost sshd[16432]: Failed password for root from 222.186.21.243 port 56722 ssh2

7. May 18 14:37:01 localhost sshd[16432]: message repeated 2 times: [ Failed password for root from 222.186.21.243 port 56722 ssh2]

8. May 18 14:37:01 localhost sshd[16432]: Received disconnect from 222.186.21.243: 11: [preauth]

9. May 18 14:37:01 localhost sshd[16432]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.243 user=root

10. May 18 14:37:03 localhost sshd[16438]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.243 user=root

11. May 18 14:37:05 localhost sshd[16438]: Failed password for root from 222.186.21.243 port 45780 ssh2

12. May 18 14::42:26 localhost sshd[18573]: Received disconnect from 222.186.21.244: 11: [preauth]

13. May 18 14::42:26 localhost sshd[18573]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.244 user=root

14. May 18 14::42:29 localhost sshd[18579]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.244 user=root

15. May 18 14::42:30 localhost sshd[18579]: Failed password for root from 222.186.21.244 port 38062 ssh2

16. May 18 14::42:35 localhost sshd[18579]: message repeated 2 times: [ Failed password for root from 222.186.21.244 port 38062 ssh2]

17. May 18 14::42:35 localhost sshd[18579]: Received disconnect from 222.186.21.244: 11: [preauth]

18. May 18 14::42:35 localhost sshd[18579]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.244 user=root

19. May 18 14::42:39 localhost sshd[18585]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.244 user=root

20. May 18 14::42:41 localhost sshd[18585]: Failed password for root from 222.186.21.244 port 50864 ssh2

21. May 18 14::42:47 localhost sshd[18585]

In a sense this is a good thing, because it shows the other side and have not had time to placement backdoor, but to take brute force approach. Then the next thing I do is to be added to the other IP "hosts.deny"

1. $ sudo vi /etc/hosts.deny

2.  

3. ALL:110.164.67.47

4. ALL:222.89.166.12

5. ALL:221.229.166.30

6. ALL:58.218.205.69

7. ALL:58.218.204.239

8. ALL:58.218.211.155

9. ALL:222.186.21.236

10. ALL:58.218.204.225

11. ALL:58.218.204.241

12. .......

Because hackers may try brute force root user, so even prohibit root ssh login, change sshd_conf file.

1. $ sudo vi /etc/ssh/sshd_config

2. #把PermitRootLogin 属性 yes 改为 no

3. PermitRootLogin no

This, I can think of to detect and prevent the other means have been almost the same, it's time for a minimum set of security awareness to be summed up.

• Set username and password must be sensitive combination of alphanumeric and special symbols, increase the difficulty of brute force, it is possible, then you can change the password on a regular basis.
• Prohibit ssh root login
• Regular maintenance tool hosts.deny file, you can choose to install some third-party maintenance hosts.deny automatically according to some rules, such as "DenyHosts"
• In order to avoid hackers scan for known vulnerabilities port server program, modify the default port service program is good, such as ssh service does not use port 22
• Set iptables firewall to open some general rules, ubuntu system can use ufw

Here are some suggestions of users and added:

• Prohibit password, modify sshd_config same set properties "PasswordAuthentication no", then the public and private keys manner landing.
• Use fail2ban, fail2ban can monitor your system logs, and then match the error message log (regular expression matching) the implementation of appropriate shielding action (under normal circumstances is to call the firewall shield), such as: when someone test your SSH, SMTP, FTP password, as long as you reach the preset number of times, fail2ban firewall shield will call this IP, but also can send e-mail notification system administrator, is a very practical, very powerful software!