Get into the habit of writing together! This is the first day of my participation in the "Nuggets Daily New Plan · April Update Challenge", click to view the details of the event .
Summary: I rented a server on HUAWEI CLOUD to build minecraft public service. After the server ran stably for 96 days, I was attacked by DDOS for the first time on March 27, 2022. The attack started at 11:07 and ended at 11:17 when Huawei's traffic was blocked and locked into a black hole.
abnormal detection
Opened the QQ group at noon, and found that a large number of players responded that the game server could not be logged in, and immediately suspected that the server process had collapsed, but in the QQ group, it was found that there were still players playing on the server. ssh login background view, but found that the ssh service has been stopped. This is a very strange performance. I immediately logged into the HUAWEI CLOUD console and found a large number of early warning reports and notifications. That's how I know what happened. The server suffered a DDos attack and was locked into a black hole by the service provider. Unblock naturally after 24 hours. As a result, all incoming and outgoing traffic is banned, but the game service is still running, which is why there are still players during the service period, but others cannot enter.
re-investigation
According to the traffic analysis and source tracing methods provided by HUAWEI CLOUD, I found the source IP of the attacker. I have some knowledge of DDOS before, usually a large number of broilers, clustered traffic attacks, and the IP is often located abroad. But this time the attack on our server is obviously not advanced. The 5000Mbps traffic attack only comes from two IPs, both located in Guangdong, China. The attack method is single and only UDP flood is used.
Although the attack is simple, it still hits our service into a black hole, causing the game service to be paralyzed in the short term. This taught us a lesson that while gradually operating and expanding the scale of servers, security issues should not be underestimated. How to avoid network attacks such as ddos? How to enhance the security of the server? The project team spent the afternoon communicating and discussing security policies, learning and testing some tools to try to make some security upgrades.
Safe Arming
Through this incident, we took the opportunity to upgrade the security performance of the server. The simplest point is that port 22 of the server is always exposed by default, which has the risk of ssh brute force cracking. In order to enhance security, the ip segment is divided by security group. Filter processing is also done for other ports.
For ddos processing:
- 首先封禁ddos ip，禁止流量再进入。
- 使用iptable划分ip白名单，添加协议白名单保护。
- 使用iptable的 limit 限制恶意流量进入
iptable配置示例：
# vim /etc/sysconfig/iptables
# 先全部禁用
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-N whitelist
#再添加白名单
-A whitelist -s 1.2.3.0/24 -j ACCEPT
-A whitelist -s 4.5.6.7 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j whitelist
#设置端口开放，以下端口不受白名单限制
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j whitelist
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j whitelist
#设置icmp协议不使用白名单，如果需要的话就删掉
-A INPUT -p icmp -j ACCEPT
#其他协议的白名单开放情况
-A INPUT -i lo -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
#防止dos流量攻击
-A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
COMMIT
#配置完毕后，运行命令重启防火墙使规则生效
#systemctl restart iptables.service
复制代码二次攻击
28日11:07，服务器黑洞24小时过后自动解封，自认为已经做好防护的我们如期的开启了服务器，还兴致勃勃的加入了新的插件。万万没想到的是，就在下午3点34分，服务器再次瘫痪，流量无法进出，看到shell的异常连接，不由得倒吸一口冷气。
打开华为云，果然又收到了故障提醒，服务器再次被打进了黑洞，我们的防御手段形同虚设。
经调查溯源，第二次攻击源ip分别来自美国和中国安徽。
反省，我们的防御手段为什么无效？除开购买价格高昂的ddos高防，我们应该如何做防御？
二次布防
由于缺少安全攻防知识，我们自己只会用pentmenu进行压力测试，再结合服务器运营商提供的可视化流量统计查看攻击强度以及防御强度。
测试效果并不理想，根本不能确定防御的有效性。 最后，是在我的催促盘问下，提给华为云的工单有了安全工程师的反馈。
我们了解到： 对于ddos公网源ip的攻击，无法使用acl 安全组拦截，就算流量走不到服务器，ip还是会被打进黑洞。 同时，我们也了解到cloudflare这个服务提供商，可以免费为站点提供cdn支持，免费提供了防dos的能力。
Cloudflare 如何工作？
Cloudflare 可保护和加速任何线上网站。一旦您的网站加入 Cloudflare，其网络流量就会通过我们的智能全球网络进行路由传输。我们会自动优化您的网页传送，以便您的访问者获得最快的页面加载时间和最佳性能。我们还会阻止威胁并限制滥用机器人和爬网程序浪费您的带宽和服务器资源。
回过头来整理一下我们当前的处境：
- 公网源IP已暴露。
- 攻击者ip设置了跳板（肉鸡），不能依靠banip来防御。
- 服务器安全组和iptable防御策略无效，公网ip一旦被打，依然会进黑洞。
最终得出了如下的布防策略：
我认为，整个布防最最重要的一点就是隐藏公网源ip
Arming effect
After the second arming, the ddos attacker can only hit the proxy machine or cloudflare's cluster. Under the condition of multiple agents, even if one is killed, it will not affect the normal operation of the game server. The following figure shows the data graph of one of the agents successfully resisting the DOS attack.