This article originated from a personal public account : TechFlow , original is not easy, seek attention
Today is the thirteenth part of the distributed topic . In today ’s article, we will not talk about empty theory, let ’s talk about a practical point.
As we all know, Weibo programmers often work overtime from time to time. Unlike other programmers, programmers in other positions may be able to control overtime, but Weibo programmers are not. why? Because programmers can't predict when the stars have new big data, once there is new material, it is appropriate for Weibo to collapse. Many fans even use Weibo to collapse to measure the popularity of a star.
This is of course only a paragraph, but for those of us who are engaged in technology, we ca n’t help but ask a question. Weibo is also a pretty big company with no money or people. Why is the system so unstable?
Of course, I haven't stayed on Weibo. For the sake of rigor, let's assume that we have opened a similar new app, called Wei Tehao. Let's take a look at what problems we will encounter during the growth of the app from 0. Through the false project of Weite, we can somewhat speculate about the reasons behind the instability of Weibo.
The birth of Weite
Weite was born on a certain day in a certain year, in order to simplify the problem, we assume that weite has only two functions, namely , weite and seeweite .
Well, for a small business that has just been born, you can't afford a well-known architect. Of course, how easy it is . Let's break down the functions, and we will find that there are only three functions in total. Users send micro-features, follow other users, and view micro-features. These three functions are very simple, almost naked.
Let's look at the user's micro-blog first. We only need a table to record the content of the micro-blog. This table looks like this:
The second function is to follow users. We can use another table to record the following relationship between users, which is also very simple:
With these two tables, we want to view Weite. It is simple to use the userid to follow the user in the table to associate the user concerned, and then use the id of the user concerned to associate the sent Weite.
We use SQL to express this query relationship.
select wetwi.*
from (
select followeeid
from follower
where followerid = curid
)a join wetwi
on a.followeeid = wetwi.userid
After some testing, Weite successfully went online, and the boss tried it out and felt very satisfied. New users have also risen quickly, and the responsible programmer A has been appreciated by the boss. He has been promoted all the way to become the management , and is no longer directly responsible for the technical problems of the project.
A member of this person, B, was assigned a heavy responsibility and was responsible for the project, but at this time, the problem appeared.
Weite one stage
With the increase in the number of users, the number of users' attention began to increase rapidly, and soon an average user paid attention to dozens of other users. Every time the user refreshes, it will load micro-content published by dozens of people at the same time. In addition, Weite supports pictures. At most, dozens of pictures and a large amount of text need to be loaded at one moment. The server is too late to transmit so much data . Frequent users report that it always times out when refreshing.
B thought about it, this simple, we can be divided into two steps: open source and throttling.
First of all, we need to open source. In addition to increasing the number of machines , we need to rent a CDN and put the pictures uploaded by users on the CDN without storing them in the database, thereby speeding up the data transmission speed.
What about throttling? It's also simple. We use another table to store the last time the user refreshed . In this way, when we query, we only need to query the data after this time point. So we added a table to record the time.
In this way, the system finally stabilized, and the number of users complaining was further reduced. Soon there was an explosive increase in the number of users. Finally, one day, one after another hot events, the system was finally unable to bear it, and it went down one after another.
Why is it down?
Because the database query takes time , it is inherently slow, and after the join operation is added, the system becomes slower. Now because of the need to judge the time of Weite to send for filtering, an additional query is added, and it will naturally be unable to bear after reaching the bottleneck.
B绞尽脑汁，终于想到了办法，就是缓存。他缓存了用户最后一次刷新的时间，这样就可以减少一次数据库的查询。系统稍微变快了一点，但是并没有坚持多久，终于老板忍无可忍开除了B，花大价钱从大公司挖来了一个架构师C。
与此同时，微特的用户量进一步增长，到了第二个阶段。
微特二阶段
C熟悉了环境之后，气得直骂娘，不从架构考虑问题，只会修修补补要不得啊。怎么可以直接用DB存储数据呢，搞成这样不崩溃才有鬼。
新官上任三把火，搞走了几个混吃等死的老白兔，C又招来当年的几个手下，开始大张旗鼓地重构系统。在分布式系统当中，DB往往都是瓶颈，能不用DB的就不应该用DB。而且微特的模式非常明显，用户发布了新内容之后，最关心的就是他的粉丝，那么我们直接在他更新的时候，就把内容推送到粉丝的缓存里，当用户刷新的时候，直接从缓存拉取数据，这不是要方便得多吗？
C画出了新的架构图（可见C也是灵魂画手）：
微特直接通过kafka等消息系统推送给用户，效率要比DB高多了，C的一帮手下也给力，很快就完成了架构升级。上线之后，果然效果非凡，系统立刻变得稳定了许多，用户抱怨大幅减少。
在这几年里，公司声名鹊起，积累了大量的资源，一举去美国上市了。公司的高管都财富自由了，C来得晚没分到多少，并且看到当年埋了大坑的A混的风生水起，加上长久以来的不和，气不过跑路了。
就在这风生水起的时候，意想不到的事情又发生了。而我们的微特也来到了阶段3。
微特三阶段
C走了之后，系统从此再也没有大的改动，大家都只想把眼前修修补补的工作做好。反正公司也上市了，老板也自由了，底层的员工只想安安稳稳赚点工资。加上系统也稳定得很，没有人觉得有什么改动的必要。
但是由于微特现在太火了，使用的用户越来越多，除了普通用户之外，还把明星艺人吸引来了。这些明星艺人，一个个都有好几千万的粉丝。不是之前升级了架构，改成了通过kafka推送消息而不是去DB查询吗？这下好了，这些明星每次一发微特，都需要发送好几千万次消息。
有几天不知道是过节还是碰巧，这些明星艺人偏偏一起发推，直接就把发送消息的系统挤压挂了。这一挂，虽然微特整个系统没有崩溃，但是吃瓜群众发现再也搜索不到微特了。于是一起在网上骂微特，这一骂，需要发送的消息更多，系统更加扛不住，彻底成了死循环，公司的口碑急转直下。
公司也想解决这个问题，奈何之前做了太多修修补补的工作，整个系统变得非常复杂，很多模块耦合在了一起，加上也没有一个很好的方案，一直也没有一个人下得了决心整个重构，反而往上加了更多补丁。
直到后来，A离职了之后，公司新来的CTO招来了一个业内技术骨干D。D提出了新的想法，将前面两种方案作了折中。大V发送的时候，并不推送到所有粉丝，而是只推送给当前在线的粉丝。离线的粉丝还是通过查询来获取数据。这样一来对流量进行了分流，微特系统变得稳定了许多。
但是这并没有真正彻底解决问题，系统仍然有时候会宕机，但是一时半会也没有什么特别好的办法，系统也变得极为臃肿彻底经不起折腾。于是微特就这样修修补补，变成了程序员加班不规律的公司。
后续
所谓的微特当然是个段子，故事也不是真的，但是技术架构的变迁还是很有参考意义。一个看似简单的问题，真正落地到应用场景当中，往往没有那么简单。系统眼前的稳定也不能代表长远，这也是我们在开始一个项目的时候就需要眼光长远做好规划的原因。更是一个优秀的架构师的价值所在，只不过可惜的是，世上的道理总是这样，知道的人太多，做到的太少。
希望今天这个的这个故事在博君一笑的同时也能给大家一点启发，一个成功的优秀的系统背后，究竟藏着多少思考和心血呢？
今天的文章就是这些，如果觉得有所收获，请顺手点个关注或者转发吧，你们的举手之劳对我来说很重要。