One, background
A long time ago to write a DAO framework and MVC framework, some time ago has rewritten the DAO framework -GDAO ( handwriting DAO framework (a) - from "1" , source: https://github.com/shuimutong/gdao.git ) and MVC framework -GMVC ( handwriting MVC framework (a) - re-start , source: https://gitee.com/simpleha/gmvc.git ).
Before also tried to write RPC framework (Source: https://github.com/shuimutong/grpc_beta.git ), the method the client, the server calls were finished, but the network transmission did not write.
In fact, write frame or, write blog Ye Hao, mainly to learn. So now when I start, and I was thinking about what to write well?
My idea is very simple: the current framework used, to realize their own try to implement it again. Standing this point in time, writing framework which has two choices: continue to write RPC; write MQ. Both frameworks personally think there are a lot of similarities: MQ though is asynchronous, there is also a "long connection" in the attached. For example, subscribers, there will be a long connection attached to the MQ server. RPC framework, regardless of subscription, send, but the client, the server, when the client finds the server, the client and the server will try to stay connected.
Compared MQ, RPC framework does not require persistent action, and not only need to consider the MQ persistent, you also need to consider how efficient persistence.
By some comparison, I finally choose to implement MQ framework to learn serialization, network transmission, efficient aspects of persistence. Selected frame of reference is kafka.
Second, how to start
Having chosen to achieve MQ, then where to start it?
Kafka first look at it.
1, mq general process frame
Producers and server establish a connection to send a message to the server.
Subscribers and establish a connection to the server, the server will be subscribing subscribe to the news pushed to the subscriber, or the subscriber himself pulling specified message.
2, look kafka
1) Why is there such a high qps kafka
kafka services can be deployed on mechanical hard disk, each message received will be saved, but you can easily qps over 10,000, even up to one million level (data from the Internet, yet I pro-test). How did you do it?
Can learn through the official website, I have a record of 3:00.
First, disk sequential read and write. Secondly, based on the java, using the external memory heap, gc avoid interference. Again, all data written in duplicate, to prevent loss. In addition, zero-copy technology.
Combine all aspects of the above, the ultimate success of fame kafka!
2) is a push message or a pull message
Push message, and if the message surge may cause the subscriber collapse out, so kafka uses a pull method. How often pull, decided by the subscribers themselves.
In fact, by way of pulling the message also has drawbacks, such as subscribing to pull the message, where it will have been circulating request. If there are no new messages, the subscriber will be frequent requests, wasted resources.
To address this issue circulating air request, kafka added a feature. When the pull message may be provided: if there is no message, the request on the first ram live until the message, and then returns the result. In this way, we can avoid this kind of problem.
3, write Framework Program
After watching a little kafka, I feel this is very high-tech. While the above lists only a few, in fact there are many very important points not mentioned. But I just want to learn to write. Write if I bring kafka see through later, you may have to wait a long time. (In fact, I've taken a long time)
So I decided to take up the first shelf. Nor from the high plains have to play!
1) substantially Program
Write server, network transmission using http, serialization using json (string transmission, in fact, does not matter json), data persistence using DB.
Although the use of http transmission, but needs to be encapsulated into the specific tools to prevent the client requests to write their own tools to facilitate post-upgrade.
2) a message producers
Messages include: theme, message content
3) subscriber pull news
Theme, message content, offset (how many pull)
4) the server process
After the news came, there will be a message DB, ed offset.
Pull message action, according to offset, how many return taken.
The offset shall be submitted to the client encapsulated in, it can be implemented to automatically submit.
5) frame composition
a, server
b, the production of the client message (Service Configuration)
c, consumer messaging clients (Service Configuration)
6) Configuration Item
a, service address
b, topic number
General plan has been set, it would begin to implement it
Third, the server function carding
1, Registration No. topic
Theme No. verify whether there is, then there is no new
2, receiving a message
Parameters: topic number, the message body
First check whether there is a number of topics, there is the saving message to the DB, and returns the result
3, pull news
The number of themes number, offset, want to take a message: Parameters
Check whether there is theme number, calibration offset is normal, normal number of messages of hope (if there is a message) is returned.
4, change the offset
Parameters: topic number, offset
Check whether there is theme number, check offset.
Repeat consumption function post-do.
IV Conclusion
Handwritten MQ, temporarily want it so much, think back on another plus.
To achieve a simple server, the next step is code to achieve.
Write code ......