Stock futures data data calculation features:
1. Each stock or futures account has a lot of data to calculate without involving other accounts, such as equity, yield, volatility, Sharpe...
2. The number of accounts is generally relatively large, and there is a need for concurrent processing.
3. It is not necessary to return the result to the producer after the consumer is finished running.
And Kafka just meets these needs:
1. When the number of partitions is greater than the consumer data, multiple consumers can be dynamically added to play a role in load balancing
See: consumer group and partition in http://oracle-api.iteye.com/admin/blogs/2374848
2. Although multiple Consumers consume a Group at the same time, the order cannot be guaranteed, but the order of messages in each partition can be guaranteed. When using the producer to send data, the hashcode of the account ID is used as the basis for which partition is allocated, which can ensure the message order of the same fund account.
3. For messages that are not consumed in time, Kafka can play a buffering role.
4. Kafka stores all historical message data and can re-read messages at any time or from a certain location.