Some considerations based on Doris real-time data development

3 million words! The most complete big data learning interview community on the whole network is waiting for you!

The recent development of Doris is obvious to all. New features such as hot and cold separation continue to be added. This makes Doris greatly improved in terms of ease of use and cost.

Some storage real-time data warehouses based on Doris have begun to have some practice in more and more scenarios. You have also seen that this kind of scheme frequently appears in community sharing. But we have to look at this solution objectively. The storage-based real-time data warehouse has its advantages and disadvantages. In the production environment, we must carefully evaluate our personal business scenarios. In this article, I will briefly talk about this issue based on my personal practice and thinking. .

Why is there such a scheme?

In many cases, the real-time computing business based on OLAP such as Doris is based on the following considerations.

In more cases, the difficulty of real-time data development based on Flink is significantly higher than that of offline tasks (the two are not in the same order of magnitude). The development of real-time data storage based on Doris can significantly reduce the development threshold, but there is a possibility of abuse.

Secondly, Flink is not good at large windows, large states, and flexible computing scenarios (note that it is not good at this, not impossible). High, but Doris can significantly reduce this.

Finally, the observability of Flink-based computing data is poor. For example, the state data is invisible. There are significant thresholds for troubleshooting and debugging, and it is very difficult to repair historical data.

So you can see that there are not small thresholds for the above-mentioned real-time data development based on Flink. So we have a qualitative conclusion, below the scale of 100 million (or tens of millions) of data, we can use an analysis engine like Doris to perform layering and timing scheduling like offline data, and process large window data (general time span More than 30 days), under the premise of ensuring performance, the development cost of real-time data is reduced, and the observability of data is greatly improved, and the efficiency of development and operation and maintenance is also improved to a certain extent.

Compared with some Flink-based solutions

Low threshold, simple development

Anyone can develop such tasks;

Simple operation and maintenance

Because it does not consider state compatibility like Flink, it does not require a large amount of resources to be occupied for a long time. Only need to schedule resources when running SQL;

Improve development efficiency

There is no need to have a deep understanding of Flink (of course this is not a good thing), there are almost no parameter bars, the test is simple, and there is no need to start the scheduling container (such as the scheduling of TaskManager and Task);

Data debugging is convenient, and the intermediate results can be seen on the ground

There is no state data of Flink, all data is available in the table.

The above points are some advantages, but this solution based on Doris also has obvious shortcomings, which require special attention!

Obvious delay

If you use Doris, then we most likely cooperate with scheduled scheduling. Generally, the scheduling cycle is above 30 seconds, which means that the real-time performance of data is greatly reduced, and some real-time observation indicators such as real-time GMV, online number and other scenarios are not applicable;

data size limit

If you adopt Doris, it means that your TPS cannot be too high. This is not the field that Doris is good at, and everyone needs to pay special attention. In addition, the data of a single scan cannot be too large. As we said earlier, the performance guarantee is better only when the data size is below 100 million (or tens of millions).

Finally, if you really choose Doris-based real-time data development, it means that Doris will become your cost and operation and maintenance center. There must be very strict supporting tools, such as alarm, task operation monitoring, task standardization, scheduling and blood relationship capabilities. Pay special attention to resource and SQL performance issues. Once they become bottlenecks, they will affect all Doris-based tasks.

If this article is helpful to you, don't forget to "Like", "Like", and "Favorite" three times!