Preface
birth of artificial intelligence dating back to the 1950s, the Dartmouth Conference, McCarthy proposed the concept of AI, but after the heat of the early development of artificial intelligence gone through several valleys, from the 1990s until nearly two decades since the start of the end of this time, the AI really ushered in the golden age. Especially in the last 10 years, various factors have contributed to its growing: In theory, machine learning, especially statistical learning theory and neural networks continue to make breakthroughs, the effect is significant; the external environment, advances in hardware and software technology for artificial intelligence model implementation provide enough computing power; in addition, a very important factor is in terms of data, the development of big data technology to make artificial intelligence finally get rid of the shackles of the data, the model can enhance the ability of the sample on the basis of sufficient. We can say that now in all areas of research and development of intelligent model the vast majority without the support of big data technology.
Conversely, artificial intelligence for Big Data technology also has a very important role. On the one hand, the use of data for large data collection technology to the need to find the value of which is through some intelligent analysis process; on the other hand, through the intelligent analysis of available data, we can derive more data features even further guidance data production direction. So today we talk about the use of big data, inevitably involves the concept of artificial intelligence and machine learning.
Agility Big Data stack as a platform for real-time data infrastructure platform, is the outcome of the further development of the theory of Big Data and technology, will naturally have research and intelligence aspects of the layout. Agility Big Data is the main goal of intelligent, agile big data combined with the implementation of the concept, the development of flexible, intelligent lightweight model, and the data stream in real-time intelligent processing on large data platform agility, and ultimately stop large intelligent data analysis practice.
To achieve these goals, our artificial intelligence, machine learning, real-time computing and other technologies, and related business knowledge, as well as product user experience is conducted in-depth research and analysis, this series will be our philosophy and in the above process obtained some experience to share with you the results.
Intelligent real-time data processing
Prior to the public as the number of series of articles, with the development of technology, we were able to get an unprecedented mass data, if we can quickly and efficiently process the data, found that high-value information which will undoubtedly greatly enhance enterprise resilience, in order to make strategic as well as tactical adjustments quickly in the complex and volatile business scenario. Therefore, real-time data processing technology has become the main direction of development of big data. Real-time data processing is bound to impact on intelligence analysis model closely associated with the data, it can be said, in order to quickly identify, adapt to changes in the external environment, organizations have begun to combine real-time data processing capabilities and the ability to AI, intelligent rapid delivery of data analysis services.
In fact, for the intelligent processing real-time data streams has been a priori in many industries. For example, in the field of Internet live, real-time filters based on video streaming, real-time effects algorithms have been widely used in many APP deft, vibrato, etc., and foreign Twitch and other live website, also launched a real-time game data analysis and other AI plug-ins to enhance live effect; in the field of sports data, based on real-time scores and team scores and player statistics analysis and trend forecasting are various sports data provider, such as Opta Sports, etc., has been applied; in the transport sector, based on real-time traffic information traffic congestion prediction system has also been implemented. Examples of such numerous, but they reflect the real-time data processing has been AI in different fields, different business scenarios widely used and play an irreplaceable role.
Many scenes in the financial sector, the data processing for real-time AI also exist have many needs, such as real-time risk control, real-time data to predict, in real-time anomaly detection, real-time user analysis, etc. The following figure shows the real-time product recommendations of a data flow diagram can be used for financial products recommended in the scene, such as net loan, insurance, funds, stocks and other products.
The figure depicts the following process: We can get a lot of different user behavior data point buried in the interactive side, the data will be enterprise platform for real-time data acquisition, and users together to provide products and other types of data to calculate layer models, such as user interest model, portraits product models. These models describe the characteristics of users and products, and ultimately to the recommending model calculations, sorting, filtering, to obtain a final list of recommendations. This process we can stream real-time according to the collected user behavior data to user interest model is updated and correct in order to achieve real-time tracking of the user is interested in the content. The figure does not reflect a process of real-time updates on product model portrait, although relatively user behavior data, the characteristic data products is relatively stable, but in practice there are many products on the high requirements of timeliness, its portrait we also need to feature real-time maintenance, such as stock market data and other information. These product streams from other sources can be aggregated into the enterprise in real-time data platform and provide to the product model is reconstructed portrait of product characteristics, and ultimately to the recommendation model product recommendations. A good real-time product recommendation system may be sensitive to capture the user's needs and responding to changes in the product, the user can efficiently carry out personalized for precision marketing, improve the user experience, while also able to increase the number of one-off and acquisition (CPA), a huge business value.
In the figure above enterprise real-time data platform played a key task of providing real-time data for the recommended model. In an agile data environment, agile big data platform will be well supported above work, an implementation architecture as shown below:
In the figure, can be easily butted wormhole dbus and a plurality of different data sources, real-time data acquisition, the source of real-time data pipeline. In addition wormhole support processing on stream, it is suitable for access products portrait model and user interest model of the product with the characteristics of the user in real-time characterization of these characteristics after storage decimated by the moonbox necessary, enter the recommended list of recommended models get the required final back to the interactive side. In addition, if coupled with support davinci BI data, we can easily achieve real-time monitoring of business indicators, for us to assess the effect of the recommendation. The whole process flexible and easy to integrate a variety of different open-source platform to quickly build real-time data applications, open-source selection can switch at any time as needed to support rapid iterative trial and error, combined with the existing algorithm model we are able to quickly support the achievement of intelligent consumer products this real-time recommendations scene.
Agility AI
As previously described, real-time data processing AI, based on the business component of the large data agility combined with third party member open, via a simple configuration to quickly organize, quickly achieve the underlying algorithm running support infrastructure. This makes the whole system seems the only trouble is that we have to advance the development of all kinds of good intelligence model, which for some business organizations still have a certain technical threshold; In addition, for certain businesses, the rapid advance and cost control is the primary consideration of the factors, then the targeted development of customized intelligent algorithm model, and adjust the call interface makes it possible to access data in real-time architecture, it is clumsy. For example, many data analysis business, may not need to be too precise model performance, it is preferred to ensure promptness convenience, business logic implemented embodiment of the analysis system. We have let the data processing become agile, intelligent data then how will it become more agile? To solve this problem, we propose the implementation of quick thinking AI, that is the basis of existing data products on a large agility, based on the business scene pluggable design and development of a series of real-time intelligence model operator, which covers the business model common intelligent data analysis requirements within the scene, with strong versatility and reusability, seamless access to real-time data stream data over the internet to the internet agile large outputs the analysis result, the real-time traffic flows into each end if necessary, ultimately based on intelligent analysis process real-time data streams. With the support of big data products and agile agile AI, the business can quickly build real-time data analysis to real-time data processing platform based on business intelligence scenarios, and then the entire intelligence data governance processes real-time data display, and can be flexibly adjusted according to results trial and error, which greatly reduces the real-time business intelligence analysis of implementation costs.
In the implementation of the above-mentioned idea of agile AI, we start building agile AI algorithm library, which is a lightweight set of common data model based on business division. Design in which each model should follow the following principles:
• Lightweight, to the complexity of the model of real-time data processing to ensure appropriate control;
• independence, minimizing deployment environment dependence or independence to ensure that the environment, to avoid the model introduced to the system as a whole depend on the environmental changes;
• unity, each model features a single try to ensure the parallelism of each model function;
• universal data, unless there is some essential features of the model, the model should ensure that each universal adaptability to access data through a certain configuration or mapping which can be adapted to most business scenarios.
In order to achieve the above requirements, we will inevitably make some trade-offs in some areas in the development of the model, for example, if you want universal model will certainly cause a degree of decline in performance, how to find a reasonable compromise in these contradictions, issues are also to be considered in the design. Currently, we have begun to develop agile AI model for a number of areas, after the actual test and application, it will be integrated into the current agile big data product stack in the near future. In addition, in the future we can also publish the relevant interfaces and protocols, users also have the ability to make their own models added to the library.
Conclusion
intelligent real-time data analysis is an important direction for future development of artificial intelligence technology and big data, how to reduce the economic cost of this implementation process, time costs, technology costs and the cost of change, is agile and nimble AI focus on big data resolve key issues. In this paper, agile big data product presents a solution ideas, hope that our products can help organizations to easily, quickly, build their own big data in real-time intelligent analysis system flexibility. (Source: CreditEase agile big data team)
