Large-scale enterprise-level cloud products - data statistical analysis system (offline processing - stream processing - batch processing)
course viewing address: http://www.xuetuwuyou.com/course/249
The course is from Xuetu Wuyou.com: http://www .xuetuwuyou.com
lecturer: Youfan
Course Introduction
This set of courses is a real case of big data. It is suitable for students who have a foundation in big data to learn. If they master this set of big data cases, they are fully qualified for enterprise-level big data development. I wish all the big data students who have transformed to get their ideal income.
Course Description
This course will be explained through a real enterprise-level cloud product project, which is a professional mobile application statistical analysis platform in China, reaching 1.4 billion active devices every day, covering more than 80% of new mobile phone consumers every month, covering almost all iOS consumers, through this project system, help mobile application developers to count and analyze traffic sources, content usage, user attributes and behavior data, so that developers can use the data to make decisions on products, operations, and promotion strategies. Provide basic statistics, active users, frequency of use, duration of use, page access, regional analysis, version analysis, channel analysis, device analysis, operating system, resolution, operator, networking method, custom event analysis, terminal anomaly analysis, churn User analysis and other statistical analysis methods.
The development environment and technologies used in the course:
system: window7,
development tools: eclipse, IDEA,
this course project is a comprehensive project, the technology covers java web, big data, virtualization, linux server, etc.
Specifically including: spring , spark, spark streaming, spark mlib, hive, flume, kafka, hadoop, hbase, mongodb, dubbo, distributed cache, redis, docker, nginx, easyui, highcharts and more.
This course is explained according to the real enterprise-level development project process. By studying this course, you can experience the real large-scale big data project development process. After completing this course, you can master big data technology, java web technology, docker virtualization technology, distribution mode technology, caching technology, linux, etc.
(1) Analysis of the overall needs of the
project 1. Project background
In this era of the explosion of the Internet, mobile networks and mobile devices have gradually become a must-have for people. The number of users of mobile devices has reached hundreds of millions. It is conceivable that apps will become popular, but every app must be operated to achieve The purpose of profit, then how to do this operation? To accurately analyze user behavior with the help of big data technology, there will be a great demand.
2. Project requirements
In this era of the explosion of the Internet, mobile networks and mobile devices have gradually become a must-have for people. The number of users of mobile devices has reached hundreds of millions. It is conceivable that apps will become popular, but every app must be operated to achieve For the purpose of making profits, how to do this operation? It is necessary to use big data technology to accurately analyze user behavior, which will definitely have a great demand. There is an urgent need for a system to help large and medium-sized enterprises to quickly analyze the behavior of app users. Just by accessing the provided SDK, you can easily understand the behavior of users and enjoy the era changes brought about by big data technology. Through this product, you can learn: app application trends, app channel promotion, user retention, user behavior analysis, user attribute analysis, application error analysis, user data mining, and need to display these analysis results in real time.
3. System function
You can view the user activity of the app, the new users in each period, the terminal usage classification of the app, the statistics and viewing of silent users, and loyal users.
(2) Project architecture design and technology selection
1. Overall architecture design of the project
2. Software selection
(3) Construction of the overall environment of the project
1. Machine selection, node planning, etc.
2. Cluster environment construction
(4) Contents of related projects
1. Physical Architecture, Logical Architecture
2. Design and development of reporting data services
3. Environment construction and program development for real-time data processing
4. Design of log collection system
5. Design and development of offline tasks
6. High concurrency, caching, virtualization, etc.