(1) Project introduction

1. Project background introduction

(1).Basic introduction

With the popularity of smart phones, people are more accustomed to watching news through mobile phones. Due to the accelerated pace of life, many people can only use fragmented time to obtain information. Therefore, the demand for mobile information clients is also increasing. It is against this background that the Dark Horse Toutiao project was developed. The Dark Horse Toutiao project is implemented using the current hot 微服务+大数据技术architecture. This project mainly sets out to obtain the latest and hottest news information 通过大数据分析用户喜好精确推送咨询新闻.

(2). Business Description

Functional Architecture Diagram:
insert image description here

(3). Project Terminology Introduction

Project: generally refers to the entire project or a certain project module of Heima Toutiao

Project: generally refers to the source code project of a certain project of Heima Toutiao

User: generally refers to the user of the Heima Toutiao APP client

Self-media people: generally refers to users who send articles through the dark horse self-media system

Administrator: generally refers to the user of the Heima Toutiao management system

App: generally refers to the dark horse headline APP

WeMedia: generally refers to the dark horse headline self-media system

Admin: generally refers to the dark horse headline management system

2. Technology stack description

(1). Technology stack - basic six-layer technology

The basic six layers include front-end (Weex, Vue, Echarts, WS), gateway (GateWay), DevOps (unit test, code specification) and other important and difficult technologies
insert image description here

Weex+Vue+WebSocket: Use the Weex cross-platform development tool to integrate the integrated VUE framework to complete the dark horse headlinesMobile function development，And integrate WebSocket to realize the push of instant message (article recommendation, private message)

Vue+ Echarts: The key to self-media system development using Vue,Integrate the Echarts chart framework to complete functions such as fan portraits and data analysis

Vue+Echarts+ WebSocket: The management system is also developed using Vue, integrates Echarts, completes website statistics, content statistics and other functions, integrates WebSocket,Realize automatic update of real-time data on the system Kanban

Spring-Cloud-Gateway: The gateway service set up before the microservice to implement the API request routing in the service registration, andControl Flow Control and Fuse HandlingThey are all commonly used architectural methods, and these functions are naturally supported by Gateway

PMD&P3C: 静态代码扫描工具, scan the project code in the project, check abnormal points, optimization points, code specifications, etc.,Provide the development team with unified standards and improve the quality of project code

Junit : In the continuous integration thinking, 单元测试it is biased towards the automation process, and the project passesThe integration of Junit+Maven realizes this process

(2).Technology stack-service four-layer technology

The four layers of service include middleware (Kafka, Mycat), computing (Spark, Neo4j, Hive), indexing, microservices, big data storage and other key and difficult technologies

insert image description here

Use Spring Boot快速开发框架and build project engineering; and combine Spring Cloud family bucket technology to realize micro-services such as back-end personal center, self-media, and management center.

Use WebMagic爬虫技术and improve the automatic collection of system content

Completed Kafka_Internal system message notification; and client system message notification; and real-time data calculation

Use MyCat数据库middleware calculations to separate and sub-table system data to improve system data layer performance

Use Redis缓存technology to realize thermal data calculation, NoSession and other functions,Improve systemicityperformance indicators

Use Zoookeepertechnology to complete the coordination and management after the big data node,Improve the high availability of the system storage layer

Use Mysql to store user data, toGuarantee the high performance of upper layer data query

Use Mongoto store user hot data toGuarantee high expansion and high performance indicators of user hot data

Used FastDFS作as static resource storage, on whichRealize functions such as caching and elimination of hot static resources

Use Habsetechnology to store cold data in the system to ensure the reliability of system data

Use ES搜索技术, index cold data and article data toGuaranteed cold data and article query performance

use Sqoop、Kettletools such asOffline warehousing of big data; or data backup to Hadoop

Use Spark+Hiveoffline data analysis to realize various statistical reports in the system

Use Spark Streaming + Hive+Kafkato realize real-time data analysis and application; for exampleArticle recommendation

Use Neo4jknowledge graph technology to analyze data relationships, produce knowledge results, and apply them to upper-level business to help users, self-media, and improve operational effects/ability.For example, fan level calculation

Use it AI技术to complete the system automation function to improve efficiency and save costs.Such as real-name authentication automation

(3). Technology stack - distribution