Overview Introduction
Big data platform data service framework. Realize Kafka real-time data filtering, cleaning, conversion, and consumption, and realize Spark SQL to read and write data of non-relational databases such as Redis and MongoDB.
Why recommend this framework?
Data processing service framework based on big data platform.
Combined with the actual use scenarios of big data projects, some general functions are extracted to form a big data platform data processing framework.
The main functions currently implemented are:
1. Parameter information configuration module, which can realize the configuration using the database and Properties file.
2. Integrate Kafka and realize the functions related to Kafka's producers and consumers.
3. Integrate MongoDB and realize MongoDB. Data reading, writing, etc. realize the interaction between SparkSQL and MongoDB data through DataFrame, and realize special reading methods such as paging reading, streaming reading, etc.
4. Integrate Redis to realize Redis reading and writing Into and so on, realize the data interaction between SparkSQL and Redis through DataFrame
5, SparkStreaming streaming Kafka, MongoDB data
6, manually record the offset of Kafka, realize the record based on the database and record based on Zookeeper
Source download
Source download address, just copy and download
https://gitee.com/hy-wux/DataService-Framework.git
Scan QR code to follow
(the public)
(WeChat)
Disclaimer
[ Writing instructions ] The above content is shared with programmers who like to program and have dreams, and hope to help you. The above article belongs to the original public account. If you need to reprint, please indicate the source.
[ Disclaimer ] This public platform is not an advertiser, nor does it advertise for other third-party websites or individuals. The source code and articles shared here are some of the projects that I think are good in the project and study. Some voluntary download or payment behaviors generated by users. Not directly related to the platform
【Submission Email】[email protected]
Contributions are welcome to share your high-quality source code or articles