Message push architecture design

Said it in front

In the reader community (50+) of Nien, a 40-year-old architect , some friends have recently obtained interview qualifications from first-tier Internet companies such as Alibaba, NetEase, Youzan, Xiyin, Baidu, NetEase, and Didi. They met a Some very important interview questions:

What are the requirements for an enterprise-level message notification system? How to satisfy?

How to design the architecture of an enterprise-level message notification system?

Nien reminded that issues related to system architecture are the core knowledge of architecture and a key problem online.

Therefore, here Nien will give you a systematic and systematic review, so that you can fully demonstrate your strong "technical muscles" and make the interviewer "can't help himself and drool" .

This question and the reference answers are also included in the V103 version of our " Nion Java Interview Guide " for reference by subsequent friends to improve everyone's 3-level architecture, design, and development levels.

For the latest PDFs of "Nien Architecture Notes", "Nien High Concurrency Trilogy" and " Nien Java Interview Guide ", please follow this public account [Technical Freedom Circle] to obtain them. Reply: Get the e-book

Article directory

Architecture goals:

Build an enterprise-level unified basic push service that supports push through multiple channels and can uniformly integrate email, SMS, chat, DingTalk, corporate WeChat and other public social applications:

Chat - Wechat/QQ
In-site push notifications (mobile devices and web browsers)
Off-site push notification (mobile device, APP is not opened)
SMS (such as login password, marketing activities)
e-mail
DingTalk
Enterprise WeChat

Enterprise-level unified basic push service is a universal feature that applies to all modern distributed applications, regardless of programming language and technology.

The evolution of push capabilities

The first stage (modularization): Do it your own way and encapsulate it individually

Within the company, the business volume in the early days was relatively small, and each system basically had its own push module, with various types:

Chat module
SMS module
Email module
websocket module

It is relatively simple to package modules individually, but it is difficult to achieve decentralization and ensure the quality of each system module in a unified manner.

The second stage (framing): integrated framework

In order to reduce repetitive design and development costs, a unified push framework was designed

The same set of microservice frameworks share a unified push framework

In order to solve the above-mentioned problems of decentralized implementation, the enterprise has unified and implemented a basic library that integrates various push functions for business parties to call uniformly.

Chat basics starter
SMS basic starter
Email basics starter
websocket basic starter

Therefore, we encapsulated the logic of springboot-starter into the service governance framework . When the microservice service is started, each service performs operation and maintenance management and configuration management of various starters.

The third stage (servitization): push service

Integrated into the framework, each set of services needs to solve three high-level problems repeatedly.

Push service has a large amount of data and needs to solve the problem of cross-database query
Push service has high performance requirements and needs to solve high concurrency issues

Large data volume and high concurrency mean:

Heavy investment in hardware resources
High operation and maintenance costs

Such basic services need to be precipitated, stripped, and concentrated into unified, basic services, with a dedicated team responsible for maintenance, iteration, and operation and maintenance. Reduce repeated investment and repeated construction costs, truly reducing costs and increasing efficiency.

As a result, the push framework evolved into push service

The location of push services in business systems

A business application basically has many atomic services orchestrated and integrated, and finally builds a complete architecture diagram.

Access layer , this is the portal for external requests to enter the internal system, and all requests must pass through the API gateway.
The application layer , also known as the aggregation layer, provides aggregation interfaces for related businesses and calls middle-end services for combination.
Atomic services include atomic technical services and atomic business services, which provide relevant interfaces according to business needs. Atomic services provide reusable capabilities for the entire architecture.

For example, on the video website platform of Bilibili, the comment service is an atomic service that is needed for videos, articles, and communities at Bilibili. In order to improve reusability, the comment service can be an independent atomic service and cannot be used with specific needs. Tightly coupled.

In this case, the comment service needs to provide a reusability capability that can adapt to different scenarios.

NOTE: Please click on the image to see a clear architectural diagram!

Similarly, functions such as file storage, data storage, push services, and authentication services will be precipitated into atomic services. Business developers can quickly build business applications by orchestrating, configuring, and combining them based on atomic services.

Of course, this article only focuses on push services. For other atomic services, Nien, a 40-year-old architect, will introduce them in other articles. For details, please continue to pay attention to Nien’s public account, Technology Free Circle.

Push service functional requirements:

send notification
Prioritize notifications
Send notifications based on customers' saved preferences
Supports single/simple notification messages and batch notification messages
Analysis use cases for various notifications
Notification message reporting

Push non-functional requirements (NFR):

High performance : qps > 1W
High Availability (HA): 99.99%
Low latency : TP99 below 10ms
Highly scalable : Extensible/pluggable design to add more adapters and providers, API integration with all notification modules and external integration with clients and service providers/vendors
Cross-platform : supports Android/iOS mobile devices and desktop/laptop web browsers
Self-scaling : scales workloads on-premises (VMware Tanzu) and on public cloud services such as AWS, GCP, or Azure

Push system design architecture:

NOTE: Please click on the image to see a clear architectural diagram!

Considerations and components in the design of these solutions include:

1. Notify the client:

These clients request single and batch messages through API calls. They will send notification messages to the simple and bulk notification services.

Simple notification client : A client dedicated to sending a single notification, responsible for sending a single notification to the user. These clients are often used to send important notifications to specific users, such as password retrieval or account anomaly reminders.
Batch notification client : A client specifically used to send batch notifications, responsible for batch push notifications to users. These clients are typically used in scenarios where a large number of users need to be notified, such as internal corporate notifications or marketing campaigns.

2. Notification service:

These services serve as entry points to interact with clients by exposing REST APIs.

They are responsible for constructing notification messages by calling "template services". These messages will be verified using a "validation service".

Simple notification service : This service will provide an API, mainly responsible for processing simple notification requests, and provide an API integrated with back-end services to send notifications to users. This service is typically used to handle fewer notification requests, such as simple notifications for a specific user or event.
Batch notification service : This service will provide an API, mainly responsible for processing batch notification requests, and provide an API integrated with back-end services to send notifications in batches. This service is typically used to handle large volumes of notification requests, such as batch notifications within an enterprise or batch push for marketing campaigns.

This service will also manage notification messages. It persists sent messages to a database and maintains an activity log.

The same message can be resent using the APIs of these services.

It will provide API to add/update/delete and view old and new messages.

It will also provide a web dashboard which should have filtering options to filter messages based on different criteria like date range, priority, module user, user group, etc.

3. Template service:

This service is primarily responsible for template management of all available one-time passwords (OTPs), SMS, email, chat, and other push notification messages.

It also provides REST API to create, update, delete and manage templates.

In addition to this, it will also provide a user interface (UI) dashboard page that will enable users to inspect and manage various message templates from the web console.

4. Message distribution service

Scheduled distribution service:

The service will provide APIs to schedule notifications immediately or at a specified time. Can be any of the following:

Second
minute
per hour
every day
weekly
per month
Per year
Custom frequency, etc.

There may also be other automatically triggered services that trigger messages based on scheduled times.

Message verification service:

This service is solely responsible for verifying notification information against business regulations and expected formats. Bulk notifications require consent from an authorized system administrator.

Message priority service:

This service is responsible for prioritizing notifications into three levels: high, medium, and low.

Notification messages have a higher priority and a time-limited expiration, and they will always be sent with a higher priority.

The "Universal Egress Processor" will receive messages and send and process them from three different queues, high, medium and low, according to the same priority.

During non-working hours, batch notifications can be sent with low priority.

In-app notifications during transactions can be sent to medium priority, such as email, etc. Businesses can prioritize notifications based on their importance.

5. Event priority queue (message queue):

This service provides the event center function and is responsible for receiving high, medium and low priority information of the notification service.

It sends and receives notifications based on business priorities. Businesses can prioritize notifications based on their importance.

The service contains three topics internally for receiving and sending notifications based on business priorities:

Low priority : Mainly used to send batch notifications during non-working hours.
Medium priority : suitable for application notifications sent during transactions, such as emails, etc.
High priority : Notification messages have a higher priority and a time-limited expiration, they will always be sent with a higher priority.

6. Universal outbound handler:

The service receives notification information in the event center by polling the event priority queue and processes it according to its priority.

High-priority notifications will be processed first in the "high" queue, and so on.

Finally, it sends the notification information to the specific adapter through the event hub.

Additionally, the service obtains target users/applications from the user selection service for distribution of notifications.

During the processing process, the general exit processor will perform corresponding operations according to the priority of the event to ensure that important events are processed first.

In this way, enterprises can determine the processing sequence based on the priority of notifications, thereby improving notification processing efficiency.

In addition, the universal outbound handler can further distribute messages according to channel type:

This service sends messages to various supported adapters.

These adapters convert based on different devices (like desktop/mobile) and notification types (like SMS/OTP/email/chat/push notifications).

7. Notify the adapter:

These converters will receive incoming information from the message queue (rocketmq) and pass it to external partners according to the format they support.

Here are some converters, more can be added as needed:

QQ Notification Adapter Service
WeChat chat notification adapter service
In-app notification adapter service
Email Adapter Service
SMS Adapter Service
OTP adapter service

8. Channel Provider:

These are external SAAS (on cloud/on-premises) service providers, leveraging their infrastructure and technology for the actual notification delivery.

They may be paid push channel services like AWS SNS, MailChimp, etc.

QQ Supplier Integration Service
Wechat supplier integration service
App Push Notification Vendor Integration Service
Email provider integration services
SMS provider integration services

9. User selection of services:

The service offers the ability to select target users and various application modules.

This might include sending bulk messages to specific user groups or different application modules.

May be AD/IAM/eDirectory/User Database/User Groups, depending on customer preference.

Inside the service, it will use the "User Profile Service" API to consume and check the customer's notification preferences.

10. User Profile Service:

This service provides various functionality, including the management of user profiles and their preferences.

It also manages the relationship between internal user IDs and external channel IDs.

The relationship between DingTalk user ID and user ID
Enterprise WeChat user ID and user ID association relationship
The relationship between users and mailboxes
etc.

It will also provide features such as unsubscribing from notifications and how often you receive notifications.

The "Notification Service" will rely on this service in order to send notifications based on the user's notification preferences.

In addition, this service can also be used to collect statistics and analyze users' preferences for notifications to help enterprises optimize notification strategies.

11. Analysis Services

The processor will be responsible for performing all analytics, identifying notification usage, trends and generating reports.

It will extract all final notification information from the analytics database (Cassandra) and notification database for analysis and reporting purposes.

Here are some use cases:

Total number of notifications per day/second
Which notification system is used most frequently?
Average size and frequency of messages
Filter messages based on priority and more…

12. Notification Tracker

This service will continuously monitor the event hub queue and keep track of all notifications being sent.

It captures notification metadata such as transmission time, delivery status, communication channel, message type, etc.

13. Notify database: Mysql database cluster

Notification database is used to store all notification information, including sending time, status, etc.

It consists of a database cluster where the leader is used to perform all write operations and read operations are performed on read replicas/followers.

This database cluster will persist all notifications for analysis and reporting.

It is based on the philosophy of "write more, read less".

It provides good performance and low latency, adapts to a large number of notifications, as it handles a large number of write operations internally, and synchronizes with other database nodes to maintain high availability and reliability of redundant data/messages.

In case of any node crash, messages will always be available.

Of course, it can also be a NoSQL cluster database.

Say it at the end

Interview questions related to basic technologies such as push services are very common interview questions.

If everyone can answer the above content fluently and thoroughly, the interviewer will basically be shocked and attracted by you.

In the end, the interviewer loved it so much that he "can't help himself and his mouth watered" . The offer is coming.

During the learning process, if you have any questions, you can come and talk to Nien, a 40-year-old architect.