1, the application level
The introduction of cache
Increase the cache layer is the application layer and the database tier, hotspot data into the cache. The systems commonly used switches, white list data, writes the read high frequency low frequencies, can be stored in a JimDB (Redis) for this part of the data, JimDB (Redis) will the high frequency data stored in memory , high read and write performance. After setting a period when the write cache data, update the database successfully, asynchronous cache update data. If less demanding real-time, etc. can also cache invalidation after, take the initiative to update the cache. Introducing layer cache, the database reduced pressure and improve the system response.
Writing concurrent processing program
Multi-task concurrent processing, full use of CPU resources. No dependencies of multiple tasks can be processed in parallel, increase system capacity. Tasks such as clearing and settlement operations between each order no dependencies, you can perform multiple tasks at the same time settlement
System Optimization
The core generation process asynchronous processing, the user orders and receives a user to recharge two asynchronous processing flow, improve the system capacity. For the user, the user payment is successful, you can wait for recharge. The system can be triggered by a worker refill operation, provided a reasonable number of retries, for a certain time interval to retry. Before reaching the final state, the intermediate state displayed to the user.
Dismantling large applications, so micro as a service. The prepaid recharge applications, the core function termination single PC, a single mobile terminated, the MQ messaging terminal, terminal management background, and other five large end worker. Before dismantling, the application calls the common only service \ manager \ dao module code between the three systems, system management constructed by Maven. Needs a little background to achieve, we need to consider the PC side, the mobile terminal, back office management-side code will be affected, whether you need all the regression testing. More terror is on the line, nearly 100 top application example, modify the line of code requires all on the line, even if the system is not affected because the new version should be released online and to ensure consistent code base code.
After the micro of service from the function, application dismantling the PC side, the function between end Server, MQ message processing end, back end management, worker end of the five applications, application independent, dependent on the company's RPC framework (JSF) and message frame (the JMQ) communication. Higher cohesive single application, a lower degree of coupling between the applications. When a background of further implementation requirements, development, testing, deployment only need to focus on back-end system to manage, no longer need to focus on the other four systems.
Micro-service system design, the key point is to find how to gauge. In addition to be segmented from the function can also be disassembled for the greater cohesive end and a support end of production, different business scenarios, the method is not delimited to find the same service key after a single application according to the micro focus on, a lower degree of coupling between the applications. The large system of micro-services program there are many, the focus is to develop a good goal, and gradually move closer to the goal. In terms of granularity services, MQ messaging terminal such as prepaid recharge process of the application, the function held in a single responsibility, only responsible for receiving, parsing MQ message content, particularly forwarded to the appropriate service logic processing Server side processing. On the technology options, companies have mature technology framework, such as RPC framework JSF, messaging framework JMQ, these frameworks has a corresponding management and monitoring services and other related services and team.
Not to micro-services and the implementation of large-scale systems of micro-services, to ensure that after the service of the micro, the system more stable, more rapid response to change, develop more agile.
Separate read and write
Less demanding real-time data read from the database, the master database reduced pressure. Such as reconciliation function, the order data is read previous day, there is no need to read data from the main library. On the technology, Spring Framework itself to provide, to achieve its AbstractRoutingDataSource abstract class.
Low frequency static pages change
Recharge card applications have a lot of pages, such as QQ cards page, change the data on the page only advertising. This type of pages can be static, updated page, pushed to the storage medium, configured LOCATION Nginx, directly read the page, reducing the pressure of the back-end service.
2, the database level
When the volume of business to a certain extent, the database will become a bottleneck in the system. Calls recharge business applications include corporate orders and orders for general business users, because of the special nature of its business, using a vertical + horizontal banking scheme. According to the service type vertical segmentation, different types of traffic independently order data storage, database storage of a plurality of the same type on a service level. Vertical + horizontal banking scheme to maximize the reduced interaction between different types of service order data, reading and writing of data to improve concurrency. Ordinary users order business, according to an account PIN can be hash scattered evenly distributed to each bank, sharding rule is that hash (pin) value, while the hash (pin) value is also used as a prefix number of local order, so that you can by two dimensions account PIN number and local order in any dimension can be routed to the database. After creating a successful ERP order to save the mapping between local ERP order number and order to JmiDB, the only ERP for order number of business processes, order number can be found through local mapping relationship with the local order number will be routed to the database a. Business and corporate orders, orders each enterprise account uneven difference can reach three orders of magnitude, if we were to break up the order hash distributed to each stacks according to the account PIN will be uneven, you can not use this rule sharding . According to the local hash the order number, and then as a local prefix order number. After creating a successful ERP order, also need to save the mapping between local ERP order number and order number to JmiDB in order to ensure the follow-up business processes, according to the order number can be routed to the database ERP.
After the completion of the split, some business scenarios that need to aggregate query data, such as order management. If there is no aggregate data, it is necessary in the application, developers consider self polymerization. General polymerization solution is a query data from each bank in memory sorted according to the conditions, to return a data page, then if necessary, more complex logic. Third-party application add credit memory, the data in each sub-library of polymerization in order to elasticsearch, the query scene aggregated data read ElasticSearch. MySQL slave interactive simulation protocol, data BinLog incremental parsing database synchronization sub-library to ElasticSearch in. Because of the risk of delay from the master database exists synchronization, we need a downgrade program. After prepaid recharge applications, database write orders successfully, insert a task record, synchronize data immediately through the task model to ElasticSearch in. Ensure the synchronization of real-time data.
3, application deployment
CPU, thread, IO and other computer resources are precious and there is an upper limit, when a certain resource depletion, and that on this computer all services will stop the service. Third-party services such as low performance of certain services dependent, slow to respond, then continue if the client's request, will lead the service continued to create threads and other resources, and ultimately lead to service downtime. In this case, the isolation of computer resources is very important.
Inside the JVM semaphore isolation and isolation into the thread pool isolation, Netflix Hystrix plug-in provides the perfect support. JD-Peer (multi-room public network egress routing system) is used for each business Hystrix are isolated. Prepaid recharge docking dozens of business applications, interact through JD-Peer systems with the business. For some reason one of your business network A slow response, continued to call, all the resources that will be occupied by a business, leading to other service businesses are not available, the final shoot down.
Independent deployment, physical isolation. Each application allocates a separate container, resource isolation from the hardware level.
Multi-room deployment, from the entrance to share traffic and improve overall system throughput.
Release
灰度发布，平滑过渡，异常情况下的版本回滚，要确保回滚前的数据在老版本中可用。如京东话费充值系统在发布数据库架构升级版本时，设置了数据流向开关，并对订单打标，同时缓存标识位。 开关打开时，数据进入新的数据库，开关关闭，数据进入老的数据库。线上验证阶段，一旦发现问题，可立即关闭开关。确保系统版本发布，对用户无感知。
在保证系统服务正常可用的情况下，进行上述一系列的升级，犹如给空中的战斗机更换引擎，稍不留神就会坠机，所以除了充分的理论储备，还需要综合业务场景，从自身业务场景出发，合理设置引擎更换方案。
经过上面一系列的改造升级，话费充值应用的吞吐量、运行稳定性都达到了最优的状态，历经数次的618和双11冲击，各项运行指标保持稳定，面对流量洪峰，岿然不动。