Technology Architecture and Development Mode in Cloud Computing Era
In the DOS era, the Windows era, the Web era, and the mobile App era, great changes have taken place in our development language, technical architecture, debugging method, deployment mode, and operation and maintenance mode. So let's re-imagine what the technical architecture and development model of the cloud computing era should look like?
1. Start with an ordinary program
Our routine writing programs are generally composed like this:
1. Microsoft world: ASP.NET, VB.NET/C#, SQLSERVER
2、JAVA世界：JSP/Servlet、Struts/Spring/Mybatis、Oracle/MySQL
3. PHP world: frameworks like ThinkPHP, MySQL
Often we need to add something to the web front-end, such as AngularJS and BootStrap, to make the front-end code more structured and adaptable to more terminal devices.
To run these programs, we usually need Apache/Nginx at the front end, Tomcat/JBOSS/WebSphere/WebLogic in the middle, and .NET running environment/JAVA running environment/PHP running environment at the bottom.
OK, ready to deploy.
In the past, many cloud computing vendors engaged in the AE (Application Engine) environment, expecting the developers to deploy the application software as soon as possible. However, it is found that application developers have a lot of combinations of operating environments and version requirements, which is thankless, so only a pure virtual machine environment is provided. For various component combinations and version combinations, third-party AE The developer of the environment package provides it. You can buy it in the software service market, and it is very convenient for one-click installation and deployment. For start-ups that do not have any special performance requirements and decentralized deployment requirements, it is generally enough.
Second, talk about the support that the cloud computing IaaS product line can provide
As cloud computing, let's first specifically refer to IaaS, which is roughly these product lines:
1. Cloud host: virtual machine, Docker container.
Everyone understands that a virtual machine is no different from a bare metal when it is turned on. But Docker needs to apply the Image image to the Docker container to run, which is not a concept for many people. But Docker containers are lighter, so they start quickly, suspend quickly, and occupy less resources. Unlike a virtual machine, you also have to install an operating system and configure the operating system.
Of course, in addition to virtual machines and Docker containers, cloud computing vendors also provide physical servers for higher performance requirements.
2. Cloud network: CDN, SLB.
CDN allows static files to be buffered and distributed to the whole country, which is closest to the node accessed by users, so that the access speed is the fastest.
SLB is network load balancing, which allows your application to be distributed to each host as soon as it comes in, sharing the computing pressure.
There are also cloud networks: VPN (Virtual Private Network) is convenient for the internal wide area network of enterprise groups, high-speed transmission channels can be used to transmit large quantities of data, and private networks are used for high-performance and high-security large customers.
3. Cloud storage: file storage, object storage, block storage.
These three can store various files (document/picture/video/audio), but file storage can establish a file directory structure; object storage is flat, it has KV index, you only need to pay attention to where a file is placed It can be stored on the disk in the storage array. If you do not need to establish a file directory structure, you can use object storage; if you have high requirements for storage stability, you can use block storage, which is what we usually call RAID storage, block storage. It will ensure multiple copies of data automatically and redundantly at the bottom layer, and does not require the user's Care, so there is a problem with a certain physical disk, and the data will not be lost. You can choose different cloud storage products according to your needs
4. Cloud database: relational database, distributed relational database, NoSQL.
Relational databases, common ones are: SQLSERVER, ORACLE, MySQL, PostgreSQL.
Now there are some relational database middleware, which can do Sharding's automatic sub-database sub-table, without the need for application developer Care, such as MyCAT. But this is not a true distributed relational database. A true distributed relational database is natively distributed when the underlying architecture is designed.
Distributed relational databases, such as Greenplum, TiDB (Chinese works / old employees of JD.com), Ali also launched an Oceanbase.
NoSQL, there are KV-type in-memory Redis, memcache, and document-type mongoDB.
Cloud database service providers will provide orchestrated, automated, and autonomous database backup services and migration services at the back end, making data access more stable and guaranteed.
It can be said that IaaS provides you with servers, storage, networks, and databases, and you can put your own programs on it.
Third, talk about the scalable technical architecture
In fact, there are very general scalable technical architecture methods in this world, but there are too many shit developers in China. They write the codes of all subsystems together and cannot separate them. The front-end and back-end codes are mixed together, and the code and database are mixed. They cannot be separated from each other, which leads to the inability to disperse the access pressure and computing pressure. The only option is to continuously increase memory, upgrade CPU, upgrade network bandwidth, and upgrade the performance of storage devices.
The general scalable technology architecture approach is as follows:
1. The code of each subsystem module is separated and can be deployed independently.
2. It is necessary to develop an enterprise portal to integrate access as each independently deployed subsystem, which is also convenient for load balancing and routing
3. Each subsystem is connected through a clear unified and standardized service API interface, and is coordinated by SOA middleware in the middle, rather than cross-connected directly
4. The data to be accessed by each subsystem is defined as master data. There is a special database to place the main data, a special management system UI operation can manage the main data, there is a special background service engine to update, log, copy, distribute and push the main data, and there is a special API service for other programs to pass interface to access master data
5. The statistical report and the business subsystem are separated. The data of the business subsystem is copied and imported into the data warehouse through the engine, and the statistical report is made in the data warehouse. In this way, statistical reports can meet more complex requirements, be customized more flexibly, and do not affect the normal processing of business subsystems during runtime.
With these five basic architectural principles, many enterprise applications are already settled. Adding indexes, optimizing SQL, and archiving historical data are all tricks.
Of course, if you have higher performance requirements, the cloud network can help you: SLB can perform offloading, you can deploy one set of subsystems to multiple sets, and deploy them to different servers. The subsystem code runs, and the CPU consumes the most. , without consuming much memory, bandwidth, and storage. Of course, if you deploy multiple sets of one subsystem, another requirement arises. You need automated DevOps tools to automate one release and multiple sets of deployments on multiple machines, which can ensure the code logic on multiple servers. is consistent.
You also need to do the separation of motion and static. Static, Apache/Ngnix and browsers will do a lot of caching for you, so you don't have to go to the server again and again to get information. Moreover, the CDN of the cloud network can also help you distribute static files to various nodes across the country, allowing users to access them nearby. This way you can get higher performance.
If you still need higher performance, cloud database can help you. But you still need to change your code structure. You need to apply Redis, put the data into the memory first, and the program will automatically preferentially access the data in the memory instead of fetching the data from the disk. The data in the memory is updated regularly according to the business scenario.
If you also need higher performance, you can also use a distributed database.
If you also need higher performance, you need to choose different data storage forms and different data engines according to your data types and characteristics. So all kinds of NOSQL are produced, some are very good at processing time series data, and some are very good at processing graph mesh data, so that whether you are order sequence or social network, there is a suitable data storage engine , so that access to the data will be faster.
Fourth, talk about integration
We didn't mention integration when we mentioned scalable technical architecture just now, because we want to mention it specifically.
There are several levels of integration:
1. UI layer: unified enterprise portal, single sign-on
2. UI layer: URL call. By passing in the correct Session and URL parameters, access by HTTP GET or POST, and directly call out the UI window to which other subsystems belong. Of course, this kind of call is a bad design, and the independent deployment of each subsystem will also cause XSS cross-domain access problems. In this way, the calls between the various subsystems are forced to the logical service layer.
3. Logic layer: Expose interfaces through WebService, REST, etc., and make interface calls
4. Logic layer: Many bad programs directly write many SQL statements of SELECT, INSERT, UPDATE, DELETE in the logic layer, and they are across various subsystems. This is bad design. Therefore, in the beginning of code development, each subsystem should be a different code base and a different database, so that cross-subsystem SQL calls will not be generated from the beginning.
5. Data layer: There are many bad programs. The data tables of N subsystems are all placed in a database, and many VIEWs, stored procedures, and even triggers have been written. These are all programs, and the programs are scattered in various layers. , and they are all across the subsystem, and the wires are entangled. So I personally tend to concentrate all kinds of code in the logic layer, just make the database into data access, don't give it too many responsibilities
6. Task layer: There are still many bad programs. Some background tasks need to be triggered according to conditions or scheduled for execution. A lot of code is also written here. It is suggested that the logic code is also placed in the logic layer, and the task layer code is only for scheduling and combining, and do not write all the detailed logic code here.
7. Data layer: Master data integration. How master data is updated by multiple subsystems, how to expose services to be called by subsystems, and how to replicate and distribute data synchronously
8. Data layer: Statistical analysis integration. This needs to be made into an independent data warehouse to do
It can be seen that integration, vertical calls across layers, horizontal and horizontal calls, how many crossovers are generated, and thus many things that affect the whole body. What was originally a simple thing was made into a paste by the panacea. In order to integrate clearly, it is necessary for us to concentrate the calls in the logic layer instead of scattered everywhere.
For clear integration, we also need to introduce middleware.
1. In the past, SOA middleware was used to solve the call between interfaces. Do not let the interfaces directly call each other to generate spider webs, but form a traffic post center for intermediate command, and perform service registration, discovery, routing, security verification in the middle. Multi-protocol transport.
However, the Internet is meshed, distributed, centerless, and cannot fail at a single point. Therefore, the SOA middleware game cannot be played on the basis of the Internet architecture in the cloud era. To decentralize.
The hot microservice architecture and microservice middleware in recent years are somewhat different from SOA. The microservice architecture focuses on more fragmented service granularity and single service capability, forming the form of Open API. In the future, development programs will be directly combined based on the Open API open on the cloud, forming a scene-based function. AWS The lambda is an attempt at this idea. In the mobile era, we do not pay attention to a large integrated ERP suite with thousands of function points, but pay more attention to an APP satisfying a core scenario, and an APP only has a dozen functions. Of course, if you do too many functions, the mobile screen is so big, or the multi-touch type is not easy to input, and the mobile APP is useless.
As for Google's Protocol Buff and Facebook Thrift, they are just for smooth Service-client service invocation and data transmission between multiple languages, and they are also service middleware in a small range.
2. With single sign-on and master data Open API services, calls, data, and messages between programs are still needed. What should we do in the era of cloud computing and Internet?
Kafka and Zookeeper come on stage. Distributed Kafka, through the message queue pipeline, transmits data and trigger message notifications to both ends of the Service-client. ZooKeeper is used as an outsourcing service to coordinate the scheduling between various services.
With the microservice architecture, microservice middleware, Kafka, and Zookeeper, the centralized and central SOA middleware in the past has been disintegrated and replaced by the current distributed and non-centralized architecture, and it has really returned to the network structure of the Internet. .
5. Summarize the technical architecture mode and code development mode in the cloud computing era
1. H5 front-end technology, suitable for each device
2. Microservice technology architecture, a function point may be a service, which publishes Open API for other applications to call. The microservice technology architecture makes the code no longer mixed, the code scale is small, easy to understand and easy to modify, and it is easy for newcomers to get started
3. Microservices are deployed in Docker containers. Docker isolates the complexity of the mutual influence of the deployment of various basic supporting components. The supporting components and versions used by this microservice are deployed, and the components and versions used by other microservices are different. Fighting, which improves the stability of service operation and maintenance a lot
4. Cloud computing provides specialized object storage, block storage, CDN, SLB load balancing, and virtual private network; cloud computing provides various cloud databases, such as MySQL, Redis, MongoDB, PostgreSQL, etc.; cloud computing provides various Distributed middleware, such as Kafka message queue, nutch crawler, elasticsearch search; cloud computing vendors also provide operation and maintenance of these basic software, such as backup, migration, expansion, patch upgrade, etc. Microservices only need to write application logic code, and other required file access, data access, and data transmission can be satisfied by directly calling the API provided by cloud computing.
5. Various microservices provide standard and standardized Open APIs to the outside world. There are more and more such microservices. When we write an application, many times we can build an application by connecting various Open APIs together.
6. Our code is hosted on a proprietary cloud Github, with special version control management tools, and special R&D engineering efficiency tools, such as automated compilation tools, automated testing tools, multi-node automated unified deployment tools, and grayscale release. Tools, project planning and task management tools, document management/requirements management/bug management tools are also deployed in the underlying cloud IaaS.
6. Finally, talk about the selection of cloud computing IaaS
For small start-up companies, cloud computing is generally selected as follows:
1. Domain name convenience: convenient for domain name registration, domain name binding, and domain name filing
2. Payment convenience: such as Alipay, WeChat payment, scan the code to pay
3. The price is low: such as more than 100 yuan a month, only 1,000 yuan a year
4. There is AE package: you can choose online, pay online, or even 0 yuan promotion, and install it directly with one click. It is easy to deploy your own programs
5. Provide basic applications: such as free enterprise mailboxes. If there are free wordpress and enterprise templates, you don't need to install them, just open them directly, and bind the domain name directly, and the official website of the enterprise will be there. The shelf of an enterprise is quickly put up, and it looks like a regular company from the outside.
For mid-company selection cloud IaaS:
1. Various engines are very important. Chinese companies have done a good job in their applications, but most of their R&D efforts are mainly devoted to application R&D. They do not have the ability to develop and operate large-scale basic engines, so if cloud computing vendors can provide a lot of low-level engines , they will use it directly. Such as search recommendation engine, artificial intelligence engine, large-scale log engine, big data technology platform. You don't need to research and develop yourself, and you have ready-made high-tech to give yourself the main force.
2. Price. I want to eat good meat and I want a low price, so I can only charge as the business grows, which is easy to accept. However, because all business applications are supported by the basic engine, it is basically impossible to relocate.
3. There is a relatively guaranteed cloud security: at least you can't be attacked by DDoS, installed with broiler Trojans, or infected with viruses. The R&D personnel of Chinese companies mainly focus on application R&D, and generally do not have specialized security technical strength.
For large companies to choose cloud IaaS:
1. Stability is very important. Such as network stability, storage stability, memory CPU motherboard stability, power stability. This requires cloud computing manufacturers to invest a lot of money. If the money is not invested, only low-end products are selected, which is naturally unstable and the performance is not high.
2. Solutions are important. Large companies generally have complex applications, requiring multiple backups in different places, multiple activities in multiple places, traffic diversion, and fast switching. This requires cloud computing to have strong technical solution capabilities. Even some large companies provide services overseas, so they also attach great importance to the cloud computing resource capabilities of cloud computing manufacturers around the world, such as Hong Kong, Japan, Southeast Asia, Europe, and the United States.
3. Technical support is very important. If there is a problem, you can respond quickly, find the root cause of the problem quickly, repair or upgrade quickly. This has high requirements for the scale, technical capabilities, automation tools, and good collaboration between departments of cloud computing vendors' technical teams.
4. The service attitude of the technical service team is also very important. Originally, the customer was very anxious when there was a problem. If the technical service team's attitude was sloppy and unkind, or even deceived the customer, delayed time, and could not coordinate internally, the customer would explode.