Author: Zen and the Art of Computer Programming
1 Introduction
In the past five years, China's major Internet giants have launched many emerging products and services, ranging from cloud computing platforms, search engine systems, payment platforms, social networks, e-commerce platforms to big data analysis platforms, intelligent robot platforms, etc. It will have a significant impact on the current and future digital economic system. As a technician, how to better understand and master the technical architecture of these large manufacturers, and then better participate in technological innovation and management, will be a very important ability. This article is written based on actual experience. It aims to analyze the core technical principles and specific operational processes behind them by sorting out the technical architecture of more than a dozen well-known Internet companies at home and abroad, combined with my own accumulation in actual work. At the same time, I also hope to be able to raise some challenging technical issues, stimulate readers' interest and enthusiasm for industry technologies, and explore ways to deal with them.
2. Explanation of basic concepts and terms
First of all, we need to know what is technical architecture? What kind of thing is it? What are its characteristics? And how can these characteristics be applied to actual work? Let's take a look at the relevant knowledge.
2.1 What is technical architecture?
Technology Architecture (TA) refers to an architectural term and refers to a series of decisions, standards, principles, methods, tools, processes or processes used in engineering design and construction. According to the international standard ISO/IEC 42010:2011, "technical architecture" consists of five steps: "architecture, design, planning, construction, implementation and testing", which are used to establish technical direction, formulate technology roadmap, select design tools, design Plan, build, test and maintain technology solutions. In short, technical architecture includes many aspects, such as architectural design, technical solutions, system design, interface definition, development specifications, testing strategies, deployment plans, etc.
In software engineering, technical architecture is generally used to guide the design, development, testing, integration, and maintenance of software. The technical architecture of different industries is often different, and some industries even have no technical architecture. For example, Dianping's technical architecture does not exist because its product itself is a catering website and does not require complex architecture design. Only enterprise-level software needs to consider technical architecture.
The five steps of technical architecture are as follows:
- Architecture phase: At this time, the technical vision and goals are formulated, requirements and technical value are clarified, and strategic planning is formulated.
- Design stage: mainly high-level design, system design and detailed design. Including functional design, module design, structural design, data flow design, etc.
- Planning phase: Make plans and allocate resources through effective management of project resources, budget and time.
- Construction phase: Code writing and testing according to plans and plans and design documents.
- Testing phase: Conduct unit testing, integration testing, performance testing, compatibility testing, security testing, operational testing, etc. on the completed code to check whether its operation meets the design requirements.
Each step needs to cover the entire technology life cycle, involving product research and development, architecture design, development implementation, operation and maintenance management and other aspects. Therefore, the technical architecture is not just an empty shell, but also a product formed in the process of interconnection and mutual promotion. It requires in-depth research around needs, products, teams, etc. from multiple perspectives.
2.2 Why is there a technical architecture?
The purpose of technical architecture is to help enterprises establish scientific and reliable technical infrastructure, promote effective communication between information technology and the industrial chain, and maximize innovation and results. Below we briefly explain the characteristics of technical architecture.
2.2.1 Architecture emphasizes cross-border integration
Technical architecture not only focuses on its own internal technical capabilities, but also cares about how to effectively integrate between different fields. This cross-border integration involves two dimensions: product and technology. Products can be divided into Product Architecture, UI/UX Design Architecture and Branding Architecture, while technologies can be divided into Microservices Architecture and Container Architecture. and Big Data Architecture.
These architectural designs do not exist in isolation, but support and promote each other. For example, due to the rapid popularity of cloud computing technology, many traditional companies have begun to realize the business opportunities of cloud computing. However, the traditional software development model still dominates. In order to adapt to this new business model, technical architecture often introduces new architectural models such as microservice architecture and container architecture.
In addition, the architecture can also be empowered by management technology, intelligent algorithms and artificial intelligence technology to give organizations greater creativity and flexibility. Enterprises can improve management, optimize production efficiency, enhance competitiveness, save costs, and prevent innovation stagnation through technological architecture.
2.2.2 Architecture reflects value sharing
Technical architecture does not only focus on its own technical capabilities, but also focuses on collaborative cooperation and win-win results. Architecture not only defines the way technology is implemented, but also focuses on technical consensus and value sharing.
First, the technical architecture needs to be implemented into daily management practices. Since technical architecture is a long-term investment that requires continuous iterative updates, management needs to establish a close connection with the technical architecture. For example, management can use the architecture's decision support, implementation specifications, technical specifications, automation tools, etc. to ensure that the team's work proceeds smoothly.
Secondly, the technical architecture also needs to provide a reference for communication between different departments. Different teams often have different scopes of responsibilities and functional divisions, and require unified technical architecture standards to communicate effectively. Architecture plays a role in many areas including organizational structure, product development, and user experience.
Finally, technology architecture also needs to facilitate organizational change and innovation. Because the technical architecture improves the technical capabilities of the team, giving them better execution capabilities and management levels, it gives the organization more opportunities and space to explore and practice innovation.
2.2.3 Architecture is the driving force behind innovation
Technical architecture is not a smooth process. Today's world is undergoing profound changes, and technology architecture needs to constantly track and respond to changes.
For example, the development of cloud computing technology, blockchain technology, artificial intelligence technology, and changes in the overall development direction of the Internet industry will affect the design, implementation, and application of technical architecture.
Therefore, technical architecture is the driving force behind innovation and self-empowerment. Only by focusing on business value creation and social value realization can the technical architecture be continuously updated to provide a better environment for innovation.
2.3 Which companies have good technical architecture?
A good technical architecture must serve both business and technology. A successful technical architecture should first serve business needs and then provide technical support. Below we list several companies with good technical architecture.
### 2.3.1 Facebook Facebook has employees full of innovative spirit, focusing on "openness" and abandoning closedness and refusing to change. As the world's most famous social media company, its technical architecture has gone through three generations, and it generally shows a forward-looking trend.
The five steps of Facebook's technical architecture include: architectural design, technical implementation, monitoring, improvement and cultural transformation, as follows:
Architecture design: Facebook’s core architecture is to provide stable services for trillions of users. In terms of architectural design, Facebook uses the "hexagonal architecture" as the basis to break the traditional single large server architecture and adopt a distributed and service-oriented architecture to split the system components into small and easy-to-manage independent services.
Technical implementation: Facebook adopts a microservice architecture to disperse different functional modules to different servers and interact with the front end through external communication protocols such as RESTful API and GraphQL to achieve rapid iteration, elastic expansion and scalability.
Monitoring: Facebook has built a complete monitoring system that can identify faults, discover problems, quickly locate problems and take corresponding measures. All Facebook backend services record logs, send alerts, and monitor service dependencies through distributed tracking technology to help developers quickly locate and resolve faults.
Improvement: In terms of architectural improvements, Facebook has launched a variety of technical solutions such as an open API platform, dynamic advertising system and timeline function. These technical solutions not only meet the company's business development needs, but also effectively improve user experience.
Cultural transformation: Facebook encourages employees, customers and employees to grow by changing the platform culture. Among them, advanced culture such as flat offices, free WIFI, and dinners provide employees with learning opportunities.
In general, Facebook's technical architecture abstracts complex technical issues into a simple and easy-to-understand service-oriented architecture, helping the company better serve consumers.
### 2.3.2 Google Google's technical architecture, from the search engine in 2007 to today's AI image recognition technology, all highlights its huge software architecture, closely coordinated product portfolio and unified technology stack.
Google's technical architecture is the most representative case of global technology. It consists of five steps: design, build, deploy, improve and use.
Design: Google's design ideas have a long history. They were founded in Bell Labs and were later imitated by Amazon, Apple, Microsoft, etc. From databases to infrastructure and applications, Google spends a lot of time and money on product design.
Construction: Google's software construction process follows the methodology of Lean Development, that is, rapid construction, small batch integration, and frequent releases to achieve product perfection.
Deployment: Google has three deployment systems, namely cloud, big data and Internet of Things. Each deployment system has different responsibilities.
Cloud deployment system: Responsible for providing services such as Google Cloud Platform and Firebase to provide reliable, low-latency services to a large number of users.
Big data deployment system: Google's big data processing framework is mainly Apache Hadoop, which stores massive data in the HDFS file system and analyzes and mines it through processing frameworks such as MapReduce, Hive, and Spark.
IoT deployment system: Google’s IoT processing system is mainly Google’s cloud IOT Core, which is responsible for connecting, managing, analyzing, processing and storing IOT device data.
Improvement: Google's technological leadership is largely due to its internal technical strength and keen judgment on the market. For example, the success of the Chrome browser opened up a new browser market and profoundly affected the development of the Internet.
Usage: Google's architecture services are directly facing end users and provide a range of products and services including search, web pages, videos, pictures, music, App Store, and YouTube.
In general, Google's technical architecture reflects the high degree of business intelligence and innovative spirit. It fully takes into account future trends and creates software that can meet user needs through lean construction, deployment, improvement, and use.
### 2.3.3 Uber Uber is the most well-known shared travel company in the United States. Its technical architecture is based on mobile applications and provides users with more convenient and time-saving services through Internet of Things technology, machine learning algorithms, big data analysis and other means. ride services.
Uber's technical architecture consists of three levels: mobile applications, IoT cloud, and data analysis.
Mobile application: The Uber mobile application is a full-stack product that uses mixed development technologies, including the four major technology stacks of iOS, Android, Web and React Native, and is extended using existing open source component libraries.
Mobile applications improve user experience by separating various functional modules, strengthening permission control and security mechanisms, and making feedback adjustments based on user data.
IoT cloud: The Uber IoT cloud platform is a collection of Uber infrastructure that provides services for drivers, passengers, logistics, shared equipment, vehicles, and other parts of the system.
Uber's IoT cloud platform adopts a modular architecture and is based on cloud platforms such as AWS and GCP to achieve infrastructure automation, automatic expansion and high availability.
Data analysis: Uber's data analysis platform is a real-time analysis of user behavior and habits to gain insight into user needs and improve service quality.
Uber's data analysis platform uses machine learning, statistical models and artificial intelligence technology to analyze user historical data and provide users with improvement suggestions through real-time feedback.
In general, Uber's technical architecture combines professional knowledge, excellent product design and cutting-edge technology to provide users with satisfactory travel services.
# 3. Core algorithm principles and specific operation steps. After understanding the technical architecture, let’s take a look at the specific algorithm principles and specific operation steps.
3.1 Cloud computing architecture
Cloud computing architecture is currently the most widely used architecture model. It provides a series of solutions, including virtualization, containerization, automation, automatic expansion and elastic expansion, etc., and provides cloud computing services with pay-as-you-go or usage rights for computing, storage, network and other IT resources. The core algorithm principles of cloud computing architecture are as follows:
3.1.1 IaaS
IaaS (Infrastructure as a Service) refers to a service in which the provider provides hardware, network, storage and other infrastructure to individuals or organizations through the network or other forms. Its core algorithm principle is as follows:
Elastic expansion: When the application system requires more resources than are actually provided, it can be solved by purchasing more cloud resources. This expansion capability is called elastic expansion.
Automated deployment: Cloud service providers can complete system deployment within seconds, significantly reducing operation and maintenance time.
On-demand billing: Cloud service providers can charge for cloud resource usage based on user usage, achieving on-demand billing through price comparison rather than quantity comparison.
High availability: Cloud service providers ensure high availability of the system through redundant hardware and network settings.
Programmable: Cloud service providers can configure and program the system through scripting languages or other methods to flexibly meet users' individual needs.
Local connection: Cloud service providers provide users with local connections to access cloud resources from anywhere, reducing network bandwidth consumption.
With the development of cloud computing technology, IaaS is becoming the most popular cloud architecture model. With the popularity and application of cloud computing, more and more Internet companies are beginning to use IaaS for data center and server hosting to improve their service capabilities.
## 3.2 Distributed architecture Distributed architecture is the latest trend in the development of computer technology today. It distributes different data, tasks and computing resources to different locations through the network, and realizes data sharing, resource sharing and load balancing through message passing and remote calling. The core algorithm principle of the distributed architecture is as follows:
Distributed storage: Distributed storage is the basis of distributed architecture. By distributing data to different nodes, data sharing and fault tolerance are achieved.
Distributed computing: The core of distributed computing is to distribute computing tasks to multiple nodes for parallel processing while maintaining data consistency.
Distributed Scheduling: Distributed scheduling is a key component of distributed architecture. It achieves maximum utilization of cluster resources by mapping tasks to different nodes.
Distributed lock: Distributed lock is an important synchronization mechanism in distributed architecture. It prevents multiple processes or threads from accessing the same data or resources at the same time.
Distributed transactions: Distributed transactions are an important feature of distributed architecture. It maintains the atomicity and consistency of data operations.
With the integration of cloud computing and distributed architecture, new software systems are emerging. For example, distributed databases, distributed message queues, distributed computing engines, distributed file systems, etc. The rapid development of distributed architecture has promoted the development of cloud computing.
# 4. Specific code examples and explanations Finally, let’s take a look at some specific code examples and explanations.
4.1 Elasticsearch architecture principles
Elasticsearch is an open source search server that provides a distributed, RESTful search engine. Elasticsearch supports multi-tenancy, RESTful API, SQL syntax query and other features. Its architecture principle is shown in the figure below.
Master node: The Master node is responsible for cluster management, including cluster status maintenance, metadata sharding, copy distribution, load balancing, etc.
Data node: The Data node is responsible for data storage and retrieval. Each Index corresponds to a Lucene index file, which stores the actual document data. Data nodes use Java heap memory, disk storage, and network transmission resources.
Client node: The Client node is responsible for communicating with the cluster, receiving user requests, and sending query requests.
Ingest node: The Ingest node is mainly used to process, analyze and convert data. Scenarios such as data import and report generation can be handled by the Ingest node.
Coordinator node: The Coordinator node is mainly used to coordinate distributed operations, such as creating indexes, deleting indexes, replicating shards, etc.
The Elasticsearch architecture clearly demonstrates its distributed architecture, which can effectively handle massive data and improve search speed.
# 5. Future development trends Finally, let us take a look at the development trends of future technical architecture.
5.1 Containerization architecture
Containerized architecture is a new architectural pattern that enables software systems to be deployed in lightweight virtual containers, providing a high degree of isolation and resource utilization. The core algorithm principle of containerized architecture is as follows:
Fast startup: The containerized architecture can start in seconds, making application deployment and startup faster.
Lightweight virtualization: The virtual machine of the containerized architecture takes up less space, has fast startup speed and short startup time, and can save cloud computing resources.
Automatic deployment: The deployment process of containerized architecture is automated and does not require manual configuration and installation.
Convenient and elastic expansion: The containerized architecture enables applications to easily and elastically expand through dynamic resource allocation based on container orchestration tools.
Service interval: The containerized architecture can avoid communication interference between services through network isolation.
Observability: The management system of the containerized architecture can intuitively display the service running status and resource utilization.
With the integration of cloud computing and containerized architecture, container cloud has become the mainstream form of cloud computing in the future. Containerized architecture will redefine the way software development and operation and maintenance work, providing new opportunities for application deployment and operation and maintenance.
5.2 Service grid architecture
Service mesh architecture is an important part of microservice architecture. It realizes service transparency, enhanced reliability, and improved flexibility by automating communication and traffic management between microservices. The core algorithm principle of the service mesh architecture is as follows:
Service discovery: The service grid architecture realizes automated communication between microservices through service registration and service discovery, and realizes dynamic discovery of microservices.
Traffic management: The service grid architecture can provide quality of service, traffic control, circuit breaker, load balancing and other functions between microservices to enhance the reliability of microservices.
Security authentication: The service grid architecture provides a unified authentication and authorization mechanism to protect the private information of microservices.
Observability: The service grid architecture provides monitoring information such as the health status, performance indicators, calling relationships, and exception information of microservices, and can grasp the operating status of microservices in real time.
Configuration management: The service grid architecture can automatically configure microservices to achieve elastic scaling and resource allocation.
With the development of service grid architecture, its applications will spread across various industries and become an important part of microservice architecture.