Programming multiple roads, all lead to Rome

Backstage boss

The backend route is all about backend server business, such as web backend server, video backend server, search backend server, game backend server, live broadcast backend server, social IM backend server, etc. Most of the code is related to business logic. I want to be a big boss. , must be proficient in professional field business knowledge.

But at the same time, there are also some common technical requirements, such as familiarity with programming languages, data structures and algorithms, network programming, TCP/IP protocol, database, middleware, high-performance, and high-availability technologies.

Backend technology evolution

Architecture evolution

With the application of PC local area networks, especially relational databases, the infrastructure developed into a two-layer architecture; followed by the development of wide area networks, from a single multi-layer architecture, SOA and EDA architectures became popular; next came virtual machines, In today's cloud computing infrastructure, microservices have emerged, followed by Container as a Service, Serverless, and recently Huoyuan Yunsheng architecture. It can be seen that changes in architecture are all about making full use of IT infrastructure.

Business goal evolution

In the past, when Internet traffic exploded, it was important to withstand traffic peaks first, to achieve high concurrency and high performance, and to support horizontal expansion, which were important goals of backend design. At present, Internet traffic has peaked and competition for existing stocks has intensified. The stability of back-end services has become more and more important. Enterprises have become more determined to reduce costs and increase efficiency. R&D efficiency, monitoring and operation and maintenance platforms, automated testing, CI/CD pipelines, etc. have also become important. stand up.

Backend development language evolution

In an era when server hardware resources are expensive, C++ has both high performance and code reuse (OOP programming), and has become the main language for backend development of many major manufacturers.

The mainstream of the first generation of web backend development was PHP. At that time, the mainstream backend architecture of the Internet was the LAMP architecture. With the rise of e-commerce, the popularity of Android phones, and the emergence of big data, the development of the JAVA technology stack was promoted, and JAVA became the mainstream backend programming language of the Internet.

With the advent of the cloud computing era and the rise of cloud native computing, the Go language ecosystem has developed steadily, taking into account both performance and development speed. More and more companies are using Go language in production to implement business. Currently, many major manufacturers have begun to turn to Go as their backend development language.

The development of artificial intelligence has also promoted the development of Python language. It is simple, quick to learn, and highly efficient in development. It has become a development language that does not care about the performance of background components.

As security and stability become more and more important, Rust may become a development language for key backend components, taking into account both performance and memory safety, and be used to replace the core C++ components of the backend system;

For the future, Python, Go, and Rust have become the first programming languages to consider learning in the future of the backend.

At present, the mainstream backend languages of various major domestic manufacturers are different:

Tencent prefers C++, Go, etc. Go is becoming more and more popular
Alibaba, Pinduoduo, Meituan, and JD.com prefer Java
Bytes biased towards Go/Python
Baidu prefers C++
Huawei prefers C/C++

Middleware expert

Middleware is a basic software used in distributed systems. Since its birth in the 1980s, it has quietly played an important role in distributed environments. Based on middleware, efficient connection and communication are achieved between system software and application software, and application development can be accelerated.

Message middleware

The ActiveMQ community is relatively mature, but compared to the current situation, the performance of ActiveMQ is relatively poor, and the version iteration is very slow, so it is not recommended.
Although RabbitMQ is slightly inferior to Kafka and RocketMQ in terms of throughput, because it is developed based on erlang, it has strong concurrency capabilities, extremely good performance, and very low latency, reaching microsecond level. However, because RabbitMQ is developed based on erlang, there are few domestic companies that have the strength to do research and customization at the erlang source code level. If the business scenario does not have too high concurrency requirements (100,000 level, one million level), then among these four message queues, RabbitMQ must be your first choice. If it is real-time computing, log collection and other scenarios in the field of big data, using Kafka is the industry standard, and there is absolutely no problem. The community is very active and will definitely not be pornographic. Moreover, it is almost a de facto standard in this field around the world.
RocketMQ is produced by Alibaba and is a Java open source project. We can read the source code directly and then customize our own company's MQ. RocketMQ has been tested in Alibaba's actual business scenarios. RocketMQ community activity is relatively average, but it's okay, and the documentation is relatively simple. There is also the technology introduced by Alibaba. If this technology is abandoned, you have to deal with the risk of losing the community. If your company has the technical strength, I think it is good to use RocketMQ.
The characteristics of Kafka are actually obvious, that is, it only provides fewer core functions, but it provides ultra-high throughput, ms-level delay, extremely high availability and reliability, and the distribution can be expanded arbitrarily. At the same time, it is best for Kafka to support a smaller number of topics to ensure its ultra-high throughput. The only disadvantage of Kafka is the possibility of repeated consumption of messages, which will have a very slight impact on data accuracy. In the field of big data and log collection, this slight impact can be ignored. Kafka is naturally suitable for big data real-time computing and log collection.

Next-generation messaging middleware Apache Pulsar:

Compare kafka

The fundamental difference between Apache Pulsar and Apache Kafka is that Apache Kafka uses partitions as the storage center, while Apache Pulsar uses segments as the storage center. Apache Pulsar is a unique segment-centered publish/subscribe based on distributed log storage. Messaging systems can provide many advantages, such as a reliable streaming system including unlimited log storage, instant scaling without partition rebalancing, fast replication repair, and high write and read availability options with maximized data placement.

Caching middleware

We all know that the function of CPU cache is to reduce memory access. It also extends to distributed systems. Caching middleware can improve the access performance of component data.

Redis is a popular cache middleware. According to the principle of locality, hot and cold data are separated and is generally used to speed up high-frequency data access to the database:

Future optimization directions:

High availability
Persistence optimization
Secure encryption
IO, connection optimization
Multi-thread optimization
Data structure optimization, supporting more data structures

RPC framework

RPC refers to remote procedure call, that is to say, there are two servers A and B. An application is deployed on server A. If you want to call the function/method provided by the application on server B, it cannot be called directly because it is not in the same memory space. You need to go through Network to express the semantics of the call and convey the data of the call.

Remote service invocation framework in the microservice era. Such as grpc, Thrift, Ali's HSF, Dubbo, SOFA-RPC;

Future direction:

Support microservice technology evolution
Framework invasive improvement, language independent, communication protocol independent.
Service Mesh, Service Mesh is an infrastructure layer that runs independently outside of application services, provides safe, reliable, and efficient communication between application services, and implements the basic component functions required for service communication to run microservices, including Service registration discovery, load balancing, fault recovery, monitoring, permission control, etc.
Performance optimization, serialization protocol optimization, message encoding optimization, network IO optimization, etc.

load balancing

Load Balancing is a key component of high-availability network infrastructure. It is usually used to distribute workloads to multiple servers to improve the performance and reliability of websites, applications, databases or other services.

Hardware: F5, Redware...

Software: lvs (fourth layer), haproxy (fourth and seventh layers), nginx (seventh layer)...

Future direction:

Support more intelligent scheduling algorithms:

Round robin - Requests are distributed sequentially across a group of servers.

Least Connected – A new request is sent to the server with the fewest current connections to the client. The relative computing power of each server is taken into account to determine which server has the fewest connections.

Minimum Time - Sends the request to
the server selected by a formula that combines the fastest response time and the fewest active connections. Exclusive to NGINX Plus.

Hash – Distributes requests based on a key you define, such as client IP address or
request URL.
If the set of upstream servers changes, NGINX Plus has the option to apply consistent hashes to minimize load redistribution.

IP Hash – The client's IP address is used to determine which server receives the request.
Business is moved to the cloud and load balancing is performed on the cloud (LaaS, PaaS services).
High performance optimization, cost reduction, large traffic (IO optimization, DPDK, FPGA, P4 evolution), hardware.
High availability, observability, monitoring statistics, alarm system, smooth expansion, service elimination, and statelessness.

Kernel Master

Kernel route, explore the underlying mysteries

cloud computing

The cloud computing process is accelerating and everything is a service. As a result, the underlying technology that did not make money before can now be sold for money. The technology can be exported through cloud computing. This is the arrival of spring for the underlying technical people:

Software as a Service , Software as a Service, or SaaS for short , the function of this layer is to provide applications as services to customers.

Platform as a Service , Platform as a Service, or PaaS for short , the function of this layer is to provide a development platform as a service to users.

Infrastructure as a Service , infrastructure as a service, referred to as IaaS , the role of this layer is to provide virtual machines or other resources as services to users.

IaaS core technologies : computing , network , storage , these can be regarded as the bottom layer of software technology (the bottom layer is hardware), basically you have to deal with the kernel OS, and you need to have the ability of secondary development of the kernel and be familiar with the implementation of the operating system (mainly familiar with the Linux kernel source code).

Computing: computing virtualization, kernel scheduling system, cgroups, KVM, QEMU, virtio;

Network: network virtualization, kernel protocol stack, netfilter, netns, DPDK, smart network card, RDMA, P4, etc.;

Storage: memory virtualization, disk virtualization, SPDK, etc.;

PaaS core technology : application operating environment (container, multi-tenant elasticity, K8S), application lifecycle support (devops, automated operation and maintenance), integration, composite application construction capabilities (CI/CD), etc.

The threshold for these technical positions is very high, and the salary is higher than that of ordinary positions. If you are very interested in the underlying technology, you can pay attention.

Browser kernel

webkit

WebKit is the web browser engine used by Safari, Mail, the App Store, and many other applications on macOS, iOS, and Linux.

Chromium

Chromium is a free and open source code library for web browsers, primarily developed and maintained by Google. Google uses the code to make its Chrome web browser, which has additional features.

The Chromium codebase is widely used. Domestic browsers such as Microsoft Edge, Opera, QQ Browser, UC Browser, and many other browsers are based on this code. Additionally, significant portions of the code are used by multiple application frameworks.

chromium architecture

C++ is the main language, accounting for about half of the code base. This includes the Blink and V8 engines, implementations of HTTP and other protocols, internal caching systems, and other essential browser components. Some user interfaces are implemented using HTML, CSS and JavaScript. A large number of web platform tests are also written in these languages. About 10% of the code base is written in C. This mainly comes from third-party libraries that provide basic functionality, such as SQLite and numerous codecs. Supporting mobile operating systems requires special languages: Java for Android, and both Swift and Objective-C for iOS. (A copy of Apple's WebKit engine is also in the code base, as iOS browsers require it)

If you are interested in web core technology, you can choose the browser direction!

database kernel

Technical original intention

After the emergence of operating systems, as the scope of computer applications expanded, the data that needed to be processed expanded rapidly. Initially, data, like programs, were stored in simple files as the main form. Data organized in this way is logically simpler, but less scalable, and programs that access this data need to understand the specific organizational format of the data. When the amount of system data or user access is large, the application also needs to solve a series of issues such as data integrity, consistency, and security. Therefore, a system software must be developed that can shield the complexity of data access in the same way that the operating system shields the complexity of hardware access. From this came the data management system, the database.

Current status

All over the world, database development is basically based on open source projects. Even self-developed projects will definitely refer to existing open source projects. Therefore, to choose a database, you must play open source very well, such as MySQL, PostgreSQL, MongoDB, LevelDB, etc. At present, cloud computing vendors are vigorously developing databases. Among the domestic public cloud deployment models, Alibaba, Tencent, AWS, Oracle, Huawei, and Microsoft rank among the top six. In terms of the domestic database industry, database vendors have achieved the best results in 40 years. Development opportunities, the market environment (national de-IOE strategy) is conducive to domestic manufacturers, the technology is generally close, and some technologies are on par or even ahead.

Currently popular directions:

Distributed databases, sharding, transactions, consistency, etc.
Database performance optimization, SQL instruction optimization, software optimization, hardware acceleration
High availability architecture
Migrating existing databases to the cloud
Intelligent database, automatic management and control
Cloud native architecture

If you are interested in database technology, you can choose this path.

operating system

At present, the country is vigorously developing new infrastructure, encouraging and supporting enterprises to develop basic technologies. The operating system is currently an important direction. At present, major companies are either developing their own OS or conducting secondary development based on open source OS.

Currently popular technical directions:

Hongmeng open source OS, self-developed OS
Embedded OS, self-developed and based on embedded Linux
Mobile phone OS, secondary development based on Android
Cloud computing OS, based on Linux kernel, Redhat distribution and other secondary development

The development of OS is mainly to adapt to new hardware, performance optimization (scheduling performance, memory allocation performance, protocol stack processing performance, file system optimization), stability optimization, virtualization technology, etc.

If you have feelings for OS and like to deal with underlying hardware, you can choose the operating system direction. The prospects are not bad at present!

Embedded

Due to the development of 5G, AI, mobile phones, smart hardware, autonomous driving, and the IoT field, various smart hardware are taking off.

Popular technical directions:

Embedded OS (Vxworks, Alios, TencentOS tiny, Huawei LiteOS, RT-Thread, etc.)
Smart home system
Mobile OS (Hongmeng, Android, etc.)
Mobile phone performance optimization (stronger performance and energy saving)
Autonomous driving technology

Enthusiasts who like hardware and geeks who like garage culture can choose!

JDK

Since Sun launched Java in 1995, the Java programming language has been popular for 25 years. There has been a lot of talk recently about the decline of Java, but Java is still the programming language of choice for domestic Chinese Internet companies, such as Alibaba, JD.com, Baidu, Tencent, Meituan, etc. With the rapid development of the Internet, big data, and AI, the domestic JAVA ecology has gradually been divided into several camps. The traditional field of J2EE enterprise applications is dominated by large manufacturers (Oracle, Microsoft), the Internet field is pivotal, and the Internet middleware is Alibaba. Cloud and pivot are promoting spring cloud, while big data and mobile Android are separate ecosystems.

OpenJDK (Open Java Development Kit) is a free and open source implementation of the Java Platform, Standard Edition (Java SE). Currently, all major companies are developing their own JDK versions. If you want to delve deeper into the Java field and want to switch to low-level development, you can choose the JDK path!

Distributed Expert

microservice

Service Mesh has maintained its popularity in the past year. In 2020 that has passed, microservices can be said to have both persistence and breakthroughs, the formation of a consensus on microservices, and rational thinking about special scenarios. We can see that the service framework is still evolving, moving towards cloud native and embracing cloudification. More and more enterprises are beginning to keep up with the pace of service-oriented cloudification.

Microservices, DDD, and mid-platform technology are not silver bullets for enterprise technology architecture design. The main disadvantage of microservices is the complexity brought about by the distributed characteristics of microservices. Developers need to implement calls and communications between microservices based on RPC or messages, which makes discovery between services, tracking of service call chains, and quality issues quite difficult.

Tencent’s open source microservices framework TARS:

However, microservices are still a popular direction of distributed distribution, which conforms to the design ideas of high cohesion and low coupling architecture. Problems arising in this process are waiting for us to solve, such as the emergence of Service Mesh .

Service Mesh runs as a Sidebar and is transparent to applications. All traffic between applications will pass through it, so control of application traffic can be implemented in Service Mesh.

Currently popular Service Mesh open source software includes Linkerd, Envoy and Istio. Recently, Buoyant (the company that open sourced Linkerd) released the Service Mesh open source project Conduit based on Kubernetes.

Middle platform architecture

At present, when a company becomes bigger and stronger, its business will inevitably increase, and reinvention of the wheel will begin. Due to business expansion, the manpower itself is insufficient, so the long-term plan will still use technology similar to the middle office, so as to save manpower. How to implement this is This tests the team and company's top-to-bottom promotion capabilities.

cloud native

Cloud native (CloudNative) is a compound word, Cloud+Native. Cloud means that the application is located in the cloud instead of a traditional data center; Native means that the application has been designed with the cloud environment in mind from the beginning, is designed natively for the cloud, runs in the best posture on the cloud, and makes full use of the cloud. Platform elasticity + distributed advantages.

At the beginning of 2021, the layout of cloud native began to accelerate. Huawei Cloud joined forces with CNCF (Cloud Native Computing Foundation) and China Academy of Information and Communications Technology to establish an original meeting to accelerate the implementation of the cloud native industry; Kingsoft Cloud released a cloud native panorama, cloud native product matrix and the latest serverless products; Nokia announced its cooperation with Google Cloud Develop cloud-native 5G technology... Almost all cloud computing products released by cloud vendors have been labeled as cloud-native.

Cloud native core technology :

Containerization : as a carrier for application packaging
Continuous delivery : Take advantage of the lightweight features of containers to build a continuous integration and continuous release pipeline
DevOps : The collaboration between development and operation and maintenance rises to a cultural level, allowing applications to be deployed and released quickly
Microservices : This is a concept of application development. Only by splitting a single application into microservices can we better realize cloud nativeness and independently deploy, expand and update it.

K8S has become the next generation of cloud native operating system

Kubernetes architecture

Kubernetes originally originated from Borg within Google and provides an application-oriented container cluster deployment and management system. The goal of Kubernetes is to remove the burden of orchestrating physical/virtual compute, network and storage infrastructure and enable application operators and developers to fully focus on container-centric primitives for self-service operations. Kubernetes also provides a stable and compatible foundation (platform) for building customized workflows and more advanced automation tasks. Kubernetes has comprehensive cluster management capabilities, including multi-level security protection and access mechanisms, multi-tenant application support capabilities, transparent service registration and service discovery mechanisms, built-in load balancers, fault discovery and self-healing capabilities, and rolling service upgrades And online expansion, scalable automatic resource scheduling mechanism, multi-granularity resource quota management capabilities. Kubernetes also provides comprehensive management tools, covering all aspects of development, deployment testing, operation and maintenance monitoring, etc.

If you want to become an Internet architect, K8S architecture is what you should know.

When should you consider changing jobs?

Generally in a company, after working hard for three years (fishing does not count), if the position has not been promoted or the salary has not doubled, you can consider changing jobs. Please remember our slogan is:

How to respond to changing technology trends

Software design has two key goals: high cohesion and low coupling . Around these two core goals, design principles such as single responsibility, opening and closing principle, Liskov substitution, dependency switching, interface isolation, and minimum knowledge are proposed.

Software engineers have always strived hard for these two goals in order to write software that is clearer, more robust, and easier to extend and maintain.

But later, people found that there were more demands, hoping that developing software would become simpler and faster. Programmers wanted to write less code, and non-professionals also wanted to develop programs. As a result, more and more stupid programming languages were used. Invented, more programming techniques and programming ideas were invented, such as libraries, components, and cloud infrastructure.

So many technologies have become dragon-slaying skills, such as assembly. Times have changed. After the founding of the People's Republic of China, animals can no longer become spirits, and there are no dragons to kill. Then many software engineers turned into parameter adjustment engineers, Call API experts, and library package experts. , assembly expert, this is the result of efficient division of labor, and also the result of technological development.

Looking at the development history of the technological Internet in the past two decades, the big trend is the sinking of technology . Especially in recent years, with the development and popularization of cloud computing, the infrastructure has become thicker and thicker, and business development has become easier and easier. , are becoming less and less technical, and the performance, load, security, and scalability issues that previously plagued small teams no longer exist. This can’t help but make the greasy uncles in the Internet industry remain silent, as if they will be involved in history in any minute. Flooding and irreversible.

Although it is undeniable that the importance of technology is decreasing, it is not so pessimistic. Thinking back to the PC era, when VB, Delphi, and MFC appeared, there were similar arguments. What you see is what you get. You can develop PC desktop programs with just a click of the mouse. Isn't it very high-end? At that time, coders probably had a lot more worries than they do now. But with the rise of the Internet, back-end development emerged, and coders quickly found new battlefields, such as networks, distribution, databases, and massive services, disaster recovery and error prevention, so they came up with a bunch of new tricks.

Technology is always developing forward , and we need to deepen our understanding of basic knowledge, adapt to changes, and delve into a field. At the same time, we need to try more new technologies, broaden our horizons, and increase our ideas for solving problems. , when you have a skill, even if the domestic 35 crisis, you can still go to foreign companies (Google, Facebook, Amazon, Microsoft, etc.) to retire.

Backend technology trend guide | How to choose your own technology direction