C/C++ Linux background server development senior architect study knowledge points route summary (2021 architects full version)
Foreword:
The editor has previously shared with you an article summarizing the knowledge points of the architect system, and today it has been changed and updated on the original basis (2021 version).
I believe you all know that it takes a certain amount of time and experience to become an architect before you can reach the level of an architect. Today I am sharing the technical knowledge points that you need to master to become an architect. He is a back-end development position in well-known companies such as Tencent and Ali.
Today’s update is based on the knowledge points of Linux senior Internet architects. The basic knowledge points will not be written here today. Friends who need to know can read my previous article: https://www.toutiao. com/i6769094897262658056/
Linux Senior Internet Architect Column
table of Contents:
One: Source code analysis topic
Two: Topics on Middleware Development
Three, cluster
Four: Linux system topic
Five: performance and testing topics
Six: Operation and maintenance statistics topic
Seven: Distributed Architecture Topics
Eight: Challenge the limits of network performance DPDK column
Nine, comprehensive analysis of the Linux kernel code column
Attached to the end of the article the overall idea map
One: Source code analysis topic
Source code reading ability is an essential way to improve technology
1. Nginx source code
Knowledge points include: Nginx infrastructure, HTTP architecture, inter-process communication mechanism, Nginx advanced data structure, slab shared memory, upstream mechanism design
2. Redis source code
Knowledge points include: Redis storage system principle, data model and key-value mapping, memory operation and disk synchronization, master-slave synchronization, atomic operation, and analysis of underlying IO implementation
3. Skynet source code
Knowledge points include: network module realization and data sharing, timer realization, actor realization and cluster cluster, lua auxiliary library realization, load balancing and hot update scheme, c service development and docking client, skynet strange skills
4. ZeroMQ source code
Knowledge points include: message model publish/subscribe/push-pull model, communication protocol inproc/ipc/tcp/pgm, performance analysis and classic MQ comparison, underlying network communication implementation mechanism, zmq system architecture and implementation principles
Two: Topics on Middleware Development
You must have the ability to make wheels, and you must also have the consciousness not to make wheels. SDK is a necessary packaging ability for advanced development.
1. High-performance components
Knowledge points include: how to design memory pools, message queues in high concurrency scenarios, implementation of coroutine frameworks, handwritten thread pools, asynchronous request pools, connection pools,
2. Highly concurrent network IO
Knowledge points include: the difference between select, poll and epoll models, Linux system IO model, understanding of Linux server high-concurrency programming, network access layer architecture design, and you must know how to play Epoll
3. Concurrency
Knowledge points include: multithreading and thread safety, transaction isolation level, fair lock & unfair lock, CopyOnWrite container, pessimistic lock & optimistic lock & CAS&ABA issues, transaction ACID characteristics
4. Web services
Knowledge points include: reverse proxy Nginx, the most extensive web server Httpd, high-performance web platform OpenResty
5. Timing scheduling
Knowledge points include: Distributed timing scheduling Opencron, the principle of trigger start on time, Linux timing task cron configuration
6. Message queue
Knowledge points include: message bus VS message queue, Redis message push, how to ensure the order in which consumers receive messages
7. Web security
Knowledge points include: CSRF principle and prevention, advanced Dos attack-Hash collision attack, script injection file upload vulnerability, SQL injection, XSS attack principle and solution
8. Network protocol stack
Knowledge points include: tcp/udp/ip/eth, multi-process to achieve high concurrency, sliding window, delayed confirmation, timing retransmission, flow control, tcp handshake wave
Three, cluster
Must-have product-level skills .
1. MySQL cluster
Knowledge points include: MySQL Cluster evaluation guide, read-write separation sub-database sub-table, Binlog mechanism analysis, sanity check and troubleshooting, data model and query design
2. Redis cluster
Knowledge points include: Redis Cluster solution, node automatic discovery, master/slave election, cluster fault tolerance, ASK steering/MOVED steering mechanism
3. FastDFS cluster
Knowledge points include: tracker and storage node, file upload/download test, fastdfs-nginx-module principle, fastdfs realization principle
4. K8S cluster
Knowledge points include: K8S core components, seamless connection with new applications, rapid deployment and expansion of applications, automatic deployment, management containerization
5. Nginx cluster
Knowledge points include: Nginx implementation principles, classic application scenarios, implementation mechanisms of high-availability solutions, LVS and Nginx clusters
6. MongoDB cluster
Knowledge points include: highly available MongoDB cluster solution, document management mechanism and implementation, configuration of sharded tables and shard keys, and MongoDB cluster environment deployment
7, etcd distributed registration
Knowledge points include: etcd concept and cluster environment deployment, etcd characteristics and data structure, principles and internal election mechanism etcd cluster implementation mechanism
8. rpc microservice tars
Knowledge points include: the construction and deployment of tars, the principle of microservice rpc
Four: Linux system topic
Linux is a topic that every engineer will discuss
1. Environmental programming
Knowledge points include: system file operation, all equipment access files, semaphore mmap shared memory, system time timestamp timer, network IO programming epoll mechanism, lock mechanism, mutual exclusion lock spinlock, condition variable atomic operation, process/thread data synchronization mechanism
2. System operation and maintenance
Knowledge points include: system log management and configuration, server environment construction and configuration, integrated tool installation, shell command file operation
Five: performance and testing topics
Program optimization is an essential topic for all program products
1. Understand performance optimization
Knowledge points include: capacity evaluation average QPS peak QPS single machine QPS, CDN network acceleration principle, connection pool performance optimization, performance optimization methodology
2. MySQL tuning
Knowledge points include: understanding the underlying principles of MySQL B+Tree mechanism, index optimization revealing, SQL statement optimization, detailed explanation of SQL execution mechanism
3. Test theory
Knowledge points include: in-depth analysis of TDD test-driven development, full-link stress testing experience, Nginx gray-scale release A/B Test based on IP, real traffic testing tool tcpcopy, unit test minUnit
4. Linux tuning
Knowledge points include: kernel code organization structure, system IO parameter tuning configuration, network protocol stack parameter configuration, kernel code compilation and kernel replacement
5. GCC tuning
Knowledge points include: rapid compilation project Makefile writing, compilation mechanism compilation principle
Six: Operation and maintenance statistics topic
Teamwork, product development, must have code management capabilities
1、Jenkins
Knowledge points include: building Jenkins automated deployment environment, test\pre\production multi-environment release, Jenkins integrated git to achieve automatic deployment, Jenkins multi-environment configuration, authority management and plug-in use
2. Virtualization
Knowledge points include: Xen virtualization construction and principle detailed explanation, KVM classic detailed explanation, OpenStack architecture knowledge combing, open source Linux container OpenVZ
3、Docker
Knowledge points include: Dockerfile compilation and image compilation, Docker-Compose deployment and orchestration, Docker command and image version submission, and the principle of Docker
Seven: Distributed Architecture Topics
If you use copper as a mirror, you can dress up; if you use ancient times as a mirror, you can know the rise and fall; if you use people as a mirror, you can know the gains and losses. Improve your vision from the technical architecture of the Internet's first-line products
1. Architecture combat
Knowledge points include: Sina portal load balancing solution analysis, JD massive storage hash storage engine decryption, Huya core video live broadcast architecture technology secret, JD distributed KV storage design and challenges, Ali network fault intelligent management solution, Xiaoai Dingdong Voice core technology speech synthesis actual combat, WeChat security and open cloud platform architecture exploration, the road to the construction of iqiyi big data platform
2. Architecture middleware
Knowledge points include: remote invocation of rpc from entry to proficiency, multi-process efficient communication shared MM strategy, Nginx inter-process communication analysis, why high-performance servers need memory pool, Baidu large-scale timing indicators automatic anomaly detection actual combat
3. Architecture strategy
Knowledge points include: subscriber-publisher mode to achieve high concurrency architecture, Alibaba Cloud Elasticsearch architecture analysis and performance optimization practice, two-stage submission and multi-stage submission, UDP fragmentation principle analysis and implementation methods, Reactor mode and thread pool to achieve high-concurrency services
4. Architecture principle
Knowledge points include: message-driven architecture design methodology, dynamic domain name resolution and CDN principle, Zookeeper principle, distributed lock principle, P2P decentralization principle, application layer protocol design methodology
For self-study friends, it takes a long time, the technology stack is not perfect, the structure is not structured, and the self-discipline is poor. Here to share with you a learning video link, after subscribing to listen to it for free: https://ke.qq.com/course/417774?flowToken=1013189
Eight: Challenge the limits of network performance DPDK column
1. Know DPDK
Knowledge points include: mainstream packet processing hardware platforms, interpretation of data packet processing capabilities
Potential of packet processing: DPDK accelerated network nodes, DPDK accelerated computing nodes, DPDK accelerated storage nodes, DPDK methodology
Practice review theory: Skeleton, L3fwd
2. Cache and memory
Knowledge points include: Cache system, Cache address mapping and transformation, Cache write strategy, Cache prefetch, Cache consistency, TBL and huge pages, Intel data direct DDIO, non-uniform memory access NUMA
Storage system: evolution of system architecture, memory subsystem
3. Parallel computing
Knowledge points include multi-core performance and scalability: the pursuit of performance level expansion, multi-core processor affinity, DPDK multi-threading
Instruction concurrency and data parallelism: instruction concurrency, single instruction multiple data
4. Synchronous mutual exclusion mechanism
Knowledge points include atomic operations: atomic operations on the processor, Linux kernel atomic operations, implementation and application of DPDK atomic operations
Read-write lock: Linux read-write lock core API, DPDK read-write lock implementation and application
Spin lock: disadvantages of spin lock, Linux spin lock API, DPDK spin lock implementation and application
Lock-free mechanism: Linux kernel lock-free ring buffer, DPDK lock-free ring buffer
5. Message forwarding
Knowledge points include the introduction of forwarding framework: network processing module division, DPDK run to completion model, DPDK pipeline model
Forwarding algorithm: exact matching algorithm, longest prefix matching algorithm, ACL algorithm, packet distribution
6. PCIe and packet processing I/O
Knowledge points include the perspective of PCIe transactions: abnormal interrupt mode, type polling mode, mixed interrupt polling mode
Data transmission capability on PCIe: Brust advantages of receiving and sending packets, batch processing and delay hiding, using Intel SIMD instructions to parallelize packet sending and receiving, network card DMA descriptor ring queue, packet sending and receiving—CPU and I/O
Mbuf和Mempool：Mbuf、Mempool
7. Optimized network card performance
Knowledge points include the polling mode of DPDK: asynchronous interrupt mode, asynchronous interrupt mode, mixed interrupt polling mode
NIC I/O performance optimization: the advantages of Burst receiving and sending packets, batch processing and delay hiding, parallel forwarding of Intel SIMD instructions
Platform optimization and configuration tuning: performance impact of hardware platform, performance impact of software platform
Queue length and various threshold settings: receiving queue length, sending queue length, receiving queue can release the number of descriptors threshold, sending queue can release the number of descriptors threshold, sending packet descriptor release threshold
8. Flow classification and multiple queues
Knowledge points include multiple queues: the origin of network card multiple queues, Linux kernel support for multiple queues, DPDK and multiple queues, queue allocation
Flow classification: packet type, receiver extension RSS, Flow Director, quality of service, virtualized flow classification method, flow filtering
Use of flow classification technology: DPDK combined with network card Flow Director function, DPDK combined with network card virtualization and cloud filter function, reconfigurable matching table
9. Intel hardware features
Knowledge points include: network card hardware uninstall function, DPDK software interface, hardware and software function realization, fragment group package uninstall
Calculation and update function offloading: VLAN hardware offloading, IEEE1588 hardware offloading function, IP TCP/UDP/SCTP checksum hardware offloading function, Tunnel hardware offloading function
10. IO virtualization on X86 platform
Knowledge points include: packet sending and receiving process under transparent transmission of PCIe network card, virtual configuration of I/O transparent transmission
X86 platform virtualization overview: CPU virtualization, memory virtualization, I/O virtualization
I/O transparent transmission virtualization: Intel VT-d, PCI SR-IOV
11. Paravirtualized Virtio
Knowledge points include: Virtio use
Virtio specifications and principles: device configuration, virtual queue configuration, device use
Virtio network device driver design: Virtio network device Linux kernel driver design, based on DPDK user space Virtio network device driver design
12. Vhost optimization solution to accelerate packet processing
Knowledge points include the evolution and principle of vhost: Qemu and Virtio-net, Linux kernel mode vhost-net, user mode vhost
User mode vhost design based on DPDK: message mechanism, address translation and virtual machine memory mapping, vhost feature negotiation, virtio-net device management, checksum and TSO function uninstallation in vhost
DPDK vhost programming example: message sending and receiving interface, using DPDK vhost lib, using DPDK vhost PMD
13. DPDK and network function virtualization
Knowledge points include: network function virtualization, OPNFV and DPDK, NFV deployment,
Evaluation of VNF's own characteristics: performance analysis methodology, performance optimization ideas
VNF design: VNF virtual network interface selection, IVSHMEM shared memory PCI device, network card polling and mixed interrupt polling mode selection, hardware acceleration function consideration, service quality assurance
Case analysis and business case: Virtual BRAS, Brocade vRouter 5600
14. DPDK performance acceleration in Open vSwitch
Knowledge points include: virtual switch, OVS
DPDK accelerated OVS: OVS data path, DPDK accelerated data path, DPDK accelerated OVS performance
15. Storage software optimization based on DPDK
Knowledge points include: Ethernet-based storage systems, optimization of Ethernet storage systems
SPDK introduction: user-mode TCP/IP stack based on DPDK, user-mode storage driver, implementation and performance of iSCSI target in SPDK
Nine, comprehensive analysis of the Linux kernel code column
1. Process management and scheduling
Knowledge points include: process priority, process life cycle
Process representation: process type, namespace, process ID number, process relationship
System adjustments related to process management: process copy, kernel thread, start new process, exit process
Implementation of the scheduler: overview, data structure, processing priority, core scheduler
Completely fair scheduling: data structure, CFS operation, queue operation, select the next process, process periodic scheduler, wake up preemption, process new process
Real-time scheduling: nature, data structure, scheduler operation
Scheduler enhancements: SMP scheduler, scheduling domain and control group, kernel preemption and low-latency related work
2. Memory management
Knowledge points include (N) Memory organization in UMA model: overview, data structure
Page table: data structure, creation and operation of page table entries
Initialize memory management: establish data structures, architecture-specific settings, memory management during the startup process
Physical memory management: the structure of the partner system, avoiding fragmentation, initializing the memory domain and node data structure, allocator API, allocating pages, releasing pages, allocation of discontinuous pages in the kernel, kernel mapping
Slab allocator: alternative allocator, memory management in the kernel, the principle and implementation of slab allocation, general cache
3. Process virtual memory
Knowledge points include: the principle of memory mapping
Process virtual address space: layout and establishment of the process address space
Data structure: tree and linked list, representation of virtual memory area, priority search tree
Operations on regions: associate virtual addresses to regions, merge regions, insert regions, create regions
Memory mapping: create mapping, delete mapping, non-linear mapping
Reverse mapping: data structure, establishment of reverse mapping, use of reverse mapping
Correction of abnormal user space page faults: on-demand allocation/paging, anonymous pages, copy-on-write, acquisition of non-linear mapping
4. Device driver
Knowledge points include access to devices: device files, character devices, block devices and other devices, using ioctl for device addressing, representation of master and slave device numbers, and registration
Associated with the file system: device file members in the inode, standard file operations, standard operations for character devices, and standard operations for block devices
Character device operations: represent character devices, open device files, read and write operations
Block device operations: block device representation, data structure, adding disks and partitions to the system, opening block device files, request structure, BIO, submit request, I/O scheduling, realization of ioctl
Resource allocation: resource management, I/O memory, I/O port
Bus system: universal driver model, PCI bus, USB
5. Virtual file system
Knowledge points include: file system type
Common file model: inode, link, programming interface, file as a common interface
proc file system: /proc content, data structure, initialization, loading proc file system, management of /proc data items, process-related information, system control mechanism
Simple file system: sequential file, file system written with libfs, debug file system, pseudo file system
sysfs: data structure, mount file system, file and directory operations, add content to sysfs
The structure of VFS: structure overview, inode, process-specific information, file operations, directory entry cache
Processing VFS objects: file system operations, file operations
Standard functions: general read routines, failure mechanisms, permission checks
6. System call
Knowledge points include the basis of system programming: tracking system calls, supported standards, and restarting system calls
Interrupt: interrupt type, hardware IRQ, processing interrupt, data structure, interrupt current processing, initialization and distribution IRQ, processing IRQ
Soft interrupt: open soft interrupt processing, soft interrupt daemon
tasklet: create tasklet, register tasklet, execute tasklet
Waiting queue and completion volume: waiting queue, completion volume, work queue
Implementation of system calls: structure of system calls, access to user space, tracking system calls
7. Page recycling and page swapping
Knowledge points include: exchange tokens, data write-back
Overview: page can be swapped out, page thrashing, page swap algorithm
Page recycling and page swapping in the Linux kernel: organization of the swap area, checking memory usage, selecting the page to be swapped out, handling page faults, and reducing the kernel cache
Manage exchange areas: data structure, create exchange areas, activate exchange areas
Swap cache: identify swapped out pages, swap cache structure, add new pages, search for a page
Page recycling: overview, data structure, determining the activity level of pages, shrinking memory domains, isolating LRU pages and centralized recycling, shrinking active page linked lists, recycling inactive pages
Handling exchange page missing exceptions: swap in pages, read data, exchange pre-reading
Initiate memory reclamation: kswapd performs periodic memory reclamation and swaps out pages when there is severe memory shortage
Shrink other caches: data structure, register and delete shrinkers, shrink cache
Share the overall idea map at the end of the article, the clear version can be obtained by +qun720209036
