Pay attention to the public account [Exploration of High Performance Architecture], reply [pdf] in the background, and get free classic books necessary for computers
Hello everyone, I am Yule.
Graduated for more than ten years, has been engaged in back-end work, worked on network transmission projects, was responsible for the recommendation engine, and is currently responsible for the advertising engine and advertising settlement engine. The main languages used are c/c++, python and shell as auxiliary development.
Welcome to communicate and make progress together.
written in front
At a certain time on a certain day, I suddenly realized that I have graduated and worked for ten years, and I should write something to record my ten years, which can be regarded as a psychological comfort for myself, and since I entered the workplace, Summary of the first decade.
college life
I went to university in 2003. Before I went to university, I had no contact with computers at all. The university newspaper volunteers were also ignorant. At that time, the two most popular majors were computer science and biological engineering. What is often said on TV is: the 21st century is the century of biological science. As for me, because I have never touched a computer, I was rather curious. When I filled out the application form, I wrote the most introverted industry major: computer science and technology.
So, one day in September, I took a train that traveled nearly half of China, and came to a certain city in the north, where I started my so-called college life. I still clearly remember that the first time I took a computer class, I turned on the computer in the computer room, double-clicked to open My Network Places, entered the URL, and the first song I heard was Liu Ruoying's "Love You So Much", so many years later , every time I hear the familiar melody of this song, I will recall the scene of my first computer class in my freshman year.
At that time, I had just escaped the oppressive high school life, like a bird with wings, flying freely in the air, adhering to the learning mentality of "Long live sixty, one more waste", muddling along, both basic and professional courses Passing with an average score, it can be regarded as leaving no regrets.
In this way, I went through the ignorance of the freshman year, the hesitation of the sophomore year, and the sinking of the junior year. It was not until the third year that I began to think about my future, followed my heart, and chose to take the postgraduate entrance examination.
postgraduate life
Muming came to a school in East China that is famous for its hard work, and started her introverted life. After taking a lot of professional courses, it can be regarded as a punishment for my undergraduate period. Every day at three o'clock in the first-line life, dormitory, cafeteria, library, began to learn my own professional skills. The first programming book I bought was c++ primer. I borrowed c++ university tutorials in the library and started to learn programming. With extensive exposure to professional books, I started to understand the underlying principles more deeply and explore the c++ model in depth. Effective C++, more effective C++, C++ Meditations, etc., many books were started at that time, which can be regarded as laying a solid foundation for myself to enter the industry later.
Internship life
At that time, I began to have a sense of crisis about my employment, and the students around me got internship offers from Alibaba Cloud, Xunfei and other companies. Finally, when I was about to collapse, I got an internship offer from a certain institute of the Chinese Academy of Sciences. I remember that the first sentence of my roommate was: Congratulations, the daughter-in-law who has been married for many years has become a mother-in-law.
Came to Beijing, lived in the dormitory arranged by the institute, followed the mentor to do a national 863 project, related to ddos attack. Because I am an intern, I have been exposed to some optimization classes, such as memory pools.
At the same time as an internship in the institute, I started to contact the csdn website. Except for the project, I spent all day in the c++ section to answer questions. At the beginning, the points were very high and the ranking was also very good. However, due to other reasons, after work, I no longer log in to the csdn website. It’s a forum, it’s not a small regret, I can still learn a lot of things in it, I still remember that there is a group with excellent technology (transliteration pancake hall), I was lucky enough to join, but I still quit later.
With the end of the project in the institute, I chose to go to another large foreign company for internship. The company is in Xierqi (you should be able to guess which one it is). The project team is to make a complete set of office systems under Linux, similar to Windows The following word is based on openoffice. The entire project has millions of lines, and it takes two or three hours to compile it. I am responsible for solving memory leaks. What needs to be mentioned here is that the mentor at that time was a big cow. Basically, the entire company would go to him for problems, so that I could A deeper understanding of technology.
New to the workplace
Enter the traditional industry
The first company is a traditional industry. I was fortunate enough to enter the company's most powerful technology research institute to do network-related research and development. That is, in this company, I started to study TCP and read the detailed explanation of TCP/IP over and over again. , If you don’t understand, check the information, and even download the source code for analysis and research. It was also in this company that I first came into contact with the Linux system.
Since the company's business is oriented to state-owned enterprises and mainly runs on the Windows system, the first requirement is stability, so the update iteration is very slow, and sometimes a function has not been optimized for several years. Later, as the Linux system was used more and more in China, the business of the department also needed to run on the Linux system. At that time, there were two options, one was to rewrite a code on the Linux system, and the other was to use the existing code to Completely transplanted to the Linux system, that is, the same code can be used in both Windows and Linux systems. In the end, for the sake of efficiency, I chose the second solution.
Everyone knows that although C++ is a cross-platform language, the API name and underlying implementation depend on the system, such as the WaitForSingleObject function, which is std::lock under Linux gcc, which requires the same function, such as WaitForSingleObject , the system used under Windows has its own API, while under Linux it is a function redeveloped, and its function is similar to that under Windows. At the beginning, I was responsible for completing this part. The coredump appeared for the first time, and it was the first time to use gdb for debugging.
onboarding internet
In those few years, when the official Internet industry started to catch fire, I was eager to move in my heart, so I entered an Internet industry after several ups and downs, which was my first contact with the Internet.
When I first joined the company, I was really not used to the fast pace. It often took a few days for a request to go online, which was absolutely impossible in the traditional industry before. After eating half of it, I ran up to solve it. . . , I learned a lot from this company, became more and more proficient in the business, and began to be responsible for the entire recommendation engine, laying the foundation for later exposure to advertising.
contact advertising
Due to the bottleneck in the company's business development and some other reasons, it began to look at opportunities again. Since the recommendation engine and the advertising engine are similar in structure and function (index->filter->rank, etc.). So fortunate to enter the advertising industry.
This company has had a great influence on me. If the last company gave me the opportunity to enter the Internet, then in this company, I began to think about how to make the service performance better and how to improve the system design. Being forward-looking, it can be regarded as gradually getting in touch with the architecture, starting to understand the entire business from the architecture, and deducing the rationality of the architecture from the business.
As we all know, advertisements have high requirements on rt (Response Time). Every time rt increases by a few milliseconds, the conversion rate of advertisement display will increase. Therefore, for services that request tens of billions of times a day, the business processing performance is very high. The performance of the service will affect the entire advertising link. for example:
- The set of ad candidates is too large, resulting in longer processing times
- A piece of code consumes too much CPU
- Too many useless parameters on the transmission chain
- other
The memory is relatively deep, and a DMP service is implemented. For performance reasons, all operations are performed in memory. It uses a lot of memory allocation and release. After going online, I found a phenomenon, that is, the memory utilization rate has always been high. From the surface phenomenon, it means that the memory has not been released. Even if the program shows that delete or clear is called, the memory usage is still high. In order to solve this problem, various memory detection tools were used, and logs were added (both applying for memory and releasing memory), and found that there was still no problem, so I went to investigate the underlying library to study the principle of memory allocation. By studying the source code , investigating the reasons for new and or malloc, and finally found that in the underlying API, even if delete or free is called, the memory will not be released to the operating system immediately, but will be placed in the buffer (this is the memory usage is relatively high The reason) will be reclaimed by the operating system after a period of time. If you want to be reclaimed by the operating system immediately, you need malloc_trim(0).
Still the same DMP service, due to its large memory usage, the performance is not very high, a large number of Misscache, and finally, by changing the page size in the system from the default 4k to 32k, this problem is solved (this is because the page is too small If not, it will lead to a large number of page fault interrupts, for details, please refer to <In-depth understanding of the operating system>).
programming ability
As programmers, programming ability is the foundation of our life.
As far as my own feelings are concerned, I think there are several parts to the growth of programming ability.
primary
Basic programming ability, that is, you can use it . In other words, using various system APIs can achieve our goals.
Programming starts with learning the grammar of the language, such as if/else while, etc. As a junior programmer, it is recommended to use entry-level books when getting started, instead of reading some more advanced books. (For c++, <c++ primer> is not suitable for entry-level readers, but for people with a certain basic knowledge of c++. The name of this book is too tempting).
In addition to reading documents, programming is a super practical job, so you must write more codes. Only in this way can you really become proficient. There is a saying "There is nothing else, only hands are familiar", which is very suitable for entry-level programmers. As an entry-level programmer, try not to use the IDE. This will make you dependent on the IDE and make handwriting difficult. I hope that all API names can be typed out as much as possible. Over time, I will become more and more proficient.
In the primary stage, the core is to write more, study hard and practice hard. This is very important. Only in this way can we lay a solid foundation for the future.
intermediate
The primary goal is to be able to proficiently use the programming language to implement functions, and the intermediate goal is to improve the ability to analyze and solve problems .
As a programmer, it is very normal to have problems in the process of writing code, but how to effectively and efficiently troubleshoot problems is the biggest gap in programming ability between individual programmers. Recognition from colleagues in other departments.
In terms of the ability to check problems, the first thing to master is some basic debugging skills and easy-to-use debugging tools, such as commonly used gdb, gperf, btrace, etc. Proficiency in debugging tools is very beneficial to solving problems. Sometimes the gap in the ability of people when checking problems may be just because others know one more tool than you. In addition to debugging skills and tools, the higher level of checking problems will have a very big relationship with the advanced stage of programming ability. , That is to understand the principle, a programmer who understands the principle has a clear gap in the level of checking the problem. Try to write some programs that will cause problems for yourself, be more active to see how other people check the problems, and be more active to participate in the troubleshooting. Many people who have strong ability to check the problems in the end are mostly just because "there is no one else, but they are familiar with it." you".
In addition to checking problems, more powerful programmers will avoid problems very well in the process of writing code. They will know what to do in this area so as not to fall into the pit, and what problems will arise in this area. What kind of conditions or logs etc.
At the intermediate stage, it is recommended that you take the initiative to analyze and solve problems, and become an excellent programmer who can write high-quality code and troubleshoot problems effectively.
advanced
For programmers with advanced programming ability, a standard is to understand the implementation principle of the underlying API , know what it is, and know why it is so.
In the process of learning the underlying implementation and troubleshooting problems, it is very important to understand more and more how the programming language works. The most direct way to learn the underlying implementation is to look at the source code, but this is difficult for many people, so you can try to start with documents or some books (<STL source code analysis>), and find the corresponding source code to analyze while reading the book, so that It can achieve twice the result with half the effort.
Some people have said that language is just a tool, to solve and analyze problems, don't be limited to language, I very much agree with this statement. For example, although c++ has excellent performance, its implementation process is cumbersome and there will be many pitfalls. For some fierce and fast needs, such as analyzing online logs, you can use python or shell to solve them.
Architecture capability
As more and more codes are written, inadvertently, when writing codes, I start to consider whether the performance is optimal, whether the architecture is reasonable, and whether it can be flexibly expanded in the future.
Completely building a system was five years ago, that is, when I worked for five years, and I joined the current company at that time. At the beginning of commercialization, there is nothing, and everything has to be built from scratch. For example, when a flow comes, how to request various business lines, and when an advertising order comes, if it can quickly enter the advertising system. Then it is necessary to allocate modules reasonably.
As more and more codes are written, more and more problems are solved, more and more difficult, and more and more modules are designed. In a subtle way, the architectural ability will be developed. When designing a system, start Be forward-looking, consider decoupling, and consider expansion.
Architectural abilities cannot be learned at once, more need to come from practice, to practice, and over time, this ability will be improved.
experience
team spirit
In a company with complete processes, requirements review, development, testing, and online processes are indispensable. The lack of each link may lead to online failures. Therefore, the completion of a requirement requires the cooperation of all departments.
There are problems online, and everyone works together to solve them. This is the process of growth.
don't show off
Code needs to be inherited. Everyone doesn't want their code to be scolded by those who take over, so the code must be concise, clear, and clear. Don't deliberately use obscure programming skills to show off some technical strength. I believe that it will not be long before not only others, but even I may not be able to understand this piece of code.
Reasonable structure
There are tens of thousands of architectural solutions, and the most important thing is to choose a suitable one. Don't go to architecture for the sake of architecture, but consider efficiency and functional requirements.
Achieve a balance between architecture and development efficiency, and then look backward to see if the existing architecture design can meet the needs.
Both offense and defense
The so-called attack, that is, attack, can also be understood as a siege. That is to achieve product goals and meet functional requirements; all guards, that is, guards, can also be understood as guarding the city. That is, online problems can be guarded and quickly resolved.
As the saying goes, it is easy to attack a city, but difficult to defend it. Online bugs can never be solved, you need to be patient, and you need to have the ability to both attack and defend.
code review
Codereview is essential, which requires teamwork and can solve many problems in the early stage of launch.
But there are also disadvantages, which may lead to conflicts among everyone. For example, when a function is implemented, A says that it can be implemented in this way, and B says that it has better performance. Over time, it will lead to conflicts between teams. This requires finding a balance point, and it is not necessary to mention dispensable opinions. . .
code comment
Code is the best comment.
This is the ultimate goal, but many people can't reach it, including me of course. So as many comments as possible in the code, the reason:
- for future maintenance
- Make it easy for others to take over
Especially for functions with complex functions, you can write them in 1, 2, 3, etc., and when necessary, write them on CF, and add URL links to the function comments.
code structure
The code structure must be clear, concise and clear. You can know its function by looking at the function description, and you can know what it does through the class name.
What the code structure shows is actually a modular logic idea of the program. The structure is simple and clear, which also shows that the logic of the program is simple.
code style
Programmers work in teams, and everyone has their own different programming styles. If there is no unified style, the entire code will be messy.
The Google code specification that is now used in the industry is also recommended to everyone.
development efficiency
C++ programmers have a characteristic, that is, they don't like to use IDE, which can be regarded as their unique pride. They like to use vim (I am), and all code development is operated with vim, which seems to be more B-style.
beyond technology
Learning is like rowing upstream.
Programmers are a process of continuous self-learning and self-improvement. They need to keep learning and learn new knowledge in order not to be eliminated by technology.
Don't forget the original intention, stick to the end
I interviewed too many people. Many of these people graduated from prestigious schools and have impressive resumes, but they all have a characteristic, that is, they change careers. Some switch from technology to products, and some from technology to testing, and gradually lose their competitiveness in the industry.
Technology is our foundation, and we must not give up on this. Even if we have achieved management, please get in touch with the code as much as possible.
goal, ideal
The ideal is a dream, which may not be realized, but it is the belief that supports the inner abundance. Only with ideals can we set goals, otherwise the goals will always change, and there will be a lot of anxiety in my heart.
Share with you.