In the past month and a half, I participated in the development of the company's project from 0 to 1 as the back-end development team leader for the first time, and I will record this development experience.
1. Background introduction
First some background. The company I work for is engaged in smart community-related business, and the project developed is a system upgrade tool to facilitate company implementation colleagues to install and upgrade the system.
There are four people involved in back-end development, including me. Among the other three colleagues, one is a fresh graduate and two work in big data. According to the company's technical plan, the development pace of internal projects should be short and fast, and should be completed in Python language, while all external projects should be completed in Java language.
The project goes through the normal development life cycle, including requirements collection, product design, system design, detailed design, coding, testing and other processes. The detailed design is a detailed design for the interface. It took a total of 3 days to complete, and 45 function points were designed, including 40 interfaces and 5 initialization preparations. The coding time is planned to be 3 weeks. Finally, the relevant development was completed within the time point.
I play the role of team leader during this development process. My main tasks include:
(1) Construction of project framework. This development is a process from 0 to 1. Before this, there was no framework for Python projects.
(2) Implementation of key technologies. Including common interfaces and complex technical points.
(3) Task allocation. All interfaces are assigned to designated members according to the workload to complete the most interface development.
2. Project framework construction
There are actually not many commonly used project frameworks for web development in Python. I have three candidates:
Django's front-end and back-end non-separation framework, Flask's easiest-to-use framework, and FastAPI asynchronous high-performance framework.
Comparing these three frameworks, I chose Flask from three perspectives: business logic, company technology stack, and complexity.
Business logic
The business logic has no special requirements for performance. It just calls the operation and maintenance ansible script through the interface. There are no high concurrency, calculation-intensive tasks, etc., so all three can be satisfied.
technology stack
The company's technology stack is separated from the front and back ends, so a framework like Django that does not separate the front and back ends is not suitable, although Django can also do pure back-end development (anti-blocking).
Complexity
In terms of complexity, Flask is definitely the simplest. Django is known to be large and comprehensive, but its configuration is complex. FastAPI is an asynchronous framework and requires learning asynchronous programming. Although it is very smooth when used as a synchronous framework, the learning cost needs to increase a lot. The other three colleagues have never done Python projects, so they try to reduce the learning cost.
After measuring these three directions, the Flask framework was finally determined, paired with the peewee orm database framework. Core technologies include:
(1) web framework Flask
(2) Database ORM framework peewee
(3) Database sqlite
(4) Operation and maintenance script execution module subprocess
(5) WSGI service Gunicorn
(6) Code checking tool pre-commit
Before coding, I had prepared a complete project framework and written a demo of the database CRUD interface. During the subsequent development process, my colleagues imitated the relevant interfaces, which improved development efficiency to a certain extent.
3. Implementation of key technologies
When you lead a team to develop and lead a member to do a Python project for the first time, you naturally have to assign challenging tasks to yourself. Select three representative explanations on the implementation of key technologies. The three are: general interface for system command execution, streaming logs, and sqlite multi-thread writing problem solving.
System command common interface
The project is mainly used for the installation and upgrade of other systems developed by the company, so it is necessary to call relevant scripts written by operation and maintenance personnel using ansible. The format of the called ansible script is as follows:
ansible-playbook 03.mysql.yml
ansible-playbook 08.zk.ymlYou need to execute the above command in the specified path. During the detailed design stage, we knew that we needed to use Python to call system commands, so we asked our fresh graduate colleagues to investigate the subprocess module and output relevant documents. Firstly, it gives newcomers a learning direction, and secondly, it takes this opportunity to become familiar with the technologies required for the project.
During the development phase, based on the understanding of relevant modules, the development of common interfaces was completed. When writing a universal interface, be sure not to change it overnight, as the code that relies on it must also change accordingly. It's acceptable once or twice, but if you do it more often, you'll probably be asked to greet your ancestors. Therefore, the implementation level of the interface is not only completed, but also called by myself, and only after confirming that there are no problems is it declared complete.
Be patient in debugging until it is completed, and then tell other developers in the group until there are no problems. A large number of calls to this command are required to execute scripts throughout the system, and they are all completed smoothly in the end, with no bugs caused by the interface.
Streaming logs
According to the design of the product, when a component is installed, a log needs to be displayed on the web page, and the log format must be the same as the installation log in the terminal, that is, scrolling and printing line by line. The product's requirement for the log is that the logs are displayed in full scrolling mode, and the page must be refreshed to be able to display the logs in full again. In order to realize this function, three options were investigated:
1. Regular refresh. Disadvantages: There are tens of thousands of lines in the log. Every time the entire log is read to the front end, the front end will freeze, and the printing will not be continuous, which makes the experience not good.
2. websocket. Active push from the backend to the frontend can be completed, but refreshing the page will not start the push from the beginning.
3. Streaming response. You can cut large files into small pieces and pass them to the front end in batches. When the page is refreshed, it will be pushed from the beginning again, which meets the requirements.
After comparison, we finally used streaming response, which is the response method of ChatGPT. But there is a question that needs to be solved: when will the push end? Because the installation is writing logs to the file, and the streaming log is reading the logs, if the logs have been read and the installation has not ended, then it is definitely necessary to wait instead of stopping the response.
The solution is to insert a mark character after the installation is completed. When the streaming log reads the mark character, it indicates that it is over. If the mark character is not read, it will wait. The core code is as follows:
def log_flow():
query = request.values
log_path = query["log_path"]
def generate():
with open(log_path, "r") as f:
while True:
chunk = f.read(800)
if not chunk or chunk.isspace():
time.sleep(0.1)
if chunk == 800 * "-":
break
content = json.dumps({"content": chunk})
yield f"event: message\ndata: {content}\n\n"
time.sleep(0.05)
return Flask_response(generate(), mimetype="text/event-stream"Effect:
sqlite3 multi-threaded writing problem
In terms of database storage, the leader decided to use sqlite3. We also tried to use MySQL based on reasonable reasons, but it didn't work. The leader said that the project only requires a lightweight database, and sqlite3 is lightweight, so it is suitable. And it has been used very maturely in other projects. Well, since the leader insisted, I had no choice but to bite the bullet.
There were no problems at first, but problems were discovered in the middle and late stages of project development. The interface often reported errors:
File "/home/ljk/.virtualenvs/idt_dev/lib/Python3.8/site-packages/peewee.py", line 3246, in execute_sql
cursor.execute(sql, params or ())
File "/home/ljk/.virtualenvs/idt_dev/lib/Python3.8/site-packages/peewee.py", line 3014, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/home/ljk/.virtualenvs/idt_dev/lib/Python3.8/site-packages/peewee.py", line 192, in reraise
raise value.with_traceback(tb)
File "/home/ljk/.virtualenvs/idt_dev/lib/Python3.8/site-packages/peewee.py", line 3246, in execute_sql
cursor.execute(sql, params or ())
peewee.OperationalError: database is lockedAfter querying, I found that sqlite3 does not support multi-threaded writing. SQLite3 supports transactions, which are accomplished using library locks. When a write starts, the entire database is locked, and an error will be reported if there is another write operation.
First of all, this problem is technically difficult to solve. This is the architectural design of sqlite3. Can it still support multi-threaded writing? It can only be solved through business logic. After a meeting and discussion, several solutions were reached:
- Branch library. Divide write operations into different libraries to complete. Since the write operation will lock the library, separate the libraries into different libraries to avoid the problem of library lock.
- Global write queue. Put all writes into a message queue and turn multi-threaded writing into serial writing.
- Global write flag. Before starting all write operations, determine whether there is a writable flag, and write if it can be written. Otherwise, the interface returns to tell the front-end database that it is busy.
After voting, everyone unanimously decided to use the third way to implement it, which has the least technical difficulty and the least intrusive code. Because the third option was proposed by me, I will ultimately complete it. For the specific process, please refer to another document "Analysis of SQLite Lock Table Problems in Peewee Operation".
In the end, the problem was solved. Although it was not elegant, it was partially the optimal solution in terms of current time cost and risk control. In the future, we will investigate the write queue implementation of sqliteQueueDatabase provided by peewee, an ORM framework.
Excerpted from peewee extension plug-in playhouse:
SqliteQueueDatabase is available as a regular SqliteDatabase. If you want simple read and write from multi-threaded access to sqlite database.
SQLite only allows one connection to write to the database at any given time. Therefore, if you have a multi-threaded application (such as a web server) that needs to write to a database, you may occasionally see errors when one or more threads trying to write cannot acquire the lock.
SqliteQueueDatabase is designed to simplify operations by sending all write queries over a long-lived connection. The benefit is that you can see multiple threads writing to the database without conflicts or timeouts. However, the disadvantage is that you cannot issue a write transaction containing multiple queries - essentially, all write operations run in autocommit mode.
4. Personal feelings
It was my first time leading a team to develop, and I realized a lot of things.
The hero of the project
The first is to truly understand what protagonist consciousness is. Various articles say that you need to have a sense of ownership of the project to grow faster. I feel like you can only feel this awareness if you stand in such a position.
The first person responsible for the progress of the project is you. Any problems, big or small, that arise in the project will come to you. The leadership meeting will ask questions about the progress of the project, are there any obstructions, and can it be completed on schedule? Team members will ask if this verification framework is OK? Is there any problem with this syntax? The tester will ask you who should give this bug? Therefore, you must know this project thoroughly, from a configuration item of the code to the control of project progress. Any problems during the development process must be addressed promptly. The team leader is just a brick, moving it wherever it is needed~
Team harmony
Let’s talk about team harmony. In the past, when I was doing a small development, I just had to complete my own tasks. Does the team atmosphere affect the speed of writing code? Leading a team to develop is different. There are colleagues with various characteristics in the team, including those who work hard without reporting progress, those who are capable but have a bad temper, and those who have a good temper and make slow progress. In short, people of all kinds of personalities exist.
During this development, I encountered a person who was very capable but had a bad temper. When he saw minor technical problems, he would directly start a group chat. Who is not a passionate young man? As you can imagine when encountering this situation for the first time. The leaders had to talk to us once or twice about this, and even alarmed most of the department leaders. The team atmosphere was particularly bad during that time, and no one spoke. I didn’t dare to say anything more, for fear that the atmosphere would get worse and the project wouldn’t be completed as scheduled. By then, it would become my problem if it wasn’t my problem. So I can only choose to endure it and avoid differences as much as possible. Fortunately, the leader's talk played a big role in comforting and criticizing when needed. Later, there was no verbal conflict and the project was successfully completed on schedule.
5. Summary
Leading a team on a project for the first time was a challenge and improvement for me. From a technical perspective, it allows me to look at problems from a higher perspective when faced with technology selection in the future; from a personal perspective, this is a rare opportunity for me to be in charge of the development team and connect with the testing team, front-end team, operation and maintenance team, etc. This is an exercise for my communication.
Finally, I hope to do better next time so that all team members can make some progress.
Development tool recommendations
The principle is to concretize certain recurring scenes and processes in the development process into components, APIs, and database interfaces to avoid reinventing the wheel. This greatly improves programmers' productivity.
Official website: www.jnpfsoft.com/?csdn . If you have free time, you can expand your knowledge.
This is a simple, cross-platform rapid development framework based on Java Boot/.Net Core. The front-end and back-end encapsulate thousands of common classes for easy expansion; it integrates a code generator to support front-end and front-end business code generation to meet rapid development and improve work efficiency; the framework integrates various commonly used classes such as forms, reports, charts, and large screens. Demo is easy to use directly; the back-end framework supports Vue2 and Vue3.
In order to support application development with higher technical requirements, from database modeling, Web API construction to page design, there is almost no difference from traditional software development. However, the low-code visualization mode reduces the repetitive labor of building "add, delete, modify and check" functions .