linux command often test
Introduction
Early adopters LAMP / LNMP architecture
Linux + Apache(Nginx) + Mysql + Php/Python
Now micro-architecture service container technology
Why learn linux?
Most enterprise applications run on linux server
Proficiency in linux server
Learn linux operating principle and common tools
Learn to view files, processes, memory-related commands for debugging and troubleshooting
How to check linux command usage
man
info
- help
- eg： pip --help
man alternative tool tldr, Pip install tldr
File / directory operation command
Common file manipulation tools
chown/chmod/chgrp
ls / rm / cd / cp / mv / touch / rename / ln (soft links and hard links), etc.
- locate / find / grep find and locate search
- eg：find . -name "*.pyc" -delete
File Viewer
Or the log file viewer
Editor vi / nano
cat / head / tail to view the file
more / less interactive view files
Process operations command
Common tools to master the process of operation
ps check process
kill kill the process
top / htop monitoring process
Memory operation command
Master the common memory operations tools
- free to view available memory
- Understand the specific meaning of each column
- Troubleshoot memory leaks
Network operations command
Master common network tools
ifconfig see card information
lsof / netstat to view port information
ssh / scp remote login / copy. tcpdump packet capture
User / Group Operation Command
Master the common user and group operations
useradd/usermod
groupadd/groupmod
to sum up:
- Multi-purpose to become familiar
Operating system threads and processes often questions
The difference between threads and processes
Processes and threads comparison
Encapsulation process is to run the program, resource scheduling and allocation system is the basic unit
A thread is subtasks process, cpu scheduling and allocation basic unit, in the process to achieve concurrent
A process can contain multiple threads, there is a thread-dependent process, and share process memory
Thread Safety
py which operations are thread-safe?
Operation is safe to use in a multithreaded environment, to get the correct results
Like thread thread-safe operation is performed sequentially rather than concurrently executing (i + = 1)
Generally if related to the write operation needs to consider how to make multiple threads access the data security
Thread synchronization mode
Learn thread synchronization way, how to ensure the security thread
Mutex (lock): By mutual exclusion mechanism to prevent multiple threads simultaneously access a common resource
Control the number of threads the same time multiple threads access the same resource: semaphore (Semphare)
Event (signal): holding a plurality of thread synchronization by means of notice
Way of inter-process communication
Inter-Process Communication between processes or data transfer signal
Pipeline / anonymous pipes / named pipes (pipe)
Signal (signal): such as Ctrl + c is generated using the user program termination signal SIGINT
Message Queue (Message)
Shared Memory (share memory)
Semaphore (Semaphore)
Socket (socket): The most common way is to use the web application in this way
py using multiple threads
threading module
threading.Thread class to create a thread
start () method to start a thread
Can () wait for the end of the thread with a join
import threading
lock = threading.Lock()
n = [0]
def foo():
with lock:
n[0] = n[0] + 1
n[0] = n[0] + 1
threads = []
for i in range(5000):
t = threading.Thread(target=foo)
threads.append(t)
for t in threads:
t.start()
print(n)py how to use multiple processes
py has GIL cpu-intensive programs can be achieved with multi-process
multiprocessing multiprocessing module
- Multiprocessing.Process class implements a multi-process
- Generally used in cpu-intensive program, to avoid affecting the GIL
#多进程
import multiprocessing
def fib(n):
"""worker function"""
if n <= 1:
return 1
return fib(n-1) + fib(n-2)
if __name__ == '__main__':
jobs = []
for i in range(10, 20):
p = multiprocessing.Process(target=fib, args=(i,))
jobs.append(p)
p.start()Operating system memory management mechanism common questions
Modern programming languages generally have garbage collection mechanism
What is the paging mechanism
In order to efficiently manage the operating system memory, reducing fragmentation
Separation of logical and physical addresses of the memory allocation management scheme
Dividing the logical address of the program fixed-size pages (Page) is divided into the same physical address size frame (Frame)
Logical addresses and physical addresses corresponding to the page table by
What is segmentation mechanism
Segment is to meet the needs of some logic code data sharing, data protection, dynamic link, etc.
Segment table mapping from logical addresses and physical addresses relation through the interior of each segment is a contiguous memory allocation, and between paragraphs is assigned to a discrete
The difference between paging and segmentation
Paging vs segments
Page is a discrete distribution mechanism for memory utilization point of view put forward segment is out of the user's perspective, the management mechanism for data protection, data isolation purposes, the page size is fixed, the operating system determines the segment size uncertain The user program decision
What is virtual memory
By being part of unused memory on the hard drive information into the principle of locality, when the program runs only part of the necessary information into memory
Memory temporary unwanted content on the hard disk, the system seems to offer much larger than the actual memory capacity, called virtual memory
What is the memory jitter (bumps)
The nature of frequent scheduling behavior of page
Frequent scheduling page, the process continues to generate a page fault replacement pages, and continue to need this page again
Running too many programs, the page replacement policy is not good, terminate the process or increase the physical memory
Py principle of garbage collection?
The main reference count (disadvantages: circular reference can not be resolved)
Tag removal and introduction generational recovery solution to resolve the reference count of the reference
+ Labeled reference count-based removal and recovery supplemented generational
Reference counting can not solve the problem of circular references
Two objects refer to each other can not be cleared
Clear marks (Mark and Sweep)
- Generational recovery
- Three Generations of 012 points each generation using a doubly linked list
Reclamation Threshold tag
Thread exercises
Writing multithreaded reptiles
How to use the threading module py
Threading the use py module to complete a multi-threaded reptile
Requirement 1: maximum number of threads can be passed to the class list of URLs to crawl and needs
Requirement 2: This class may provide a method of treatment response by way of inheritance