Python craftsman | Summary of the main points of the book

foreword

I have learned Python several times before and after, and learned many usages in Python, but they are all tutorials such as quick start, based on the mode of "what to do --> how to do". In the process of programming, I often feel confused, so I decided to learn and understand Python knowledge more systematically.

But what to learn, videos, or books? After thinking about it, I chose the book. The knowledge system of the book is often more systematic and detailed. I opened Douban to see which books had higher ratings. I picked out a few of them: "Python Programming from Introduction to Practice", "Smooth Python", "Python Craftsman Cases, Skills and Engineering Practice", "Effective Python", compared their directories, and finally chose this one Python artisan.

The reading experience is not bad. If the Python language is an axe, then some introductory books are like product manuals given when you buy an axe, and this book is like Li Kui's "Black Wind Axe Method". ( I can't think of a suitable analogy, so let's do it )

I read this book for the first time in 7 days, and I feel that it is relatively easy to understand overall, but starting from "Chapter 8 - Decorator", the difficulty has increased, and there are some things that I haven't understood when I read it.

This article can quickly review various considerations and review your own code while developing.

---

Note : The following content is completely excerpted from the book "Python Craftsman Cases, Skills and Engineering Practice".

1 Variables and comments

1. Variables and comments determine the first impression

• Variables and comments are the closest thing to natural language in code, and their readability is very important.
• Even if it is the same algorithm, the variables and annotations are different, and the feeling will be completely different.

2. Basic knowledge

• Python's variable assignment syntax is very flexible, and you can use *variablesasterisk expressions to assign values ​​flexibly.
• Two points for writing comments: don't use them to mask code, and don't use them to explain why.
• Interface annotations are written for the user, so they should concisely describe the function's responsibilities without including too many internal details.
• Variables can be typed using Sphinx -style documentation or type annotations.

3. Variable names are important

• Variable names follow PEP8 principles, as do other parts of the code.
• Try to give variables descriptive names, but evaluating descriptiveness also needs to be combined with the scene.
• On the premise of ensuring descriptiveness, the length of the variable name should be as short as possible.
• The variable name should match the type it represents.
• A super short name of one or two letters is fine, but be careful not to overdo it.

4. Code organization skills

• Organize code according to its responsibility: keep variable definitions close to usage.
• Properly defining temporary variables can improve the readability of the code.
• Unnecessary variables can make code look long and verbose.
• Do not have too many variables in the same scope. The solution: extract data classes and split functions.
• Blank lines are also a special kind of "comment", and proper blank lines can make the code more readable.

5. Code maintainability skills

• Keep variables consistent in two ways: name consistency and type consistency.
• Explicit is better than implicit: don't use locals()batch fetch variables.
• Think of interface comments as a function design tool: write comments first, code later.

2 Numbers and strings

1. Numerical basic knowledge

• Python's floating-point numbers have precision problems, please use Decimalobjects to do precise decimal operations.
• The Boolean type is a subtype of Integer, and Boolean values ​​can be used as 0 and 1.
• Using float('inf')infinity simplifies the boundary handling logic.

2. Basic knowledge of strings

• Strings are divided into two categories: str (text type for humans to read) and bytes (binary type for computers to read)
• You can convert between the two strings by .encode()and.decode()
• Priority recommended string formatting methods (from front to back): f-string, str.format(), C language style formatting
• Use the built-in method of string starting with r to process strings from right to left, which can come in handy in specific scenarios
• String concatenation is not slow, don't be afraid to use it for performance reasons

3. Code readability skills

• When defining numeric literals, you can interpolate _characters to improve readability
• Don't have "magic" literals, replace them with constants or enum types
• Preserving math expressions does not affect performance and improves readability
• textwrap.dedent()Multiline strings fit better into code using

4. Code maintainability skills

• When manipulating structured strings such as SQL statements, using proprietary modules is easier to maintain than bare-handled code
• Use Jinja2 templates instead of string concatenation

5. Language internal knowledge

• Use the dis module to view the Python bytecode to help us understand the internal principles
• Use the timeit module to easily perform performance testing on Python code
• The Python language evolves very fast, don't be easily swayed by the "experience" of the old version

3 container types

1. Basic knowledge

• When making a function call, what is passed is not the value or reference of the variable, but the reference of the object pointed to by the variable
• Python's built-in types are divided into two types: mutable and immutable. Mutability will affect the behavior of some operations, such as+=
• For mutable types, copy them when necessary to avoid unexpected effects
• Common shallow copy methods: copy.copy, derivation, slice operation

2. Lists and tuples

• enumerateYou can get the subscript while traversing the list using
• The multiple return value of the function is actually a tuple
• There is no tuple comprehension, but tuplea generator expression can be converted to a tuple using
• Tuples are often used to represent some structured data

3. Dictionaries and collections

• Before the Python 3.7 version, the dictionary type was unordered, and then changed to preserve the insertion order of the data
• OrderedDictOrdered dictionaries can be obtained in versions prior to Python 3.7 using
• Only hashable objects can be stored in collections, or used as keys in dictionaries
• OrderedDictOrdered deduplication can be quickly implemented using ordered dictionaries
• fronzensetAn immutable collection object can be obtained by using
• Sets can conveniently perform set operations, calculate intersection and union
• dictDon't create custom dictionary types via integration

4. Code readability skills

• Named tuples are more readable than plain tuples
• List comprehensions allow faster traversal, filtering, processing, and building new lists
• Don't write overly complicated derivations, just use plain code instead
• Don't think of comprehensions as loops with less code, just write ordinary loops

5. Code maintainability skills

• When the accessed dictionary key does not exist, you can choose to catch the exception or make a judgment first, and it is recommended to catch the exception first
• Use get, setdefault, pop method with parameters to deal with border logic
• Use a named tuple as the return value, which is more scalable than a normal tuple
• When the dictionary key does not exist, defaultdictthe handling can be simplified using
• Inheritance MutableMappingcan easily create custom dictionary classes and encapsulate processing logic
• Using generators to return members on demand is more flexible and memory-efficient than directly returning a result list
• Dictionaries can be merged conveniently using dynamic unpacking syntax
• Do not modify while traversing the list, otherwise unpredictable results will appear

6. Key points of code performance

• The underlying implementation of the list determines that its head operation is very slow, and dequethe type does not have this problem
• When you need to determine whether a member exists in the container, it is faster to use a dictionary/set

4 Conditional Branch Control Flow

1. Idiomatic writing of conditional branch statements

• Don't explicitly compare to boolean values
• Use the boolean rules of the type itself, omitting the zero value judgment
• move notthe negation logic of the representation inside the expression
• Use operators only when you need to determine whether an object is of type None, True, orFalseis

2. Python data model

• Definition __len__and __bool__magic methods to customize boolean rules for objects
• Define __eq__methods that modify ==the behavior of an object when performing operations

3. Code readability skills

• Duplicate or similar codes are prone to appear in different branches, and extracting them outside the branch can improve the readability of the code
• Expressions with multiple negations can be made easier to understand using "De Morgan's Law"

4. Code maintainability skills

• Keep ternary expressions as simple as possible
• Flatness is better than nesting: use "early return" to optimize multi-level branch nesting in code
• When conditional expressions become particularly complex, try wrapping new functions and methods to simplify
• andThe priority oris higher , don't forget to use parentheses to make the logic clearer
• When using oroperators instead of conditional branches, be careful to avoid the pitfalls (?)

5. Code organization skills

• bisectThe module can be used to optimize the range class branch judgment
• The dictionary type can be used to replace simple conditional branch statements
• Try to summarize the rules in the conditional branch code and rewrite them in a more streamlined and more scalable way
• Conditional expressions can be streamlined using any()and built-in functionsall()

5 Exception and Error Handling

1. Basic knowledge

• A trystatement supports multiple exceptclauses, but remember to put the more precise exception class first
• tryA branch of the statement elsewill be executed in the absence of an exception, so it can be used instead of a tag variable
• A statement without any parameters raiserepeatedly throws the current exception
• Context managers are often used to handle exceptions, and their most common use is in the substitution finallyclause
• Context managers can be used to ignore exceptions in a certain piece of code
• Context managers can be easily defined using @contextmanagerdecorators

2. Error handling and parameter verification

• When you can choose to write conditional judgment or exception capture, prefer exception capture
• Don't let the function return an error message, throw a custom exception directly
• Manually verifying the legality of data is very cumbersome, try to use professional modules to do this
• Don't use it assertfor parameter validation, raiseuse it instead(?)
• There is an additional cost to handling errors, and it would be nice if it could be avoided by design
• When designing an API, you need to carefully consider whether it is really necessary to throw errors
• Use the "null object pattern" to save some error handling for edge cases

3. When you catch the exception

• Overly vague and broad exception catching may save the program from crashing, but it may also cause bigger trouble
• Exception capture is expensive in precision, only captures statements that may throw exceptions, and only captures possible exception types
• Sometimes it is not necessarily a bad thing to let the program crash early
• Very high-risk behavior when exceptions are completely ignored, in most cases, at least one error log is logged

4. When you throw an exception

• Ensure that the exceptions thrown in the module are consistent with the abstraction level of the module itself
• If the abstraction level of the exception is too high, replace it with a new lower-level exception
• If the abstraction level of the exception is too low, wrap it into a higher-level exception
• Don't let the caller use string matching to judge the exception type, try to provide distinguishable exceptions

6 Loops and Iterables

1. The principle of iteration and iterator

• Using iter()the function will try to get an iterator object
• Using next()the function will get the next content of the iterator
• A cycle can be forsimply understood as whilea cycle plus continuous callsnext()
• Custom iterators need to implement __iter__and __next__two magic methods
• A generator object is a type of iterator
• iter(callable, sentinel)An iterator(?) can be constructed from a callable

2. Iterators and iterable objects

• Iterators and iterable objects are different concepts
• An iterable object is not necessarily an iterator, but an iterator must be an iterable object
• Using g on an iterable iter()returns an iterator, which returns itself
• The iterated process of each iterator is one-time, but iterable objects are not necessarily
• Iterable objects only need to implement __iter__methods, while iterators also need to implement additional __next__methods

3. Code maintainability skills

• By defining a generator function to decorate the iterable object, the code inside the loop can be optimized
• itertoolsThere are many functions in the module that can be used to decorate iterable objects
• Generator functions can be used to decouple loop code and improve reusability
• Don't use multiple , it's better breakto split into a function and then directlyreturn
• Using next()functions can sometimes accomplish some unexpected functions

4. File operation knowledge

• Reads file contents using standard practices, can be slow on large files without newlines
• Call file.read()method can solve the performance problem of reading large files

7 functions

1. Basic knowledge of function parameters and returns

• Do not use mutable types as parameter default values, Noneuse instead
• Using the marker object, you can strictly distinguish whether a certain parameter is provided when the function is called
• Define keyword-only parameters, which can force the caller to provide parameter names to improve readability
• Functions should have a stable return type, do not return multiple types
• Use the return Nonesituation - the operation class function, the query class function indicates the expected missing value
• When execution fails, Nonethrowing an exception is more appropriate than returning
• If returning the result early can improve readability, return it early without pursuing a "single exit"

2. Code maintainability skills

• Don't write long functions, but there is no standard length, 65 lines is a red flag
• Cyclomatic complexity is a common indicator for evaluating the complexity of functions, and functions with a cyclomatic complexity of more than 10 need to be refactored
• Abstraction and layered thinking can help us better build and manage complex systems
• Code within the same function should be at the same abstraction level

3. Function and state

• Stateless pure functions without side effects are easy to understand and easy to maintain, but most of the time "state" is unavoidable
• Avoid using global variables to add state to functions
• When the function state is simple, you can use the closure trick
• When the function requires more complex state management, it is recommended to define a class to manage the state

4. The influence of language mechanism on functions

• functools.partial()Can be used to quickly construct partial functions
• functools.lru_cache()Can be used to add caching to functions
• List comprehensions are more readable than mapand should be usedfilter
• lambdaFunctions are just a kind of syntactic sugar, you can use operatormodules, etc. to replace it
• There are many recursion restrictions in the Python language. If possible, please try to use loops instead

8 decorators

1. Basics and skills

• The most common way to implement decorators is to use the principle of closure to implement through multi-layer nested functions
• When implementing the decorator, remember to wraps()update the wrapper function's metadata using
• wraps()Not only retains metadata, but also retains additional properties of wrapper functions
• Decorators for optional parameters can be easily implemented using keyword-only parameters

2. Use classes to implement decorators

• As long as it is a callable object, decorators can be used
• Instances of classes that implement __call__methods are callable
• Class-based decorators are divided into two types: "function replacement" and "instance replacement"
• "Function replacement" decorators are no different from normal decorators, only with fewer layers of nesting
• Implementing "instance replacement" decorators through classes has natural advantages in managing state and appending behavior
• Mixing classes and functions to implement decorators can flexibly meet various scenarios

3. Use the wrapt module

• Use the wrapt module to easily make decorators compatible with both functions and class methods
• Using the wrapt module can help you write decorator code with a flatter structure

4. Decorator design skills

• The decorator advances the wrapper call to the point where the function is defined, and most of its advantages stem from this
• When writing decorators, consider whether your design can take advantage of decorators well
• In some scenarios, class decorators can replace metaclasses, and the code is simpler
• Decorators and decorator patterns are very different, don't confuse them
• There should be only one layer of shallow packaging code in the decorator, and the core logic should be placed in other functions and classes

9 Object-Oriented Programming

1. Basic knowledge of language

• The data of the class and the instance are stored in a __dict__dictionary attribute named
• Flexible use of __dict__attributes can help you do some things that are difficult to accomplish with conventional methods
• Class methods can be defined using @classmethodclass methods, which are often used as factory methods
• Use @staticmethodcan define static methods, static methods do not depend on instance state, it is a stateless method
• Dynamic attribute objects can be defined by using @property, and the behavior of obtaining, setting, and deleting of the attribute object supports customization

2. Object-oriented advanced features

• Python uses MRO algorithm to determine method priority in multiple inheritance
• super()What the function gets is not the parent class of the current class, but the next class in the current MRO chain
• Mixin is an effective programming pattern based on multiple inheritance, using Mixin well requires careful design
• The function of metaclass is quite powerful, but it is also quite complicated. Unless you develop some framework class tools, you rarely need to use metaclass
• By defining __init_subclass__hook methods, you can execute custom logic when a class is inherited

3. Duck typing and abstract classes

• "Duck typing" is one of the most distinctive features of the Python language. In this style, no strict type checking is generally done.
• Although "duck typing" is very useful, it has two obvious disadvantages - lack of standard and too implicit
• Abstract classes provide a more flexible subclassing mechanism, we can change isinstance()the behavior by defining abstract classes
• Through @abstractmethoddecorators, you can require subclasses of abstract classes to implement a certain method

4. Object-oriented design

• Inheritance provides a very powerful code reuse mechanism, but it also brings a very tight coupling relationship
• Incorrect use of inheritance can easily lead to code out of control
• Modeling the behavior of things rather than the things themselves makes it easier to hatch good object-oriented designs
• Care should be taken when creating inheritance relationships. It is sometimes better to use composition instead of inheritance

5. Function and object-oriented cooperation

• Object-oriented in Python does not have to be particularly pure, you can design better code by adding a little coordination with functions
• Like a module, you can requestsuse functions to implement some easier-to-use APIs for your object-oriented modules
• In Python, we rarely apply the true "singleton pattern", in most cases, a simple module-level global object is enough
• Using the "pre-bound method pattern", you can quickly wrap common function-like APIs for common instances

6. Code writing details

• Python's member private agreement is not strict, if you want to mark a property as private, use a single underscore prefix is ​​enough
• When writing a class, the sorting of class methods should follow some special rules, putting the content that readers care most about at the top
• Polymorphism is a basic concept in object-oriented programming, and it is also one of the most imaginative thinking tools
• Possible intervention opportunities of polymorphism: many similar conditional branch judgments, many type-specific isinstance()judgments

10 Principles of Object-Oriented Design (Part 1)

1、SRP

• A class should have only one possibility to be modified
• Writing smaller classes is generally less likely to violate the SRP
• SRP also applies to functions, you can make functions and classes work together

2、OCP

• Classes should be closed for modification and open for extension
• Finding the volatile part of the code by analyzing the requirements is the key to making the class conform to OCP
• Using subclass inheritance can make the class conform to OCP
• Through algorithm classes and dependency injection , classes can also conform to OCP
• Separating data from logic and using a data-driven approach is also a good way to time OCP

11 Object-Oriented Design Principles (Part 2)

1、LSP

• LSP believes that subclasses should be able to replace parent classes arbitrarily
• Subclasses should not throw exceptions that the class does not recognize
• Subclass methods should return the same type as the parent class, or return a subtype object of the parent class return value
• The method parameters of the subclass should be exactly the same as the parent class method, or the requirements are more relaxed ( note the difference from the previous point )
• Certain classes may have implicit contracts, violations of which can also lead to LSP

2、DIP

• DIP believes that both high-level modules and low-level modules should depend on abstraction
• There is one principle in writing unit tests: test the behavior, not the implementation
• Unit tests should not be used too much mock, otherwise the design needs to be adjusted
• The advantage of relying on abstraction is that modifying the underlying module implementation will not affect the high-level code
• In Python, you can use abcmodules to define abstract classes
• Alternativelyabc , you can also Protocolperform dependency inversion with techniques such as

3、ISP

• The ISP believes that the interface that the client depends on should not contain any methods it does not need
• Designing interfaces is designing abstractions
• Writing smaller classes, smaller interfaces is a good idea in most cases

12 Data Model and Descriptor

1. String-related protocols

• Using __str__the method, you can define the string value of the object (to be str()triggered)
• Using __repr__the method, you can define the debug-friendly detailed string value of the object (to be repr()triggered)
• If the object only defines __repr__methods, it will also be used instead of__str__
• Using the method, you can provide various string values ​​( triggered) __format__when the object is used for string template rendering.format()

2. Comparison operator overloading

• By overloading the 6 magic methods related to comparison operators, you can make objects support comparison operations such as ==, etc.>=
• functools.total_orderingThe workload of overloaded comparison operators can be greatly reduced by using

3. Descriptor protocol

• Using the descriptor protocol, you can easily implement reusable property objects
• A class that implements any of the methods of , , is a __get__descriptor __set__class__delete__
• To store instance-level data in a descriptor, you need to store it in instance.dict, not directly on the descriptor object
• The use __set_name__method allows the descriptor object to know what name it is bound to

4. Data class and custom hash operation

• To allow custom classes to support set operations, you need to implement __eq__the __hash__two methods
• If two objects are equal, their hash values ​​must also be equal, otherwise the correctness of the hash table will be broken
• The hash values ​​of different objects can be the same, and hash collisions will not destroy program correctness, but will only affect efficiency
• Using dataclassesmodules, you can quickly create a data class that supports hash operations
• To make a data class support hash operations, you must specify frozen=Truethe parameter to declare it as an immutable type
• An object's hash value must remain constant throughout its lifetime

5. Other suggestions

• Although the data model can help us write more Pythonic code, don't overdo it
• __del__method is not deltriggered when the statement is executed, but when the object is garbage collected
• Do not use __del__to do any "automatic" resource reclamation work

13 Develop large projects

Introduction to some project development tools

• flake8 : Contains pycodestyle module (check code style specification) and pyflakes module (check code correctness)
• isort : format importstatement
• black : Strict code style tool, use black {filename}commands to organize code
• mypy : static type checking tool

Unit testing tools : TDD (Test Driven Development) is a proven way of working

• unittest : Standard library module.

• pytest : An open source third-party unit testing framework, more in line with python style.

  With @pytest.mark.parametrizecan write parameterized tests.

---