Currently, various Python tutorials rarely involve Python packaging and deployment technology, or the descriptions are too superficial and one-sided. I try to start from the principles, combine it with examples, and give suggestions on standard operating steps to provide a more detailed reference tutorial for Python project packaging and deployment for Python programming enthusiasts.
Other chapters in this tutorial
Although Python is a dynamically typed programming language, it does not need to be compiled in advance. However, a Python project is also composed of a set of .py files, data files, resource files, etc. Most projects will also reference third-party libraries, and there is also dependency management. Therefore, Python project management is similar to other languages such as Java and C++. When building a Python project, modules and packages are the basic concepts we face.
1. The concepts of modules and packages
A module in Python is a separate .py file that contains variable definitions, function definitions, class definitions, and other executable statements. A module is an independent unit of code that can be run directly using the interpreter and imported into other modules.
A package, on the other hand, is a collection of modules contained in a directory. Packages allow us to group multiple related modules under a common namespace, making it easier to organize and structure our code base.
Breaking your code into modules and packages can bring huge benefits:
- Maintainability. Breaking the code into modules helps us make changes to independent parts of the entire application without affecting the entire application because modules are designed to handle only one part of the application.
- Reusability. This is a key part of software development where we only need to write the code once and use it as many times as needed in as many different parts of the application. This allows us to write clean, concise code.
- Facilitate division of labor and cooperation. With modular code, different developers on the team can take on different parts (modules) of the same application without interfering with each other.
- readability. Breaking the code into modules and packages improves the readability of the code. It is easy to tell the function of different codes in the file. For example, we might have a file called databaseConnection.py: just from the name we can tell that this file handles database connections.
2. Detailed explanation of modules
Modules contain executable statements and function definitions. These statements are used to initialize the module and are executed only the first time an import statement encounters the module name. (These statements are also executed when the file is run as a script.)
Each module has its own private namespace, which is used as the global namespace for all functions defined in the module. Therefore, module authors can use global variables within the module without worrying about accidental conflicts with user global variables. On the other hand, if you know what to do you can access a module's global variables via modname.itemname using the same notation used to reference module functions.
2.1 Example of creating a module
Open your IDE or text editor, create a file, name it sample.py and enter the following code:
# sample.py
# create a variable in the module
sample_variable = "This is a string variable in the sample.py module"
# A function in the module
def say_hello(name):
return f"Hello, {
name} welcome to this simple module."
# This is another function in the module
def add(a, b):
return f"{
a} + {
b} is = {
a+b}"
print(sample_variable)
print(say_hello("小明"))
print(add(2, 3))
The above code defines a module called sample.py. It contains a variable called sample_variable whose value is the string "This is a string variable in the sample.py module". This module also contains two function definitions. When called, the say_hello() function receives a name parameter and if we pass a name to it, it will return a welcome message. The add() function returns the sum of the two numbers passed to it.
Although modules are used in other parts of the program or application, we can run them independently. To run this module, we need to install Python in the development environment. We can run it on the terminal using the following command:
python sample.py
or
python3 sameple.py
operation result
This is a string variable in the sample.py module
Hello, 小明 welcome to this simple module.
2 + 3 is = 5
We can run it as an independent module, but in most usage scenarios, a module is usually used by importing other modules or the Python main program. So, to use variables, functions and classes from one module in another module, we have to import that module. There are different ways to import a module, so let's see how to import a module
2.2 import statement
Use import module_name to import the contents of another module into this module. For example, use import sample in another_module.py
# another_module.py
import sample
print(sample.sample_variable)
print(sample.say_hello(“John”))
print(sample.add(2, 3))
The code above shows how to import functions from the sample.py module so that they can be used in another_module.py. Note that when importing a module, there is no need to include the .py extension
2.3 Use the from … import … statement
You can also use the from keyword to import specific functions or variables. Suppose there are a large number of functions and variables defined in a module, and we don't want to use them all. You can use the from keyword to specify the function or variable to use:
# another_module.py
from sample import add
print(add(10, 4))
The above code shows that we have specifically imported the add() function from the example module.
Another benefit of using the from keyword is that when using an imported function, there is no need to name it or prepend it with the name of its parent module. This will result in cleaner and more readable code.
2.4 Use the as keyword
We can use "as" to provide an alias or alternative name for a module.
Sometimes we may define module names that are quite long or unreadable. Python provides a way to provide aliases for module imports, for this we will use the as keyword:
# another_module.py
import sample as sp
result = sp.add(5, 5)
print(result)
print(sp.say_hello("Jason"))
This code shows the import of a sample module, where the module is given an alias sp. Using sp is exactly the same as calling sample. Variables and functions can also be accessed.
Using the above three methods, we can use variables or functions of one module in another module, thereby enhancing the readability of the application because we do not need to put the code in one file.
When naming module names, it is best to use lowercase letters and use _the following line between the two words. For example, there is a module for handling database connections, you can name it database_connection.py. Also, remember that names in Python are case-sensitive, so make sure to use the correct module name when importing.
Overall, using modules allows us to create and organize code in a readable and maintainable way.
2.5 Get module name in code
Inside the module, the module name (i.e. string) can be obtained through the global variable name . Modify the previous example slightly and obtain the module name inside the sample and another_module.py.
# another_module.py
import sample
print(sample.sample_variable)
print(sample.say_hello(“John”))
print(sample.add(2, 3))
print(__name__)
In another_module.py, get the sample.py module name
# another_module.py
import sample as sp
result = sp.add(5, 5)
print(result)
print(sp.say_hello("Jason"))
print(sp.__name__)
3. Detailed explanation about packages
Packages in Python are a way of organizing related modules into directories. It provides a better way to organize code. Packages are usually used to group modules that complete the same function or belong to the same component.
Packages represent hierarchical relationships with the same package, module, and function (global variable) by using the "." period. For example, the module name AB specifies a submodule named B in a package named A.
1) The relationship between Package, Module and files
Package (package), Module (module), file (file) The relationship between the three,
- Package is composed of modules, which is equivalent to the namespace of Modules.
- Module, the .py code file containing the function code is also called a module, but it cannot be called
config.py , __init__.pya module.
2) The relationship between packages and directories
Although formally, a package can be understood as a directory, not all directories containing code are packages.
The key to distinguishing the two is that the package directory contains __init__.pyfiles, and those that do not contain this file cannot be called a package directory .
Package can contain sub package, but it must also be in its directory.__init__.py
When used, use dots to indicate hierarchical relationships
mypkg.mymodule.func_1()
3) Import packages, modules, functions
iAssume that the current project is a project that processes sound files and sound data in a unified manner. Sound files come in many formats (often identified by extensions, for example: .wav, .aiff, .au), so a growing collection of modules needs to be created and maintained for conversion between different file formats. In order to implement different processing of sound data (e.g. mixing sounds, adding echo, equalizer functions, creating artificial stereo effects), an endless stream of modules has to be programmed. The following hierarchical file tree shows the architecture of this package:
sound/ Top-level package
__init__.py Initialize the sound package
formats/ Subpackage for file format conversions
__init__.py
wavread.py
wavwrite.py
aiffread.py
aiffwrite.py
auread.py
auwrite.py
...
effects/ Subpackage for sound effects
__init__.py
echo.py
surround.py
reverse.py
...
filters/ Subpackage for filters
__init__.py
equalizer.py
vocoder.py
karaoke.py
...
As mentioned in the previous section, __init__.pythe directory containing is a package. Therefore, in the above project, sound is the main directory of the project, and it is also a package. There are three sub-packages under it, filters, effects, and formats. There are several modules under each sub-package.
How to import subpackages.
Absolute quote:
The module sound.filters.vocoder needs to use the echo module under the sound.effects sub-package, quoted like this:
from sound.effects import echo
relative reference
from . import echo means importing the echo module under the current package
from ... import formats means importing the formats sub-package from the parent directory.
4) Import 3rd party packages
Third-party packages, for packages that have been verified by the python community, are usually also called libraries. The import methods are divided into: according to the source:
(1) Import the package installed by pip.
According to the PyPi method, that is, the third-party package installed by pip. When importing, the Python interpreter will search for it under Lib/site-packages/ in the python installation directory. If it is not found, an error will be reported.
import numpy
import pandas
(2) Manually import the third-party package.
You need to add the absolute path of the third-party package to the sys.path variable. For example, the path of the third-party package pkg_a is /usr/local/pkg_a, import and use examples:
import sys
# appending a path
sys.path.append('/usr/local/pkg_a')
# importing required module
import pkg_a
from pkg_a import module_a
# accessing its content
module_a.foo()
4. Module search path
Frequently Asked Questions : When importing packages and modules in the test/ test file, we often encounter the import error problem. The reason is that the test directory is not a project sub-package. According to the default import method, the module path cannot be found.
In the .py module file, when importing other modules, such as spam, the interpreter first searches for the python built-in module. These module names are listed in sys.builtin_module_names. If it is not found, it searches for a file named spam.py in the list of directories given by the variable sys.path.
sys.path is initialized from:
- The directory where the .py script is run directly from the command line (or the current directory if no file is specified).
- PYTHONPATH (contents is a directory listing).
- Lib/site-packages subdirectory under the python installation directory.
>>> import sys
>>> sys.path
['',
'C:\\opt\\Python36\\python36.zip',
'C:\\opt\\Python36\\DLLs',
'C:\\opt\\Python36\\lib',
'C:\\opt\\Python36',
'C:\\Users\\NanoDano\\AppData\\Roaming\\Python\\Python36\\site-packages',
'C:\\opt\\Python36\\lib\\site-packages',
'C:\\opt\\Python36\\lib\\site-packages\\win32',
'C:\\opt\\Python36\\lib\\site-packages\\win32\\lib',
'C:\\opt\\Python36\\lib\\site-packages\\Pythonwin']
3 ways to change sys.path:
(1) appending a path
sys.path.append(``'C:/Users/Vanshi/Desktop'``)
(2) sys.path will read the Python system variable PYTHONPATH when initialized. PYTHONPATH can be modified in the startup script, the syntax is the same as the environment variable path
set PYTHONPATH=C:\pypath1\;C:\pypath2\
python -c "import sys; print(sys.path)"
# Example output
['', 'C:\\pypath1', 'C:\\pypath2', 'C:\\opt\\Python36\\python36.zip', 'C:\\opt\\Python36\\DLL
The set variable name = value is added temporarily and will become invalid after closing the terminal.
To permanently add a modifiable environment variable,
setx ENV_NAME env_value should be used with caution.
like:
setx -m PATH “%PATH%;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin”
Add environment variable command in Linux
export PYTHONPATH=‘/some/extra/path’
(3) Use the site module to modify sys.path
Modify the sys.path variable in the program code so that it is loaded when the system starts and the corresponding module can be found.
import site
import sys
site.addsitedir('/the/path') # Always appends to end
print(sys.path)