1. Source code construction and pre-construction:
Source code construction:
Source code construction is the process of obtaining the source code of the software and then compiling it locally to generate an executable program or library file. This approach allows configuration and optimization based on specific needs, but may require longer time and larger resources to compile the source code.
Pre-building:
Pre-building is the process of downloading pre-compiled executable programs or library files from developers or official websites and then using them locally. This approach saves the time and effort of compilation and is often more convenient. However, pre-built versions may not be fully optimized for specific needs.
Advantages and disadvantages of source code construction:
Advantages:
1. Can be configured, optimized and modified according to specific needs.2. It can ensure that the compiled binary exactly matches the system and requirements.
3. Debugging and error tracking are possible, helping with customization and customization.
Disadvantages:
1. Requires long compilation time and system resources.
2. It may be complicated for users who are not familiar with the compilation process.
3. The compilation process may be affected by dependencies and dependency issues need to be dealt with.
2. Compile and install Tensorflow source code
The basic process of building TensorFlowi source code:
1. Installation dependencies: Install various dependencies required to build TensorFlow, including Python, Bazel build tool, CUDA, cuDNN, TensorRT, etc. These dependencies may vary based on system and configuration.
2. Configure compilation: In the TensorFlowi source code directory, run the configuration script to configure compilation options.
3. Build: Build TensorFlow with Bazelt.
4. Generate Wheel package: If you want to package TensorFlow into Pythonf's Wheel format for easy distribution and installation.
5. Run tests (optional): You can run TensorFlow's unit tests and integration tests to ensure that there are no problems with the built version.TensorFlow-Installation dependencies and compilation environment:
Make sure that python, pip and other tools have been installed in the environment
pip install -U --user pip numpy wheel
pip install -U--user keras_preprocessing --no-depsNumpy is an open source numerical calculation library that provides rich multi-dimensional array and matrix operation functions, as well as numerical calculation tools such as mathematics, logic, and Fourier transform.
Wheel is a binary package format for Python designed to speed up package installation. Is a pre-built software package format that can contain the code, dependencies, and metadata of Python modules for easy distribution and installation.
keras_preprocessing is an important submodule of TensorFlow that provides many tools for data preprocessing and enhancement, especially suitable for preparing data sets in deep learning tasks.
TensorFlow - Compilation Tool bazel
Bazel is an open source build tool developed by Google, designed to support efficient building, testing, and deployment of large projects. It is particularly suitable for building complex software systems, with high scalability and performance advantages. Bazel supports multiple programming languages and is widely used in many large-scale projects such as TensorFlow.
■Build speed and caching: An incremental build and caching mechanism is used to compile only the changed parts, thus significantly speeding up the build process. Bazel can also share the build cache to avoid compiling the same dependencies repeatedly.
■Multi-language support: including but not limited to Java, C++, Python, Go, Rust, etc. This makes it possible to mix different languages in a project without having to use different build tools for each language.
■High modularity: Split the project into small, reusable modules to improve code maintainability and testability. Each module can have its own build rules and dependencies.
■Declarative build: Use BUILD files to declare build rules and dependencies. This declarative approach to building makes the build process clearer and more manageable.
■Platform independent: Can run on different operating systems, including Linux, macOS and Windows.For detailed usage, please refer to: https://blog.gmem.cc/bazel-study-note
3. Compile and install pytorch source code
1. Environment preparation:
Install Python: Make sure Python is installed on your system, it is recommended to use a supported version (usually Python 3.7+).
Install dependencies: such as CMake, NumPy, Ninja, etc. These dependencies can be installed according to the official documentation.
2. Update submodules: The source code may contain submodules, and you often need to use the following command to update the submodules for normal compilation:
git submodule update --init --recursive
3. Configure compilation options: (You can use cmake -DCMAKE_INSTALL_PREFIX=<install_dir> to specify the installation directory).
mkdir build
cd build
cmake ..
4. Use the make command to compile and the make install command to install.
5. Run the test:
In the test directory, you can use the following command to run the test:
python run_test.py
This will run all test suites, including unit tests, integration tests, etc. If you need to specify a specific test module or test case, you can use command line parameters to filter the tests,
For example:
python run_test.py test_torch.py
Only tests related to test_torch.py will be run.
After the test run completes, successful tests will display "OK" while failed tests will display a detailed error message.
4. Compile and install caffe source code
1. Environment preparation (basically these can be installed from conda):
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev
libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install python-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install python-opencv
2. Modify the file Makefile.config under caffe
CPU and GPU support: You can decide whether to use CPU or GPU computing when building Caffe by setting whether to enable CPU or GPU support. This usually involves options such as USE_CUDNN, USE_NCCL, CPU_ONLY, etc.
CUDA and cuDNN paths: If GPU support is enabled, the paths to CUDA and cuDNN need to be specified.
BLAS libraries: You can choose to use different BLAS libraries such as OpenBLAS, MKL or ATLAS.
Python path: You need to specify the Python version and path integrated with Caffe.
3. Modify the file Makefile under caffe
1.opencv_core opencv_highgui opencv_imgproc opencv_imgcodecs
Add opencv_imgcodecs later
2. 找到LIBRARIES +=glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
Change the last two items to:
LIBRARIES +=glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl
hdf5_serial
4. Compilation and testing
make all
Make test
make runtest
5. Commonly used model warehouses for deep learning frameworks
TensorFlow： https://github.com/tensorflow/models
TensorFlow Model Garden is an open source project maintained by the TensorFlow team to provide the machine learning community with a collection of pre-trained models, tools, examples, and best practices to accelerate model development and research. This project brings together various types of models, covering computer vision, natural language processing, speech recognition and other fields, to help developers quickly build and train their own models.
Pre-trained models: Model Garden contains multiple pre-trained models, including models officially released by TensorFlow and models contributed by the community. These models can be used as a starting point for transfer learning or can be fine-tuned on specific tasks.
Model Library: Model Garden provides a model library for viewing code, configuration files, and documentation for various models. These models can help developers understand the structure and use of the model.
Best practices: The TensorFlow team will share some best practices for model development and training to help developers avoid common pitfalls and problems.