Creating executables for machine learning Python scripts
In my previous article, I have showed you how to create an .exe file using PyInstaller. However, it will be a bit of a challenge if your project uses too many libraries (which took me two days to solve the issues I encountered, hence thought of providing a step by step guide for Python users to save their valuable time) . Therefore, in this article I am focusing on creating .exe files for Python scripts for machine learning projects, where libraries such as pandas, numpy, scikit-learn are being used.
Here you go..
In my application, I am using following Python libraries;
1. Pandas
2. Numpy
3. Scikit-learn
4. Scipy
Most of the times, PyInstaller doesn't recognise all of these libraries. In this case you need to specify these libraries in hooks files one by one. How to do this? let's start from Pandas.
Note: Please install PyInstaller (follow the instructions given here), and keep your Python scripts ready.
1. Pandas
You can locate all the hooks files in the hooks folder of the PyInstaller distribution folder and see the names of the packages for which hooks have been written. In my case, all the hooks files are located in: ....\AppData\Local\Programs\Python\Python36-32\Lib\site-packages\PyInstaller\hooks.
Check whether it has the file named, hook-pandas.py, if not, create a one and add the following line.
hiddenimports = ['pandas._libs.tslibs.timedeltas'].
Now when you run the PyInstaller you will not get any error for Pandas.
2. Scikit-learn
After solving the module not found error for Pandas, I have got errors in different library files of Scikit-learn.
Therefore, after going through online help, and after countless trial and error efforts, creating following hooks files solved Scikit-learn issues.
(1). Create hook-sklearn.tree.py and add the following
hiddenimports = ['sklearn.tree._utils']
(2). Create hook-sklearn.metrics.cluster.py (if not exists) and add the following
hiddenimports = ['sklearn.utils.sparsetools._graph_validation',
'sklearn.utils.sparsetools._graph_tools',
'sklearn.utils.lgamma',
'sklearn.utils.weight_vector',
'sklearn.utils.fixes',
'sklearn.utils.extmath',
'sklearn.metrics.ranking',
'sklearn.neighbors',
'sklearn.neighbors.typedefs',
'sklearn.neighbors.quad_tree']
If this file already exists, you can modify it by adding the missing libraries one by one according to the error you get when running PyInstaller.
In my case, I had to add 'sklearn.neighbors', 'sklearn.neighbors.typedefs', and 'sklearn.neighbors.quad_tree'.
3. Scipy
Create hooks-scipy.py and add the following lines. This will tell PyInstaller to load all the dlls.
import os
import scipy
import platform
binaries = []
if "windows" in platform.platform().lower():
dll_path = os.path.join(os.path.dirname(scipy.__file__), 'extra-dll', "*.dll")
binaries = [(dll_path, "dlls")]
hiddenimports = ['scipy._lib.%s' % m for m in ['messagestream', "_ccallback_c", "_fpumode"]]
By following the above steps to include each library using hooks files, solved issues I encountered when creating the .exe using PyInstaller. Now when I run the following command,
pyinstaller my_project_name.py, an .exe file is created in the dist folder. (Here, my_project_name.py is the main Python script which uses the above mentioned libraries).
The article here provides more details about hooks in Pyinstaller, this will be useful in case you face a different issue with Python libraries.