In this post, I'll discuss how to clone a PyPI repository onto a Raspberry Pi and serve the content so that connected devices can install dependencies via pip installthe command
Target
There are many ways to create your own PyPI mirrors, and my method probably works for everyone right now. My goals for this project are:
The hardware to do this project must be affordable (<=$100).
There must be little or no setup required on the client computer.
I decided to use a Raspberry Pi 4 with a 200Gb SD card for storage. Use minirepoto clone PyPI, pypiserverto serve packages, nginxto create reverse proxies.
There are four steps to creating a local PyPI server:
1. Download and install the OS and system tools 2. Configure the Raspberry Pi as a WiFi hotspot, DHCP and DNS server 3. Clone/download PyPI packages 4. Configure a web server to serve downloaded packages to connected clients.
Below I will explain the above steps in detail.
1. Download and install the operating system and system utilities
The Raspberry Pi is a small credit card-sized computer that sells for at least $35. For this project, I used a Raspberry Pi 4, but other models should work as well. The Raspberry Pi 3 and 4 models have built-in WiFi adapters, which make setting up the Raspberry Pi as a WiFi hotspot or access point simpler than when using an external wireless adapter.
To get started, download and install Raspbian. Raspbian is a lightweight Debian-based operating system optimized for the Raspberry Pi. In order to work as an access point, the Raspberry Pi will need access point software installed, as well as DHCP server software that provides network addresses to connected devices.
Next, download all the utilities and packages you'll need before configuring your Raspberry Pi.
You need the following packages:
dnsmasq - DNS and DHCP server software hostapd - access point software minirepo - for cloning PyPI, for offline use. pypiserver - Create an index from a cloned PyPI package nginx - A web server
To install these packages, run these two commands:
$ sudo apt install dnsmasq hostapd nginx $ pip install minirepo pypiserver
2. Configure Raspberry Pi's WiFi hotspot, DHCP and DNS
The goal of this step is to configure a separate network as a server, so the Raspberry Pi needs to have a static IP address assigned to the wireless port. To configure a static IP, edit the dhcpcd configuration file.
$ sudo nano/etc/dhcpcd.conf
Add the following:
Configure DHCP
A lot of the defaults in the dnsmasq settings are unnecessary. Create a new configuration file:
$ sudo mv/etc/dnsmasq.conf/etc/dnsmasq.conf.orig $ sudo nano/etc/dnsmasq.conf
Add the following configuration:
This sets up DHCP for clients wlan0connecting . The second line tells the DHCP server (dnsmasq) to listen for connections from the static IP you set up in the previous step. The next line tells DHCP to provide the IP address 192.168.4.2to 192.168.4.30, with a lease time of 24 hours.
create an access point
Next, configure the access point software (hostapd):
$ sudo nano/etc/hostapd/hostapd.conf
Add the following:
Add your own network name and network password where sssidand .wpa_passphrase
Tell the system where to find this file, and hostapdopen the config file.
sudo nano/etc/default/hostapd
Find the line #DAEMON_CONFwith and replace it with this:
DAEMON_CONF="/etc/hostapd/hostapd.conf"
Add routing and masquerading
Edit /etc/sysctl.confand uncomment this line.
net.ipv4.ip_forward=1
eth0Add a mask for outbound traffic from .sudo iptables-t nat-A POSTROUTING-o eth0-j MASQUERADE
Save the new rule:sudo sh-c"iptables-save > /etc/iptables.ipv4.nat"
Edit /etc/rc.local, add the following "exit 0 "above to install these rules on startup.
iptables-restore</etc/iptables.ipv4.nat
This is important if you decide to share your internet connection later or set up a bridge on the Raspberry Pi.
The Raspberry Pi should be ready to work as an access point. If you're connecting directly to it, now is a good time to enable SSH. Reboot the Raspberry Pi and test if everything works.
Using a different WiFi-enabled device, such as a mobile phone or laptop, scan for new wireless networks. If all goes well, you should see the WiFi network you created above. Try to connect to it.
3. Clone PyPI
In this section, you will see how to clone PyPI and configure the following packages.
minirepo pypiserver nginx
Minirepo
Minirepo is a command line program that downloads packages from PyPI.org so you can use pip without internet. The easiest way to install it is to use pip.
$ pip install minirepo
When you execute it for the first time, minirepoyou will be asked the path of the local repository (where it should save the downloaded package), which is the default in Linux ~/minirepo. A JSON configuration file is created and saved as ~/.minirepo, which you can edit to your liking.
There are many alternatives for cloning PyPI, but I used minirepo because it allows you to selectively download a mirror, for example, only download all the source code for Python 3. At the time of writing this article, the entire PyPI repository is about 1TB, but by using selective downloads, I was able to get it down to about 120GB. Below is the configuration I am using in this project.
The configuration above downloads the source code for Python 3 and limits the package types to the sdist, , bdist_wheeland bdist_eggpackages . The disadvantage of using this method is that some packages that do not meet the filter criteria will not be downloaded.
Cloning PyPI takes a long time, so you have to leave it running in the background.
Pipe server
At this point, you should have PyPI mirrored on your computer. My local PyPI mirror has over 200,000 packages. Before we go any further, it's worth understanding what is pipand how it works.
Pipis the most popular tool for installing Python packages and is included with modern versions of Python. It provides basic core functionality for finding, downloading, and installing packages from PyPI and other Python package indexes, and can be incorporated into a wide range of development workflows through its command-line interface (CLI).
PipPackage installation is supported from the following places.
Install packages from PyPI (and other indexes) using the requirements specifier.
The URL of the VCS project.
local project directory
local or remote source files
Since you've cloned the PyPi packages into your local repository, pip can install those packages directly from the local PyPI mirror you just downloaded. But that's not the purpose of this article. The goal here is to allow remote clients to connect to the Raspberry Pi and download software packages over the network. This is what pypiserver does.
pypiserver, will provide a local package index, allowing pip to find packages in your repositories over the network.
First, test that it works.
$ pypi-server-p8080~/minirepo&# Will listen to all IPs.
Note that at runtime, the command to run it pypi-serveris not pypyserver.
Here, you start pypiserverand run it on port 8080. It will minirepofind the package in the folder. This process will keep running in the background until you kill it or turn off the Raspberry Pi. I'll show you how to activate it later.
If you go to your Raspberry Pi's static IP on port 8080 in a browser, you should see a message similar to the one below.
You can now install from your local repository.
pip install--index-url http://localhost:8080/simple/
Alternatively, install from a client computer.
pip install--index-url http://192.168.4.1:8080/
If you installed on a remote URL without HTTPS pypiserver, you'll get an "untrusted" warning from pip, urging you to add the --trusted-hosts option.
pip--trusted-host192.168.4.1install--index-url http://192.168.4.1:8080/
There is also a shorter way.pip--trusted-host192.168.4.1install-i http://192.168.4.1:8080/
It would be cumbersome to always specify the local pypi URL and the trusted hosts flag on the command line. If you want to always install packages from your own mirror, create this pipconfiguration .
Main directory:
On Unix and macOS, the home directory file is:
$HOME/.pip/pip.confIn Windows, the file is:
%HOME%\pip\pip.ini
In the virtual environment:
In Unix and macOS, the file is
$VIRTUAL_ENV/pip.confOn Windows, the file is:
%VIRTUAL_ENV%\pip.ini
I recommend putting this configuration file in a virtual environment.
4. Set up a web server to deliver packages
By default, pypiserverthe entire package directory is scanned on every incoming HTTP request. This can cause a significant slowdown when serving a large number of packages like we did in this example.
A faster way to serve files is to pypiserverput behind a reverse proxy and enable your web server's built-in caching. In this article, I will be using nginx, but you are free to use any web server you like.
Set up a new virtual host in nginx
Create a file /etc/nginx/sites-available/cheeshop.com. For the purposes of this article, I'll call my new web host cheeshop.com.
Run $ sudo nano/etc/nginx/sites-available/cheeshop.comand add the following.
The first part of the configuration instructs nginx to create a 10GB cache that will remain active for 2 hours. The upstream pypi part is responsible for pypiserverserving . CheeseShop is the secret code name for the Python Package Index, which is why I named the server that. You can use any name or IP address you like.
The server section specifies that port 80 will be used for incoming HTTP connections, and forwards those requests to the pypi server.
I don't own the cheeseshop.com domain name, but I can use it because we are creating a separate network with no Internet access. In order for client computers to be able to connect to cheeseshop.com, you need to tell the DNS server how to resolve it. More on that later.
To enable this new virtual host, you need to create a symlink to the configuration file you just created in /etc/nginx/sites-enabled/the folder :
$ sudo ln-s/etc/nginx/sites-available/cheeseshop.com/etc/nginx-sites-enabled/
Doing so will enable the new virtual host. Check that everything works by running sudo nginx-t. If all goes well, great! Next, make a small DNS change to map the cheeseshop.com domain to an IP address.
Open /etc/hostsand add an entry for the newly created cheeseshop.com domain:
The hosts file contains domain-to-IP address mappings that help your computer serve you the correct content. Dnsmasq will check this file on startup, so it's best to restart it:
sudo service dnsmasq restart
Also restart nginx for good measure:
sudo service nginx restart
Assuming all goes well, you should be able to install Python packages from the client machine using the hostname instead of the IP now.
use server
To test this, connect to the Raspberry Pi's WiFi network and create a new virtual environment on the client computer and run the following command inside the virtual environment:
pip--trusted-host cheeseshop.com安装-i http://cheeseshop.com django
Running the command produces the following output:
start pypiserver on boot (optional)
To ensure that the pypiserver software starts automatically on boot, create a new Linux service and manage systemdit with .
1. Create a startup script that the service will manage, name it start-pypi-server.sh. Add the following to it:
2. Copy the script to /usr/bin and make it executable:sudo cp start-pypi-server.sh/usr/bin/start-pypi-server.sh sudo chmod+x/usr/bin/start-pypi-server.sh
3. Create a unit file to define a systemd service. Name it pypiserver.service:
This defines a basic service. ExecStartDirectives specify the command that will be run to start the service.
4. Copy the unit file to /etc/systemd/systemand grant permission:sudo cp pypiserver.service/etc/systemd/system/pypiserver.service
sudo chmod644/etc/systemd/system/pypiserver.service
Start and enable the service
1. After creating the unit file, you can test the service:
sudo systemctl start pypiserver
2. View the service status of pypiserver:
sudo systemctl status pypiserver
This will produce output similar to this:
3. Stop or restart the service:sudo systemctl stop pypiserver sudo systemctl restart pypiserver
4. Finally, use enablethe command to ensure that the service starts at system startup:
sudo systemctl enable pypiserver
in conclusion
We've seen how to create your own local PyPI clone on the Raspberry Pi. you learned how
Set up the Raspberry Pi as an access point
Set up the Raspberry Pi as a DHCP and DNS server
Clone PyPi
Use a web server to serve the cloned package.
I did this as a proof of concept that it is possible to run something like PyPI offline. I'm sure there are better or more efficient ways to do this. For any suggestions or criticisms, please leave a comment below. thanks for reading.
-Click below to read the original text to join the community members-