FastDFS and Nginx combine to build a file server and realize public network access [intranet penetration]

foreword

FastDFS is an open source lightweight distributed file system. It manages files. Its functions include: file storage, file synchronization, file access (file upload, file download), etc. It solves the problems of large-capacity storage and load balancing. It is especially suitable for online services with files as the carrier, such as photo album websites, video websites and so on.

FastDFS is tailor-made for the Internet, fully considers redundant backup, load balancing, linear expansion and other mechanisms, and pays attention to high availability, high performance and other indicators. Using FastDFS is easy to build a set of high-performance file server clusters to provide file upload and download and other services.

Equipping a public network server as a file upload and download costs a lot of resources and overhead. If you build the FastDFS service locally, you only need to use the cpolar intranet port forwarding tool to configure the file to be accessed through the Internet, which greatly reduces Server fees and overheads, this article introduces the detailed tutorial of building FastDFS, and combines Nginx+cpolar tools to realize remote access to files.

1. Build the FastDFS file system locally

1.1 Environment installation

Since fastDFS is written in c language, we need to installgcc

yum -y install gcc-c++ 

FastDFS is dependent libevent库and needs to be installed. Execute the following command:

yum -y install libevent

1.2 Install libfastcommon

Enter the local directory

cd /usr/local

libfastcommon is officially provided by FastDFS, and libfastcommon contains some basic libraries needed for FastDFS to run. Click the link below to get it:

https://github.com/happyfish100/libfastcommon/releases/tag/V1.0.7

After getting it, upload it to localthe directory

Unzip libfastcommon

tar -zxvf libfastcommonV1.0.7.tar.gz

Go to the unzipped folder

cd libfastcommon-1.0.7/

compile

./make.sh

Install

./make.sh install

After libfastcommon is installed, it will automatically copy the library files to the /usr/lib64next directory. Since the FastDFS program refers to the usr/lib directory, it is necessary to copy the library files under /usr/lib64 to /usr/lib.

cp /usr/lib64/libfastcommon.so /usr/lib/

1.3 Install FastDFS

Enter the local directory

cd /usr/local

Click the following address to download the FastDFS_v5.05.tar.gz file:

https://sourceforge.net/projects/fastdfs/

After downloading the file, upload FastDFS_v5.05.tar.gzit to/usr/local/

unzip files

tar -zxvf FastDFS_v5.05.tar.gz 

Go to the unzipped folder

cd FastDFS

compile

./make.sh 

Install

./make.sh install

After the installation is successful, copy the files under conf in the installation directory to /etc/fdfs/the following

cd conf/

copy

cp * /etc/fdfs/

1.4 Configure Tracker

Next, configure and enter /etc/fdfsthe directory after the installation is successful

cd /etc/fdfs/

Modify tracker.conf

vim tracker.conf

base_path=/home/yuqing/fastdfschange tobase_path=/home/fastdfs

Save and exit after editing, and then create a correspondingfastdfs文件夹

mkdir /home/fastdfs

then start

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

Set boot

vim /etc/rc.d/rc.local

Add the run command line to the file

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

1.5 Configuring Storage

Enter the /etc/fdfs directory

cd /etc/fdfs

Modify storage.conf

vim storage.conf

base_path=/home/yuqing/fastdfschange tobase_path=/home/fastdfs

store_path0=/home/yuqing/fastdfschange tostore_path0=/home/fastdfs/fdfs_storage

Continue to modify, set the tracker server: IP, tracker_server=Linux LAN ip: 22122, save and exit after setting

Create the corresponding fdfs_storage folder

mkdir /home/fastdfs/fdfs_storage

then start

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

Set up automatic startup

vim /etc/rc.d/rc.local

Add the run command line to the file, then save

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

1.6 Test upload and download

If FastDFS is installed successfully, you can test upload and download operations through the /usr/bin/fdfs_test command.

Modify /etc/fdfs/client.conf

vim /etc/fdfs/client.conf

Modify the following content, then save:

base_path=/home/fastdfs

tracker_server=Linux局域网ip:22122

Next, test uploading files. Here, a picture is uploaded to /usr/localthe directory for use as a test.

FastDFS upload file format:

/usr/bin/fdfs_test /etc/fdfs/client.conf upload+ path to upload files

Then upload the test23.png image to FastDFS according to the format

/usr/bin/fdfs_test /etc/fdfs/client.conf upload /usr/local/test23.png

After execution, we can see the return address information, which means success, but we can’t access it yet, we need to combine Nginx to access and download

1.7 Integration with Nginx

Click the download link below to download fastdfs-nginx-module_v1.16:

[https://sourceforge.net/projects/fastdfs/files/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz/download](https://sourceforge.net/projects/fastdfs/files/FastDFS Nginx Module Source Code/fastdfs-nginx-module_v1.16.tar.gz/download)

Enter the /usr/local/folder

cd /usr/local/

uploadfastdfs-nginx-module_v1.16.tar.gz

Unzip fastdfs-nginx-module_v1.16.tar.gz

tar -zxvf fastdfs-nginx-module_v1.16.tar.gz

Enter src, then edit the configuration file

cd fastdfs-nginx-module/src/

Modify the config file

vim config

Modify these three locations, change the path that was originally /usr/local/ to /usr/, then save and exit

Then copy mod_FastDFS.conf under FastDFS-nginx-module/src to /etc/fdfs/

cp mod_fastdfs.conf /etc/fdfs/

Modify the contents of mod_FastDFS.conf:

vim /etc/fdfs/mod_fastdfs.conf

The modification is as follows:

base_path=/home/fastdfs

tracker_server=Linux LAN ip address: 22122

url_have_group_name=true #url contains group name

store_path0=/home/fastdfs/fdfs_storage #Specify file storage path

Next, copy libfdfsclient.so to /usr/lib

cp /usr/lib64/libfdfsclient.so /usr/lib/

Just create the nginx/client directory

mkdir -p /var/temp/nginx/client

1.8 Install Nginx

First install the serial environment

yum -y install pcre pcre-devel  

yum -y install zlib zlib-devel  

yum -y install openssl openssl-devel

Then click to download the Nginx package:

nginx-1.16.1.tar.gz

Upload to the root directory

Unzip and enter the unzipped folder

tar -zxvf nginx-1.16.1.tar.gz && cd nginx-1.16.

Add the FastDFS-nginx-module module and execute the following command:

./configure \
--prefix=/usr/local/nginx \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/temp/nginx/client \
--http-proxy-temp-path=/var/temp/nginx/proxy \
--http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
--http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
--http-scgi-temp-path=/var/temp/nginx/scgi \
--add-module=/usr/local/fastdfs-nginx-module/src

then compile

make

Install

make install

1.9 Configure Nginx

Modify the Nginx configuration file

vim /usr/local/nginx/conf/nginx.conf

Add server, save and exit after adding:

server {
    
    
        listen       8089;#为了防止80端口,我们改成8089
        server_name  192.168.59.133;#这里需要使用你本机的局域网ip

        location /group1/M00/{
    
    
                ngx_fastdfs_module;
        }
}

Start Nginx

/usr/local/nginx/sbin/nginx 

If the following error occurs when starting nginx:

nginx: [emerg] open() “/var/run/nginx/nginx.pid” failed (2: No such file or directory)

You can execute the following command and start again:

mkdir /var/run/nginx

Set it to start automatically at boot.

vim /etc/rc.d/rc.local

Add the running command line to the file, then save and exit:

/usr/local/nginx/sbin/nginx

turn off firewall

service iptables stop

Set the firewall not to start automatically

chkconfig iptables off

2. LAN test access FastDFS

Then according to the format, test upload the test23.png image we uploaded above to FastDFS

/usr/bin/fdfs_test /etc/fdfs/client.conf upload /usr/local/test23.png

After execution, we can see the return address information, indicating success

Then use the above address to visit in the browser, you can see the pictures we uploaded, and fastDFS is built

3. Install cpolar intranet penetration

Above, we installed the fastDFS distributed file service on the local Linux virtual machine. Next, we installed the cpolar intranet penetration tool. Through the http public network address of cpolar, we can easily access the files uploaded in fastDFS remotely, without having to Register a domain name to purchase a cloud server. The following are the steps to install cpolar intranet penetration

cpolar official website: https://www.cpolar.com/

• Install commands using one-click scripts

curl -L https://www.cpolar.com/static/downloads/install-release-cpolar.sh | sudo bash

• token authentication

Log in to the background of the cpolar official website, click the verification on the left to view your own authentication token, and then paste the token in the command line:

cpolar authtoken xxxxxxx

• Add a service to the system

sudo systemctl enable cpolar

• Start the cpolar service

sudo systemctl start cpolar

4. Configure the public network access address

After successfully starting the cpolar service, we visit the Linux LAN ip address + port 9200 on the browser, and log in to the cpolar web UI management interface.

After successful login, click Tunnel Management on the left dashboard - Create Tunnel, and create an http protocol tunnel pointing to the Nginx8089 port set above:

• Tunnel name: can be customized, be careful not to duplicate the existing tunnel name
• protocol: http
• Local address: 8089
• Domain name type: choose a random domain name for free
• Region: Select China VIP

click创建

Then open the online tunnel list, view and copy the public network address

Then open the browser, enter the public network address, access the files we uploaded in fastDFS, and the remote access is successful

5. Fixed public network address

Since the tunnel just created uses a random temporary address, the address will change within 24 hours. For long-term and more stable remote access, we will configure this public network address as fixed next.

5.1 Reserved second-level subdomains

You need to upgrade to the basic package or above to support the configuration of second-level subdomains. If you need to download files, you can upgrade to a higher package to obtain greater bandwidth and improve download speed.

Log in to the background of the cpolar official website , click on the dashboard on the left 预留, find 保留二级子域名, and reserve a second-level subdomain name for the http tunnel.

• Region: Select the server region
• Name: Fill in the second-level subdomain name you want to keep (customizable)
• Description: Remarks, which can be customized

This example reserves a fasttestsecond-level subdomain named . After the subdomain name is successfully reserved, we copy the subdomain name, and then we need to configure it in the tunnel.

5.2 Configure the second-level subdomain name

Log in to the cpolar web ui management interface. 隧道管理Click —— on the left dashboard 隧道列表, find the tunnel that needs to configure the second-level subdomain name, and click on the right编辑

Modify the tunnel information and configure the second-level subdomain name into the tunnel:

• Domain Type: Select instead二级子域名
• Sub Domain: Fill in the second-level subdomain name we just reserved (in this example fasttest)

After the modification is complete, click更新

状态After the tunnel is successfully updated, click —— on the left dashboard 在线隧道列表, and you can see the public network address of the tunnel, which has been updated to a second-level subdomain name, and copy the public network address.

6. Test access to fixed second-level subdomains

Let's test access to the successfully configured second-level subdomain name, open the browser, enter the public network address plus the resource path, access the files we uploaded in fastDFS, and the remote access will be successful if the picture appears. Now, our only private second-level subdomain name in the entire network has been created.

And this address will not change randomly anymore, it is fixed, as long as the tunnel is online, we can use this public network address to remotely access anytime and anywhere, without the need for a public network IP or setting up a router.

Reprinted from the article of cpolar pole point cloud: FastDFS and Nginx combine to build a file server, and internal network penetration to achieve public network access