Synchronize and set up mirror site under CentOS system

Setting up a mirror site is suitable for intranets and isolated networks. In particular, there are two feelings when there are active and no sources in the isolated network. Here I take the CentOS system as an example to record my own experience of setting up a mirror site.
 

Use reposync to sync

Install synchronization tool

In the past, synchronization used the rsync tool plus createrepo to generate metadata, but now there is a better way to use the reposync tool in CentOS.

# CentOS 7
sudo yum install yum-utils
# CentOS 8
sudo dns install dnf-utils

Synchronize the default warehouse

After the installation is complete, you can immediately perform warehouse synchronization.
 
The default repository of CentOS 7 contains:

• base
• updates
• extras

The default repository of CentOS 8 contains:
 

First sync

// CentOS 7
sudo reposync --download-metadata -p /home/repos
// CentOS 8
sudo dnf reposync --download-metadata --destdir /home/repos

Description:

1. "--Download-metadata" download metadata, otherwise you need to use createrepo to generate.
2. "-P" and "--destdir" specify the storage directory, the default is the current directory where the command is executed.
3. The above command will synchronize all current valid sources, you can use the command "yum repolist" to view, or view the description in "[]" in the .repo file, and enable=1 at the same time.

Update synchronization

// CentOS 7
sudo reposync --download-metadata -p /home/repos --delete -n

Description:

1. "--Delete" deletes packages that do not exist in the warehouse.
2. "-N" only updates the latest, incremental update.

Update EPEL

EPEL: Extra Packages for Enterprise Linux, which are needed in many places as an important CentOS source.

• Install epel source

// CentOS 7
sudo yum install epel-release
// CentOS 8

• Update epel source

The operation method can use the above incremental update, or "-repo <repo_id>" to update the specified source.

// CentOS 7
sudo reposync --download-metadata -p /home/repo --repo epel
// CentOS 8
sudo dnf reposync --download-metadata -p /home/repo --repo epel

According to the official EPEL document, if EPEL is used in CentOS 8, it is recommended to enable the powertools source

sudo dnf reposync --download-metadata -p /home/repo --repo powertools

Update the source mirror of other architectures

Before updating the source image of other architectures (mainly ARM), I used QEMU to virtualize an aarch64 virtual machine. Although the problem was solved, it was not as fast as x86_64. In addition, if you use vbox on windows, you still deal with it. Can't solve the virtualization problem of aarch64, so I studied it again.
 
One is to look at the parameters: reposync or dnf reposync has a parameter option of --arch, which looks like it, but it was not successful, but the noarch package was synchronized; the other is to change the source url, which was successful. The following is the modification time Several variables that need to be explained.

• $releasever: The release version number of the current system, such as "7", you can use the following command to view

  rpm -q --qf %{version} centos-release

• $basearch: The CPU architecture of the current system, such as "x86_64", "aarch64", you can use the following command to view

  rpm -q --qf %{arch} centos-release

• $infra: is stock, the custom variables in the /etc/yum/vars/ directory
  can also view the information of the centos-release package through the following command

  rpm -qi centos-release

  After knowing the above variables, you can specify $basearch as the organization that needs to be synchronized to perform source synchronization operations. For example, modify $basearch to aarch64 to synchronize aarch64 packages.

Install mirror site

The Web site is implemented using Apache (because the system comes with it).

Install Apache

sudo yum install httpd -y

Configure Apache

New configuration file repos.conf for the source mirror site

sudo vim /etc/httpd/conf.d/repos.conf

Alias /repos /var/www/repos
<Directory /var/www/repos>
  Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

Apache attribute description

Apache uses <Directory>... </Directory> to set the access permissions of the specified directory. This setting is also valid for subdirectories, which can contain five attributes:

Options
AllowOverride
Order
Allow
Deny

• Options can set the following options in combination:

  All：用户可以在此目录中作任何事情。
  ExecCGI：允许在此目录中执行CGI程序。
  FollowSymLinks：服务器可使用符号链接指向的文件或目录。
  Indexes：服务器可生成此目录的文件列表。
  None：不允许访问此目录。

• AllowOverride will determine whether to read the .htaccess file in the directory according to the set value to change the originally set permissions:

  All：读取.htaccess文件的内容，修改原来的访问权限。
  None：不读取.htaccess文件

  To prevent users from creating .htaccess files to modify access permissions, the default settings for each directory in the apache2.conf file are:

  AllowOverride None。

• Allow is used to set the host that is allowed to access the Apache server, for example:

  Allow from all  # 允许所有主机的访问
  Allow from 202.96.0.97 202.96.0.98  # 允许来自指定IP地址主机的访问

• Deny is used to set the host that denies access to the Apache server, for example:

  Deny from all  # 拒绝来自所有主机的访问
  Deny from 202.96.0.99 202.96.0.88 # 拒绝指定IP地址主机的访问

• Order is used to specify the order of allow and deny: Order allow, deny

Boolean variables of Apache's SELinux

For network services, SElinux only opens the minimum operating requirements. In order to take advantage of the functions of the Apache server, the Boolean value must be turned on to allow certain behaviors including allowing httpd script network access, allowing httpd to access NFS and CIFS file systems, allowing Execute the Common Gateway Interface (CGI) script. You can use the command getsebool to query the current Boolean variable. Then you can use the following setsebool command to open the boolean variable:

#setsebool – P httpd_enable_cgi on

The following are commonly used Boolean variables:

allow_httpd_anon_write 禁用时这个布尔变量允许的 httpd 到只有标记文件 public_content_rw_t 类型的读取访问。启用此布尔变量将允许的 httpd 写入到文件标记与一个公共文件目录包含一个公共文件传输服务，如 public_content_rw_t 类型。
allow_httpd_mod_auth_pam 启用此布尔变量允许 mod_auth_pam 模块访问 httpd。
allow_httpd_sys_script_anon_write 这个布尔变量定义 HTTP 脚本是否允许写访问到文件标记在一个公共文件传输服务 public_content_rw_t 类型。
httpd_builtin_scripting 这个布尔变量定义 httpd 的脚本的访问。在这个布尔变量启用，往往需要为 PHP 内容。
httpd_can_network_connect 禁用时这个布尔变量防止从网络或远程端口发起连接的 HTTP 脚本和模块。打开这个布尔变量允许这种访问。
httpd_can_network_connect_db 禁用时这个布尔变量防止发起一个连接到数据库服务器的 HTTP 脚本和模块。打开这个布尔变量允许这种访问。
httpd_can_network_relay 打开这个当布尔变量的 httpd 正在使用正向或反向代理。
httpd_can_sendmail 禁用时这个布尔变量防止发送邮件的 HTTP 模块。这可以防止垃圾邮件的***漏洞中发现的 httpd。打开这个布尔变量允许 HTTP 模块发送邮件。
httpd_dbus_avahi 关闭时这个布尔变量拒绝服务的 avahi 通过 D-BUS 的 httpd 访问。打开这个布尔变量允许这种访问。
httpd_enable_cgi 禁用时这个布尔变量防止 httpd 的执行 CGI 脚本。打开这个布尔变量让 httpd 的执行 CGI 脚本。
httpd_enable_ftp_server 开放这个布尔变量会容许的 httpd 作为 FTP 服务器的 FTP 端口和行为。
httpd_enable_homedirs 禁用时这个布尔变量阻止访问用户主目录的 httpd。打开这个布尔变量允许 httpd 访问用户主目录。
httpd_execmem 启用时这个布尔变量允许 httpd 的执行程序需要的内存地址。建议不要启用这个布尔变量从安全角度来看，因为它减少了对缓冲区溢出，但是某些模块和应用程序（如 Java 和 Mono 应用）的保护需要这种特权。
httpd_ssi_exec 这个布尔变量定义服务器端包含（SSI）的网页中的元素是否可以执行。
httpd_tty_comm 这个布尔变量定义的 httpd 是否被允许访问的控制终端。这种访问通常是不需要的，但是，如配置 SSL 证书文件的情况下，终端访问所需的显示和处理一个密码提示。
httpd_use_cifs 打开这个布尔变量允许 httpd 访问 CIFS 文件系统上标记的文件，如通过 Samba 挂载的文件系统，cifs_t 类型。
httpd_use_nfs 打开这个布尔变量允许 httpd 访问 NFS 文件系统上的标记文件 nfs_t 类型，如文件系统，通过 NFS 挂载。

File Types of Apache's SELinux

When SELinux is enabled, the Apache HTTP server (httpd) runs in the restricted httpd_t domain by default and is separated from other restricted network services. Even if a network service is destroyed by someone, the person's resources and possible damage are limited.

The following example demonstrates the httpd process under SELinux.

$ ps -eZ | grep httpd
unconfined_u:system_r:httpd_t:s0 2850 ? 00:00:00 httpd
unconfined_u:system_r:httpd_t:s0 2852 ? 00:00:00 httpd
……

The httpd process related to SELinux context is system_u:system_r:httpd_t:s0. The httpd processes are all running in the httpd_t domain. The file type must be set correctly for httpd access. For example, httpd can read the file type httpd_sys_content_t, but cannot write or modify it.
 
In addition, httpd cannot access files of type samba_share_t (files controlled by Samba access), nor can it access files of type marked with user_home_t in the user's home directory, mainly to prevent httpd from reading and writing files in the user's home directory and inheriting its access permissions. The file type that httpd can read and write is httpd_sys_content_rw_t.
 
Apache's default document root directory type is httpd_sys_content_t. Unless otherwise set httpd can only access httpd_sys_content_t files and subdirectories in the /var/www/html/ directory.
 
In addition, SELinux also defines some file types for httpd:

• httpd_sys_content_t is mainly used to provide files for static content services, such as files used by HTML static websites. This type of markup file can access (read-only) httpd and execute script httpd. By default, this type of file and directory tags cannot be written or modified by httpd or other processes. Note: By default, the file created or copied to /var/www/html/httpd_sys_content_t type tag.
• httpd_sys_script_exec_t is mainly used to set the cgi script in the /var/www/cgi-bin/ directory. By default, the SELinux policy prevents httpd from executing CGI scripts.
• httpd_sys_content_rw_t uses the type tag of httpd_sys_content_rw_t to read and write the type of script tag file httpd_sys_script_exec_t.
• httpd_sys_content_ra_t Use the type tag of httpd_sys_content_ra_t to read and attach the marked script file httpd_sys_script_exec_t type.
   
  To change the destination file directory of the soft link and its file attributes, run the following command

  sudo chcon -R -t httpd_sys_content_t /home/mgr/repos

  At the same time, ensure that the file operation permission of the path /home/mgr/repos is 755.
   
  Description: Use the chcon command to re-identify the file type. However, such a logo is not a permanent modification. Once the system restarts, the logo will change back. For permanent changes of file types, semanage and restorecon commands are required.

  # semanage fcontext -a -t httpd_sys_content_t "/mywebsite(/.*)?"
  # restorecon -R -v /mywebsite

  How to use the three commands chcon, semanage fcontext and restorecon:

• chcon command
  function: chcon command is used to change the attributes of SELinux files, that is, to modify the security context of the file.
  Usage: chcon [options] CONTEXT file
  main options:
  -R: recursively change the context of files and directories.
  --reference: copy the security context from the source file to the target file
  -h, --no-dereference: affect the target link.
  -v, --verbose: output the diagnosis of each check file.
  -u, --user=USER: Set the security context of the target user.
  -r, --role=ROLE: Set the role of the target security domain.
  -t, --type=TYPE: The type of security context set in the target.
  -l, --range=RANGE: set the set role ROLE in the target security context range of the target security context.
  -f: Display a few error messages.
• restorecon command
  function: restore SELinux file attribute file attribute is to restore the security context of the file.
  Usage: restorecon [-iFnrRv] [-e excludedir] [-o filename] [-f filename | pathname...]
  Main options:
  -i: ignore File that does not exist.
  -f: infilename The file to be processed is recorded in the infilename file.
  -e: directory exclude directories.
  -R -r: Process directories recursively.
  -n: Do not change the file label.
  -o outfilename: save the file list to outfilename, if the file is incorrect.
  -v: Display the process on the screen.
  -F: Force recovery of file security context.
  Note: The restorecon command is similar to the chcon command, but it sets the security context of the object related to the file based on the current policy default file context file. Therefore, the user does not specify a security context. Instead, restorecon uses the file context file entry to match the file name. Then apply a specific security context, and in some cases, it is restoring the correct security context.
• semanage fcontext command
  function: manage file security context
  Usage:
  semanage fcontext [-S store] -{a|d|m|l|n|D} [-frst] file_spec
  semanage fcontext [-S store] -{a|d| m|l|n|D} -e replacement target
  main options:
  -a: add
  -d: delete
  -m: modify
  -l: enumerate
  -n: do not print the description header
  -D: delete all
  -f: file
  -s: User-
  t: type-
  r: role to
   
  view file attributes using the following command:

  # ls -dZ /mywebsite

  Regarding changing the port number
   
  According to the policy configuration, the service may only be allowed to run on a specific port number. Trying to change the port on which the service runs, without changing the policy, may cause the service to fail to start. First check that SELinux allows HTTP to listen on TCP ports, use the command:

  # semanage port -l | grep -w http_port_t
  http_port_t tcp 80, 443, 488, 8008, 8009, 8443

  Use the command to modify:

  # semanage port -a -t http_port_t -p tcp 12345

  Then confirm:

  # semanage port -l | grep -w http_port_t
  http_port_t      tcp      12345, 80, 443, 488, 8008, 8009, 8443

  Client configuration

Modify the source file after backing up the source file

sudo vim CentOS-Base.repo 
[Base]
name=base
baseurl=http://172.16.1.254/centos/7/Base/x86_64
enabled=1
gpgcheck=0
gpgkey=http://172.16.1.254/RPM-GPG-KEY-CentOS-6

sudo vim epel.repo
[epel]
name=epel
baseurl=http://172.16.1.254/epel/7/x86_64
enabled=1
gpgcheck=0
gpgkey=http://172.16.1.254/RPM-GPG-KEY-CentOS-6

Use rsync to synchronize the mirror source site

To use rsync to synchronize the mirror source site, the site needs to support the rsync protocol. I found that China University of Science and Technology and Tsinghua University can be used in China, but the speed of my connection to Tsinghua University is too slow. The source mirror site for example.
 
Another advantage of using rsync for site mirroring is that it can mirror different architecture sources, and even other software sources such as pypi. The following lists the scripts I use synchronously.

Sync Centos 7 with rsync

#!/bin/bash

VER='7'
REPOS=(os updates extras)

for REPO in ${REPOS[@]}
do
    rsync -auvz --delete --exclude='repodata' \
    rsync://rsync.mirrors.ustc.edu.cn/repo/centos/${VER}/${REPO}/x86_64/ ~/repo/centos/${VER}/${REPO}/x86_64/
    createrepo ~/repo/centos/${VER}/${REPO}/x86_64/
done
for REPO in ${REPOS[@]}
do
    rsync -avz --delete --exclude='repodata' \
    rsync://rsync.mirrors.ustc.edu.cn/repo/centos-altarch/${VER}/${REPO}/aarch64/ ~/repo/centos/${VER}/${REPO}/x86_64/
    createrepo ~/repo/centos/${VER}/${REPO}/x86_64/
done

Use rsync to sync Centos Stream 8

#!/bin/bash

VER='8-stream'
ARCHS=(x86_64 aarch64)
REPOS=(AppStream BaseOS HighAvailability PowerTools extras)

for ARCH in ${ARCHS[@]}
do
    for REPO in ${REPOS[@]}
    do
        rsync -avuz --delete --exclude='repodata' \
        rsync://rsync.mirrors.ustc.edu.cn/repo/centos/${VER}/${REPO}/${ARCH}/ ~/repo/centos/${VER}/${REPO}/${ARCH}/
        createrepo ~/repo/centos/${VER}/${REPO}/${ARCH}/
    done
done

Use rsync to synchronize epel

#!/bin/bash

VERS=(7 8)
ARCHS=(x86_64 aarch64)

# sync epel7
for VER in ${VERS[@]}
do
    for ARCH in ${ARCHS[@]}
    do
        rsync -avuz --delete --exclude={'repodata','debug'} \
        rsync://rsync.mirrors.ustc.edu.cn/repo/epel/${VER}/Everything/${ARCH}/ \
        ~/repo/epel/${VER}/Everything/${ARCH}
        createrepo ~/repo/epel/${VER}/Everything/${ARCH}
    done
done

Sync Kubernetes with rsync

#!/bin/bash

REPOS=(kubernetes-el7-x86_64 kubernetes-el7-aarch64 minikube-el7-x86_64)

for REPO in ${REPOS[@]}
do
    rsync -avz --delete --exclude='repodata' \
    rsync://rsync.mirrors.ustc.edu.cn/repo/kubernetes/yum/repos/${REPO} \
    ~/repo/kubernetes/yum/repos
    createrepo ~/repo/kubernetes/yum/repos
done

other instructions

It is relatively simple to use rsync to synchronize. The subsequent Apache setup is similar to the previous one, so I won’t repeat it. In addition, when using rsync to synchronize the source of the University of Science and Technology of China, it is recommended to use the following command to view the specific directory, and the source directory of the root directory repo and centos is explained.

rsync rsync://rsync://rsync.mirrors.ustc.edu.cn/repo/ # 查看根repo目录下的内容
rsync rsync://rsync://rsync.mirrors.ustc.edu.cn/repo/centos # 查看根centos目录下的内容