"GreenPlum Series - Deployment and Maintenance" super detailed Docker installation GreenPlum stand-alone node tutorial

Super detailed Docker installation GreenPlum stand-alone node tutorial

1. Create a centos container

1.1 Pull centos7 image

docker pull centos:7

[root@zxy ~]# docker pull centos:7
[root@zxy ~]# docker images
REPOSITORY      TAG       IMAGE ID       CREATED         SIZE
centos         7         eeb6ee3f44bd   18 months ago   204MB

1.2 Create container

External port 6002 maps container internal ssh port 22

External port 5432 maps internal port 5432

External port 6000 maps internal port 6000

External port 6001 maps internal port 6001

[root@zxy ~]# docker run -itd --name gp-docker01 \
> -p 6002:22 \
> -p 5432:5432 \
> -p 6000:6000 \
> -p 6001:6001 \
> --privileged eeb6ee3f44bd \
> /usr/sbin/init
b0c0fe56c04889c3d22aed9f422e16647030517b97f787095b70a8bf84f757a4

1.3 Entering the container

[root@zxy ~]# docker exec -it b0c0fe56c048 /bin/bash
[root@b0c0fe56c048 /]#

1.4 Secret-free operation of containers and servers

1.4.1 Generate key

If ssh-keygenthe command cannot be found during execution, then use the yuminstallationopenssh

bash: ssh-keygen: command not found

# 1.ssh-keygen失败
[root@b0c0fe56c048 /]# ssh-keygen
bash: ssh-keygen: command not found
# 2.安装openssh
[root@b0c0fe56c048 /]# yum install openssh
# 3.再次进行生成密钥操作，不用输入，直接点击enter键即可
[root@b0c0fe56c048 /]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
......

1.4.2 Copy key

If the execution ssh-copy-idfails, use manual yuminstallationopenssh-clients

bash: ssh-copy-id: command not found

# 1.拷贝失败
[root@b0c0fe56c048 /]# ssh-copy-id 125.22.95.188
bash: ssh-copy-id: command not found
# 2.安装openssh-clients
[root@b0c0fe56c048 /]# yum install openssh-clients
# 3.再次拷贝，进行免密
[root@b0c0fe56c048 /]# ssh-copy-id 125.22.95.188
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '125.22.95.188 (125.22.95.188)' can't be established.
ECDSA key fingerprint is SHA256:uvxqg9O/HlNw5Y3FNToE/llF8GeZQy/p+GXlIS3N6cY.
ECDSA key fingerprint is MD5:d4:15:a9:ea:f9:26:8f:2f:b4:dd:8a:42:3b:58:29:3b.
Are you sure you want to continue connecting (yes/no)? yes
.....

1.5 Install ssh service and network must apply

[root@b0c0fe56c048 /]# yum -y install net-tools.x86_64
[root@b0c0fe56c048 /]# yum -y install openssh-server
[root@b0c0fe56c048 /]# systemctl restart sshd

1.6 Set the root password for the container

1.6.1 Install passwd application

Install the passwd application, you can set a password for the user of the container, which is convenient for secret-free operation of the machine

[root@b0c0fe56c048 /]# yum -y install passwd
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: ftp.sjtu.edu.cn
 * extras: ftp.sjtu.edu.cn
 * updates: ftp.sjtu.edu.cn
Package passwd-0.79-6.el7.x86_64 already installed and latest version
Nothing to do

1.6.2 Set the password for the root of the container

[root@b0c0fe56c048 /]# passwd root
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

1.7 The container itself is secret-free

[root@b0c0fe56c048 /]# ssh-copy-id localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:9iOpCm67LKlvIZyEUNwxF4BnGlvORfRAbZX79nYkW1M.
ECDSA key fingerprint is MD5:fa:38:d7:e3:40:ad:12:68:94:5b:35:8c:77:8b:f0:bb.
Are you sure you want to continue connecting (yes/no)? ys
Please type 'yes' or 'no': yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@localhost's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'localhost'"
and check to make sure that only the key(s) you wanted were added.

2. Install GreenPlum

2.1 Prepare the installation package

[root@zxy ~]# docker cp /zxy/software/greenplum-db-6.4.0-rhel6-x86_64.rpm b0c0fe56c048:/

2.2 Create users and user groups

[root@b0c0fe56c048 /]# groupadd gpadmin
[root@b0c0fe56c048 /]# useradd gpadmin -g gpadmin
[root@b0c0fe56c048 /]# passwd gpadmin

2.3 Offline installation

2.3.1 Check the installation package and install it with yum

Installation package

official website

The official recommendation is to use yum to install. The advantage of yum installation is that it will automatically download and install dependency packages for us. By default, the greenplum software is installed in the /usr/local directory and a soft link is created. But it will be more troublesome if you can't automatically download from the Internet.

[root@b0c0fe56c048 /]# ls | grep greenplum
greenplum-db-6.4.0-rhel6-x86_64.rpm
[root@b0c0fe56c048 /]# yum install localhost greenplum-db-6.4.0-rhel6-x86_64.rpm

2.3.2 Check the installation path

[root@b0c0fe56c048 /]# ll /usr/local/ | grep greenplum
lrwxrwxrwx  1 root root   29 Mar 22 02:29 greenplum-db -> /usr/local/greenplum-db-6.4.0
drwxr-xr-x 12 root root 4096 Mar 22 02:29 greenplum-db-6.4.0

2.3.3 Grant greenplum directory permission to gpadmin user

[root@b0c0fe56c048 /]# chown -Rf gpadmin:gpadmin /usr/local/greenplum*
[root@b0c0fe56c048 /]# ll /usr/local/ | grep greenplum
lrwxrwxrwx  1 gpadmin gpadmin   29 Mar 22 02:29 greenplum-db -> /usr/local/greenplum-db-6.4.0
drwxr-xr-x 12 gpadmin gpadmin 4096 Mar 22 02:29 greenplum-db-6.4.0

2.3.4 Create data directory and configuration directory

Create a data directory gpdatawith masterand primaryother nodes

Create configuration directorygpconfigs

[root@b0c0fe56c048 /]# mkdir -p /data/gpdata/master
[root@b0c0fe56c048 /]# mkdir -p /data/gpdata/primary
[root@b0c0fe56c048 /]# mkdir -p /data/gpconfigs

[root@b0c0fe56c048 /]# chown -Rf gpadmin:gpadmin /data/

2.3.5 Switch gpadmin user

[root@b0c0fe56c048 /]# su gpadmin

2.3.6 View gpadmin user environment variables

Add in ~/.bashrc filesource /usr/local/greenplum-db/greenplum_path.sh

In this way, after modifying the environment under the root user, once you switch to the gpadmin user, it will be automatically loaded.

When there are other variables that need to be added later, they are also added to this file

[gpadmin@b0c0fe56c048 /]$ cat ~/.bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
source /usr/local/greenplum-db/greenplum_path.sh

Just modified, you can manually source it.source ~/.bashrc

2.3.7 SSH Mutual Trust Settings

[gpadmin@b0c0fe56c048 /]$ gpssh-exkeys -h b0c0fe56c048
[STEP 1 of 5] create local ID and authorize on local host

[STEP 2 of 5] keyscan all hosts and update known_hosts file

[STEP 3 of 5] retrieving credentials from remote hosts

[STEP 4 of 5] determine common authentication file content

[STEP 5 of 5] copy authentication files to all remote hosts

[INFO] completed successfully

2.3.8 Configuration initialization file

2.3.8.1 Copy files to configuration directory

[gpadmin@b0c0fe56c048 /]$ cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config /data/gpconfigs/

2.3.8.2 Create host file

[gpadmin@b0c0fe56c048 /]$ touch /data/gpconfigs/hostfile
[gpadmin@b0c0fe56c048 /]$ echo "b0c0fe56c048" > /data/gpconfigs/hostfile

2.3.8.3 Modify initialization file

Mainly focus on the master and segment directories, hostnames and database ports

[gpadmin@b0c0fe56c048 /]$ vi /data/gpconfigs/gpinitsystem_config

[gpadmin@b0c0fe56c048 /]$ cat /data/gpconfigs/gpinitsystem_config

# FILE NAME: gpinitsystem_config

# Configuration file needed by the gpinitsystem

################################################
#### REQUIRED PARAMETERS
################################################

#### Name of this Greenplum system enclosed in quotes.
ARRAY_NAME="Greenplum Data Platform"

#### Naming convention for utility-generated data directories.
SEG_PREFIX=gpseg

#### Base number by which primary segment port numbers
#### are calculated.
PORT_BASE=6000

#### File system location(s) where primary segment data directories
#### will be created. The number of locations in the list dictate
#### the number of primary segments that will get created per
#### physical host (if multiple addresses for a host are listed in
#### the hostfile, the number of segments will be spread evenly across
#### the specified interface addresses).
declare -a DATA_DIRECTORY=(/data/gpdata/primary /data/gpdata/primary)

#### OS-configured hostname or IP address of the master host.
MASTER_HOSTNAME=b0c0fe56c048

#### File system location where the master data directory
#### will be created.
MASTER_DIRECTORY=/data/gpdata/master

#### Port number for the master instance.
MASTER_PORT=5432

#### Shell utility used to connect to remote hosts.
TRUSTED_SHELL=ssh

#### Maximum log file segments between automatic WAL checkpoints.
CHECK_POINT_SEGMENTS=8

#### Default server-side character set encoding.
ENCODING=UNICODE

################################################
#### OPTIONAL MIRROR PARAMETERS
################################################

#### Base number by which mirror segment port numbers
#### are calculated.
#MIRROR_PORT_BASE=7000

#### File system location(s) where mirror segment data directories
#### will be created. The number of mirror locations must equal the
#### number of primary locations as specified in the
#### DATA_DIRECTORY parameter.
#declare -a MIRROR_DATA_DIRECTORY=(/data1/mirror /data1/mirror /data1/mirror /data2/mirror /data2/mirror /data2/mirror)


################################################
#### OTHER OPTIONAL PARAMETERS
################################################

#### Create a database of this name after initialization.
#DATABASE_NAME=name_of_database

#### Specify the location of the host address file here instead of
#### with the the -h option of gpinitsystem.
#MACHINE_LIST_FILE=/home/gpadmin/gpconfigs/hostfile_gpinitsystem

2.3.9 Initialize the database

[gpadmin@b0c0fe56c048 /]$ gpinitsystem -c /data/gpconfigs/gpinitsystem_config -h /data/gpconfigs/hostfile

2.3.10 Connecting to the database

[gpadmin@b0c0fe56c048 /]$ psql -p 5432 -d postgres
psql (9.4.24)
Type "help" for help.

2.3.11 Change password

[gpadmin@b0c0fe56c048 /]$ psql -p 5432 -d postgres
psql (9.4.24)
Type "help" for help.

postgres=# \password gpadmin
Enter new password:
Enter it again:

3. Remote connection

3.1 Open ports

Query the task process and port through the command.

The 6000, 6001, and 5432 of the container correspond to the 6000, 6001, and 5432 ports of the server, and the main port 5432 is opened to allow external access.

[gpadmin@b0c0fe56c048 /]$ ps -ef | grep greenplum
gpadmin   3560     1  0 02:39 ?        00:00:00 /usr/local/greenplum-db-6.4.0/bin/postgres -D /data/gpdata/primary/gpseg0 -p 6000
gpadmin   3561     1  0 02:39 ?        00:00:00 /usr/local/greenplum-db-6.4.0/bin/postgres -D /data/gpdata/primary/gpseg1 -p 6001
gpadmin   3582     0  0 02:39 ?        00:00:00 /usr/local/greenplum-db-6.4.0/bin/postgres -D /data/gpdata/master/gpseg-1 -p 5432 -E
gpadmin   3725   405  0 02:43 pts/1    00:00:00 grep --color=auto greenplum

3.2 Modify pg_hba.conf

#添加如下一行即可，所有用户，所有IP均可访问
#生产环境中可自行选择配置
[gpadmin@b0c0fe56c048 /]$ vi /data/gpdata/master/gpseg-1/pg_hba.conf

host     all         all         all       trust

3.3 restart greenplum

3.3.1 gpstop shutdown failed

20230322:02:45:30:003728 gpstop:b0c0fe56c048:gpadmin-[CRITICAL]:-gpstop failed. (Reason='Environment Variable MASTER_DATA_DIRECTORY not set!') exiting...

When using gpstop, it prompts that the shutdown failed, prompting you to setMASTER_DATA_DIRECTORY

[gpadmin@b0c0fe56c048 /]$ gpstop
20230322:02:45:30:003728 gpstop:b0c0fe56c048:gpadmin-[INFO]:-Starting gpstop with args:
20230322:02:45:30:003728 gpstop:b0c0fe56c048:gpadmin-[INFO]:-Gathering information and validating the environment...
20230322:02:45:30:003728 gpstop:b0c0fe56c048:gpadmin-[CRITICAL]:-gpstop failed. (Reason='Environment Variable MASTER_DATA_DIRECTORY not set!') exiting...

3.3.2 Set MASTER_DATA_DIRECTORY

~/.bashrcAdd the following two lines of records to the file

[gpadmin@b0c0fe56c048 /]$ vi ~/.bashrc

MASTER_DATA_DIRECTORY=/data/gpdata/master/gpseg-1
export MASTER_DATA_DIRECTORY

[gpadmin@b0c0fe56c048 /]$ source ~/.bashrc

3.4 Turn off gp

[gpadmin@b0c0fe56c048 /]$ gpstop
.....
20230322:02:46:57:003750 gpstop:b0c0fe56c048:gpadmin-[INFO]:-Cleaning up leftover shared memory

3.5 start gp

[gpadmin@b0c0fe56c048 /]$ gpstart
......
20230322:02:47:21:003933 gpstart:b0c0fe56c048:gpadmin-[INFO]:-Database successfully started

3.6 Remote connection

Dbeaver uses the GreenPlum driver to connect to the database