Flink Docker file production

1 Dockerfile, you need to define a Dockerfile, which defines everything the process needs. The content involved in Dockerfile includes executing code or files, environment variables, dependent packages, runtime environment, dynamic link library, operating system distribution, service process and kernel process (when the application process needs to deal with system services and kernel processes, this Need to consider how to design namespace permission control), etc.;
2 Docker image, after defining a file with Dockerfile, a Docker image will be generated during docker build, and when the Docker image is run, it will actually start to provide services; 3 Docker container
, Containers provide services directly.

1. The file name should be lowercase

2、

$ docker build -t mybuilder:v1 -f ./build/Dockerfile .

//-t: Specify the name of the new Dockerfile

//-f: specify the location of the dockerfile

3、docker build -t flink16 -f /opt/dockerfile/flinkdockerfile .

4、docker images

5. flink docker code

flink-docker/1.15/scala_2.12-java8-ubuntu/Dockerfile at master · apache/flink-docker · GitHub

# Use the official Flink base image as the base
FROM flink:1.14.6
#
## Define the working directory
WORKDIR /opt/flink/lib
#
# Add the locally dependent JAR package to the directory
COPY /opt/flink/flink-1.14.6 /lib/dlink-client-1.14-0.7.3.jar .
COPY /opt/flink/flink-1.14.6/lib/dlink-client-base-0.7.3.jar .
COPY /opt/flink/flink-1.14 .6/lib/dlink-common-0.7.3.jar .
COPY /opt/flink/flink-1.14.6/lib/dlink-metadata-base-0.7.3.jar .
COPY /opt/flink/flink-1.14 .6/lib/dlink-metadata-doris-0.7.3.jar .
COPY /opt/flink/flink-1.14.6/lib/dlink-metadata-mysql-0.7.3.jar .
COPY /opt/flink/flink -1.14.6/lib/dlink-metadata-sqlserver-0.7.3.jar .
COPY /opt/flink/flink-1.14.6/lib/druid-1.2.8.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-connector-jdbc_2.12-1.14.6.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-dist_2.12-1.14.6.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-doris-connector-1.14_2.12-1.1.1.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-faker-0.5.3.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-shaded-guava-30.1.1-jre-16.0.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-sql-connector-mysql-cdc-2.3.0.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-sql-connector-sqlserver-cdc-2.3.0.jar .
COPY /opt/flink/flink-1.14.6/lib/flink-table_2.12-1.14.6.jar .
COPY /opt/flink/flink-1.14.6/lib/HikariCP-4.0.3.jar .
COPY /opt/flink/flink-1.14.6/lib/jackson-datatype-jsr310-2.13.4.jar .
COPY /opt/flink/flink-1.14.6/lib/mysql-connector-java-8.0.28.jar .

Then execute the command

docker build -t flink14 -f /opt/dockerfile/flinkdockerfile .

The COPY instruction will copy files/directories from <source_path> in the build context directory to <destination_path> within the new layer's mirror.

<target path> can be an absolute path in the container, or a relative path relative to the working directory (the working directory can be specified with the WORKDIR directive).

The target path does not need to be created in advance, if the directory does not exist, it will be copied before the file

Create missing directories.

COPY <source path>... <destination path>

COPY ["<source path1>",... "<destination path>"]

correct:

COPY ./package.json /app/

COPY package.json /usr/src/app/

mistake:

COPY ../package.json /app

or COPY /opt/xxxx /app

Notice:

In addition, it should be noted that when using the COPY command, various metadata of the source file will be preserved. Such as read, write, execute
permissions, file change time, etc.

Modified to:

# Use the official Flink base image as the base
FROM flink:1.14.6
#
## Define the working directory
WORKDIR /opt/flink
#
# Add the locally dependent JAR package to the directory
COPY ./flinklib/dlink-client-1.14-0.7. 3.jar /opt/flink/lib/dlink-client-1.14-0.7.3.jar
COPY ./flinklib/dlink-client-base-0.7.3.jar /opt/flink/lib/dlink-client-base- 0.7.3.jar
COPY ./flinklib/dlink-common-0.7.3.jar /opt/flink/lib/dlink-common-0.7.3.jar
COPY ./flinklib/dlink-metadata-base-0.7.3. jar /opt/flink/lib/dlink-metadata-base-0.7.3.jar
COPY ./flinklib/dlink-metadata-doris-0.7.3.jar /opt/flink/lib/dlink-metadata-doris-0.7. 3.jar
COPY ./flinklib/dlink-metadata-mysql-0.7.3.jar /opt/flink/lib/dlink-metadata-mysql-0.7.3.jar
COPY ./flinklib/dlink-metadata-sqlserver-0.7.3.jar /opt/flink/lib/dlink-metadata-sqlserver-0.7.3.jar
COPY ./flinklib/druid-1.2.8.jar /opt/flink/lib/druid-1.2.8.jar
COPY ./flinklib/flink-connector-jdbc_2.12-1.14.6.jar /opt/flink/lib/flink-connector-jdbc_2.12-1.14.6.jar
COPY ./flinklib/flink-dist_2.12-1.14.6.jar /opt/flink/lib/flink-dist_2.12-1.14.6.jar
COPY ./flinklib/flink-doris-connector-1.14_2.12-1.1.1.jar /opt/flink/lib/flink-doris-connector-1.14_2.12-1.1.1.jar
COPY ./flinklib/flink-faker-0.5.3.jar /opt/flink/lib/flink-faker-0.5.3.jar
COPY ./flinklib/flink-shaded-guava-30.1.1-jre-16.0.jar /opt/flink/lib/flink-shaded-guava-30.1.1-jre-16.0.jar
COPY ./flinklib/flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar /opt/flink/lib/flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar
COPY ./flinklib/flink-sql-connector-mysql-cdc-2.3.0.jar /opt/flink/lib/flink-sql-connector-mysql-cdc-2.3.0.jar
COPY ./flinklib/flink-sql-connector-sqlserver-cdc-2.3.0.jar /opt/flink/lib/flink-sql-connector-sqlserver-cdc-2.3.0.jar
COPY ./flinklib/flink-table_2.12-1.14.6.jar /opt/flink/lib/flink-table_2.12-1.14.6.jar
COPY ./flinklib/HikariCP-4.0.3.jar /opt/flink/lib/HikariCP-4.0.3.jar
COPY ./flinklib/jackson-datatype-jsr310-2.13.4.jar /opt/flink/lib/jackson-datatype-jsr310-2.13.4.jar
COPY ./flinklib/mysql-connector-java-8.0.28.jar /opt/flink/lib/mysql-connector-java-8.0.28.jar

RUN chown -R flink:flink /opt/flink/lib

ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["/bin/bash"]

It is found that the error of abnormal restart (CrashLoopBackOff) will still be reported. It is suspected that the image package is too large

change into:

# Use the official Flink base image as the base
FROM flink:1.14.6

# Recommended Disabled JAR Encryption
COPY ./flinklib/*.jar /opt/flink/lib
RUN sed -i 's/jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA ,/jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA,/g' /opt/java/openjdk/lib/security/java.security
RUN sed -i 's/DH keySize < 1024, EC keySize < 224,000; 3DES_EDE_CBC , anon , NULL , / DH keySize < 1024 , EC keySize < 224 , anon , NULL , / g ' / opt / java / openjdk / lib / security / java . security RUN
mkdir -p / flink - data / checkpoints && mkdir -p /flink-data/savepoints && chown -R flink:flink /flink-data/checkpoints/ && chown -R flink:flink /flink-data/savepoints/

Hit tag to construct local image

docker build -t flink-sql:1.14.6 -f /opt/dockerfile/Dockerfile .

success after executing the command

docker tag flink-sql:1.14.21 192.168.1.249:16443/bigdata/flink-sql:1.14.21

docker push 192.168.1.249:16443/bigdata/flink-sql:1.14.21

docker pull 192.168.1.249:16443/bigdata/flink-sql:1.14.21

delete mirror

docker rmi -f 3f5caf0973e0

cubectl get pod -n flink -owide

kubectl delete -f sql-application.yaml -n flink

kubectl apply -f sql-application.yaml -n flink

kubectl delete -f session-deployment-only.yaml -n flink

ErrImagePull exception:

vim /etc/containerd/config.toml

Revise:

[plugins."io.containerd.grpc.v1.cri".registry.auths]

[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.1.249:16443".tls]
insecure_skip_verify = true # 是否跳过安全认证
[plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.1.249:16443".auth]
username = "admin"
password = "Harbor12345"
[plugins."io.containerd.grpc.v1.cri".registry.headers]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.1.249:16443"]
endpoint = ["https://192.168.1.249:16443"]

At the same time: the image mirror is changed to download from the https address

#Flink Session cluster source code please go to
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
namespace: flink
name: session-deployment-only
spec:
#imagePullSecrets:
# - name: flink
image: 192.168.1.249:16443/bigdata /flink-sql:1.14.20
#image: flink:1.14.6
flinkVersion: v1_14
imagePullPolicy: IfNotPresent # Image pull policy, if not locally, pull
ingress from warehouse: # ingress configuration, used to access flink web page
template: " flink.k8s.io"
className: "nginx"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/"
flinkConfiguration:
taskmanager.numberOfTaskSlots: "36"
state.savepoints.dir: file:///flink-data/savepoints
state.checkpoints.dir: file:///flink-data/checkpoints
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1
taskManager:
replicas: 1
resource:
memory: "6048m"
cpu: 4

systemctl daemon-reload

systemctl restart containerd.service

View exception:

kubectl describe pod sql-session-7c98d89b5c-crmg4 -n flink

kubectl exec -it session-deployment-only-79b8d79c4c-6ttm2 /bin/bash -n flink

View the port allocated by nginx

kubectl get all -n ingress-nginx -owide

In Docker, "PV" stands for Persistent Volume (Persistent Volume), and "PVC" stands for Persistent Volume Claim (Persistent Volume Claim). They are the mechanism used for data persistence in Docker containers.

Persistent Volume (PV) is a persistent storage resource in Docker, usually supported by the underlying storage system (such as local disk, network storage, etc.). A PV can exist independently in a Docker cluster and can be shared by multiple containers.

A Persistent Volume Claim (PVC) is a request for a persistent volume that defines a container's storage resource needs. Docker uses PVCs to dynamically allocate and manage PVs to meet the persistent data storage needs of containers. PVC can set storage capacity, access mode and other properties.

By associating a PVC with a container, the PV can be accessed by using a mount point in the container to achieve persistent storage and sharing of data.

This mechanism of using PVs and PVCs ensures that containers can retain and restore data when they are restarted, redeployed, or migrated.

"EXPOSE 6123 8081" is a directive in the Dockerfile to declare the network port that the container will use when running.

In this example, the Docker container will expose two ports, namely 6123 and 8081. This means that these two ports will be available for network communication when running the container.

It should be noted that the EXPOSE command only declares in the Docker image that the container will use these ports, it does not automatically map the ports of the host. To implement port mapping between the host and the container, you also need to use the "-p" or "-P" parameter when Docker runs the command.

For example, you can use the following command to start a container and map port 6123 of the host to port 6123 of the container:

```
docker run -p 6123:6123 <image_name>
```

In this way, port 6123 on the host machine can access port 6123 of the container through the network. Similarly, you can map port 8081 in a similar way.

kubectl get -n hefty secret/flink -o yaml

Pack:

docker save -o flink-sql.tar flink-sql:1.14.20

load image:

docker load -i flink-sql.tar

Install Docker Registry

1: Set /etc/docker/daemon.json file.

After creating and modifying the daemon.json file, you need to make this file take effect

vim /etc/docker/daemon.json

{

"insecure-registries" : ["localhost:5000"]

}

a. Reload the configuration file after modification

sudo systemctl daemon-reload

b. Restart the docker service

sudo systemctl restart docker.service

c. View status

sudo systemctl status docker -l

d. View service

sudo docker info

Install Docker Registry

docker run -d -p 5000:5000 --restart=always --name registry registry:2

in:

The -d parameter means to run in the background
The -p parameter maps port 5000 of the host to port 5000 of the container
The --restart=always parameter indicates that the Docker Registry container will automatically start when the Docker daemon starts
The --name registry parameter specifies the name of the container
registry:2 is the image name and version number of Docker Registry