Big data spark on k8s

spark on k8s architecture analysis

1. Advantages of k8s

k8s is an open source container cluster management system that can realize automatic deployment, automatic expansion and contraction, maintenance and other functions of container clusters.

1. Fault migration

2. Resource scheduling

3. Resource isolation

4. Load balancing

5. Cross-platform deployment

2.k8s cluster architecture

Insert image description here

Master Node

The k8s cluster control node schedules and manages the cluster and accepts cluster operation requests from users outside the cluster;

Master Node consists of API Server, Scheduler, ClusterState Store (ETCD database) and Controller MangerServer

API Server . K8S request entry service . The API Server is responsible for receiving all requests from K8S (from the UI interface or CLI command line tool). Then, the API Server notifies other components to work according to the user's specific requests.
Scheduler . Cluster resource scheduler for all K8S Worker Nodes . When a user wants to deploy a service, it monitors the creation of the pod through watch and is responsible for scheduling the pod to the appropriate node.
Controller Manager . Monitor for all K8S Worker Nodes . Controller Manager has many specific Controllers. The Controller is responsible for monitoring and adjusting the status of the services deployed on the Worker Node. For example, if the user requires service A to deploy two copies, then when one of the services hangs up, the Controller will immediately adjust and let the Scheduler Select another Worker Node to redeploy the service.
etcd . K8S storage service . Data is stored persistently, and the storage cluster includes node, pod, rc, service and other data; it also stores key configurations and user configurations of K8S. Only the API Server in K8S has read and write permissions, and other components must pass the API Server interface. Read and write data.

Worker Node

Cluster working nodes run user business application containers;

Worker Node includes kubelet, kube proxy and ContainerRuntime

Kubelets . Monitor of Worker Node and communicator with Master Node . Kubelet is the "eyeliner" of the Master Node installed on the Worker Node. It will regularly report to the Master Node the status of the services running on its own Node, and accept instructions from the Master Node to take adjustment measures. Kubelet is responsible for the management of images and pods, and is mainly responsible for dealing with container runtimes (such as Docker projects). What this interaction relies on is a remote call interface called CRI (Container Runtime Interface).
Kube-Proxy . K8S network proxy . Kube-Proxy is responsible for Node’s network communication in K8S and load balancing of external network traffic. kube-proxy is an abstraction of service service implementation. It is responsible for maintaining and forwarding pod routes and achieving access to the internal and external networks of the cluster.
Container Runtime . The running environment of Worker Node . That is, the software environment required for containerization is installed to ensure that the containerized program can run, such as Docker Engine. In plain English, it helps install the Docker operating environment and is responsible for implementing container life cycle management.

3. How Spark on K8s works

Insert image description here

The specific process includes the following steps:

①: The user uses kubectl to create the SparkApplication object, submit the sparkApplication request to api-server, and persist the sparkApplication CRD to etcd;

②: SparkApplication controller receives an object request from kube-api server, creates a submission (actually the parameterized spark-submit command), and then sends it to the submission runner.

③: Submission runner submits the app to the k8s cluster and creates a driver pod. Once the driver pod is running normally, the executor pod is created from the driver pod. When the application is running normally, the spark pod monitor monitors the pod status of the application (which can be seen through list and status through kubectl) and sends updates to the pod status to the controller. The controller is responsible for calling kube-api to update the status of the spark application (actually The above is to update the status field of SparkApplication CRD).

④: mutating administration webhook creates svc, you can view spark web ui

k8s installation

1.Environment preparation

hostname	IP
hadoop102	192.168.10.102
Hadoop103	192.168.10.103
Hadoop104	192.168.10.104

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 临时

# 关闭swap
swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

# 根据规划设置主机名
hostnamectl set-hostname <hostname>

# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.10.102 hadoop102
192.168.10.103 hadoop103
192.168.10.104 hadoop104
EOF

# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

2. Install Docker/kubeadm/kubelet on all nodes

$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
$ yum -y install docker-ce-18.06.1.ce-3.el7
$ systemctl enable docker && systemctl start docker
$ docker --version
Docker version 18.06.1-ce, build e68fc7a

$ cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

$ systemctl restart docker

3. Add Alibaba Cloud YUM software source

$ cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

4. Install kubeadm, kubelet and kubectl

The latest version is installed by default. We specify the version number for deployment here.

$ yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
$ systemctl enable kubele

5. Deploy Kubernetes Master

Initialize on the master node (hadoop102)

$ kubeadm init \
  --apiserver-advertise-address=192.168.10.102 \  #主节点ip地址
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.18.0 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16

Since the default pull image address k8s.gcr.io is not accessible in China, the Alibaba Cloud image warehouse address is specified here.

Use the kubectl tool generated when the master node is initialized to execute on other nodes and view node information:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get node

6. Join Kubernetes Node

Execute on hadoop104/hadoop104.

To add a new node to the cluster, execute the kubeadm join command output in kubeadm init:

$ kubeadm join 192.168.10.102:6443 --token esce21.q6hetwm8si29qxwn \
    --discovery-token-ca-cert-hash sha256:00603a05805807501d7181c3d60b478788408cfe6cedefedb1f97569708be9c5

The default token validity period is 24 hours. After expiration, the token is no longer available. At this time, you need to re-create the token. The operation is as follows:

kubeadm token create --print-join-command

7. Deploy CNI network plug-in

When wget kube-flannel.yml shows that the connection failed
, it is because the website is blocked. It is recommended to add a
199.232.68.133 raw.githubusercontent.com to the /etc/hosts file.

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

The default image address is inaccessible, so the sed command is changed to the docker hub image warehouse.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
kube-flannel-ds-amd64-2pc95   1/1     Running   0          72s

spark private image production

1. Build a local private mirror warehouse

yum install docker-registry     ##安装docker-registry 
docker run --name registry -d  -p 5000:5000 --restart=always  -v /opt/registry:/var/lib/registry registry

Use the official registry image to run a private image warehouse container, customize the name registry, -d defaults to the local machine, -p specifies the mapped port 5000, -v specifies the installation directory, and the default is /var/lib/registry of the container. The custom modification directory /opt/registry, and the custom images that will be pushed up later will be in the /opt/registry directory.

2. Build spark3.0.0 on k8s image

Download and install the spark3.0.0 installation package and unzip it

tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /opt/module
cd /opt/module/spark-3.0.0-bin-hadoop3.2/bin

Build the spark image through dockerFile and upload it to your own image warehouse

./bin/docker-image-tool.sh -r <repo> -t my-tag build # repo仓库名称  my-tag版本号
docker images #可以看到已经构建好的镜像

Verify image push and pull of private image repository.

docker push 192.168.10.102:5000/spark   #将自定义镜像spark推送到私有镜像仓库192.168.10.102:5000
docker rmi 192.168.10.102:5000/spark   #为了避免干扰验证结果，先把本地tag的自定义镜像删除
docker pull 192.168.10.102:5000/spark   #从私有镜像仓库拉取自定义镜像
docker images   #查看本地镜像，应该可以再次看到192.168.10.102:5000/spark镜像

You can also access the private image warehouse through a browser through http://[ip address]:[port]/v2/_catalog.

3. Configure spark user permissions

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
##在spark-submit中添加
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

Configure spark history server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-history-server
  namespace: default
spec:
  selector:
    matchLabels:
      run: spark-history-server
  replicas: 1
  template:
    metadata:
      labels:
        run: spark-history-server
    spec:
      containers:
        - image: 192.168.10.102:5000/spark
          name: spark-history-server
          args: ["/opt/spark/bin/spark-class", "org.apache.spark.deploy.history.HistoryServer"]
          ports:
            - containerPort: 18080
              name: http
          env:
          - name: SPARK_HISTORY_OPTS
            value: "-Dspark.history.fs.logDirectory=hdfs://192.168.10.102:8020/sparkhistory"

---
apiVersion: v1
kind: Service
metadata:
  name: spark-hs-svc
  namespace: default
spec:
  ports:
  - port: 18080
    protocol: TCP
    targetPort: 18080
    nodePort: 10080
  selector:
    run: spark-history-server
  type: NodePort
status:
  loadBalancer: {
    
    }

Insert image description here

Running sparkdemo on k8s

bin/spark-submit \
    --master k8s://https://192.168.10.102:6443 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.eventLog.enabled=true \
    --conf spark.eventLog.dir=hdfs://192.168.10.102:8020/sparkhistory \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=2 \
    --conf spark.kubernetes.container.image=192.168.10.102:5000/spark \
 local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar

Insert image description here