4.2 k8s
---概念---
Kubernetes运行容器（Pod）与访问容器（Pod）这两项任务分别由Controller（常用有Deployment、ReplicaSet、DaemonSet、StatefuleSet、Job等）和Service执行。
Deployment控制RS，RS控制Pod，这一整套，向外提供稳定可靠的Service
------api----------------
获取api列表：
curl http://192.168.0.8:8080
查询某资源使用的api：
kubectl edit rs -n test a0003042-3875036520
curl命令查询：
http:
curl -i http://192.168.0.8:8080/apis/extensions/v1beta1/namespaces/test/replicasets
某具体rs
curl -i http://192.168.0.8:8080/apis/extensions/v1beta1/namespaces/test/replicasets/pod-xxxx
https：
TOKEN='xxx'
curl -H "Authorization: Bearer $TOKEN" --insecure https://192.168.0.8:6443/api/v1/namespaces/test/pods
某具体pod
curl -H "Authorization: Bearer $TOKEN" --insecure https://192.168.0.8:6443/api/v1/namespaces/test/pods/pod-xxxx
参数化查询
curl -i http://192.168.0.8:8080/apis/extensions/v1beta1/namespaces/test/replicasets/?labelSelector=app=appTest\&pretty=No|less
?labelSelector=app=cscloud-datastore #标签选择器
\&pretty=No #多参数使用&，shell中需转义
使用python3读取json，-s取消错误输出
curl -s http://192.168.0.8:8080/apis/extensions/v1beta1/namespaces/test/replicasets/?labelSelector=app=appTest|python3 -c "import sys, json;print(json.load(sys.stdin)['items'][0]['metadata']['creationTimestamp'])"
-----------kubectl-------
添加自动补全：
echo "source <(kubectl completion bash)" >> ~/.bashrc
------
get:
获取组件状态:
kubectl get componentstatus
获取所有namespace：
kubectl get ns
在指定的namespace下获取资源：
kubectl -n {$nameSpace} get pods
以yaml格式输出资源：
kubectl get {$sourceType} -n {$nameSpace} -o yaml #还可以 -o json
通用格式：
kubectl get {$sourceType} --all-namespaces
常用的资源类型（{$resourceType}）有：
po（pod）
ns（命名空间namespace）
instance（实例）
svc（service服务）：定义了一个 Pod 的逻辑分组，一种可以访问它们的策略（微服务）。
cm（configMap）：存储全局配置变量的，将分布式系统中不同模块的环境变量统一到一个对象中管理。
ds（deamonSet）：在每台计算节点上运行一个守护进程（如日志采集等）,有时pod处于pending可能是因为某个deamonSet没起来。
deploy（deployment）：用于启动（上线/部署）一个Pod或者ReplicaSet。这个如果有问题，那么其他依赖它来部署的资源就肯定不会正常了。
rs （replicasets）
指定标签：
kubectl get service -n test -l app=test
指定pod中特定container：
kubectl logs pod/tomcat-xxx-xxx -c filebeat
describe:
kubectl -n {$nameSpace} describe {$resourceType} {$resourceName}
kubectl describe -n test pod tomcat-xxx-xxx
查看口令
kubectl -n kube-system describe secret
kubectl -n kube-system describe secret|grep -E 'dashboard-read-user-token|admin-user-token' -A 12
taint(污点)：
查看node描述：
kubectl describe node 192.168.0.8
语法：
kubectl taint node [node] key=value[effect]
其中[effect] 可取值: [ NoSchedule | PreferNoSchedule | NoExecute ]
NoSchedule: 一定不能被调度
PreferNoSchedule: 尽量不要调度
NoExecute: 不仅不会调度, 还会驱逐Node上已有的Pod
添加/删除node污点
kubectl taint nodes 192.168.0.8 key1=value1:NoSchedule
kubectl taint nodes 192.168.0.8 key1:NoSchedule- # 这里的key可以不用指定value
kubectl taint nodes 192.168.0.8 key1- #删除指定key所有的effect
添加master污点（可以直接停掉kubelet.service）：
kubectl taint nodes 192.168.0.8 node-role.kubernetes.io/master=:NoSchedule #node-role.kubernetes.io/master为key, value为空, effect为NoSchedule
logs:
kubectl -n {$nameSpace} logs --tail=1000 {$podName} | less
kubectl logs -n test tomcat-xxx-xxx --tail=100 -f (-f 动态)
edit:
编辑一个资源文件，这里以编辑configMap资源对象为例（yaml格式）：
kubectl -n {$nameSpace} edit {$resourceType} {$resourceName} -o yaml
kubectl edit pod -n test tomcat-xxx-xxx
Dashboard的Token超时时间
kubectl edit deployment kubernetes-dashboard -n kube-system
args:
- --auto-generate-certificates
- --token-ttl=43200
编辑最大副本数：
kubectl -n test edit rs tomcat-xxx-xxx
delete：
删除所有deployment
kubectl delete deployment,svc -n test --all
删除所有pod（重启）：
kubectl delete pod -n test --all
删除指定pod:
kubectl -n {$nameSpace} delete {$resourceType} {$resourceName}
exec:
kubectl exec POD -n {$nameSpace} -it -- COMMAND
-i, --stdin=false: Pass stdin to the container
-t, --tty=false: Stdin is a TTY
kubectl exec -n test tomcat-xxx-xxx -it -- bash
cp:
kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar
run:
#运行终端，kubectl无-d选项
kubectl run alpine --image=mytest:5000/alpine
label:
kubectl label node 172.17.1.5 type=node
#删除标签：
kubectl label node 172.17.1.5 type-
demployment:
升级版本：
kubectl set image deployment -n ${namespace} ${deployment_name} ${deployment_name}=${image_url}
drain:
#排干某节点pod
kubectl drain 172.17.1.6 --delete-local-data --force --ignore-daemonsets
cluster-info
kubectl cluster-info
kubectl cluster-info dump|less
---------k8s-yml-------------
--使用：
#应用规则-f 指定文件
kubectl apply -f nginx.yml
#直接编辑已运行
kubectl edit deployment nginx-deployment
--镜像拉取规则
imagePullPolicy:IfNotPresent 镜像拉取规则，Always为始终拉取
--资源分配：
apiVersion: v1
kind: Pod
metadata:
name: webapp
labels:
app: webapp
spec:
containers:
- name: webapp
image: tomcat
resources:
limits: #Limits是容器最多能使用的资源量的上限
cpu: 2
memory: 2Gi
requests: #表示容器希望被分配到的、可完全保证的资源
cpu: 100m #cpu的值可以不带单位(如0.5表示半个cpu)，也可以用m (millicpu), 0.5CPU等价于500m
memory: 200Mi
ports:
- containerPort: 8080
------------DaemonSet------------------
以下来源与kubeasz中的flannel配置
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "{
{ CLUSTER_CIDR }}",
"Backend": {
{% if FLANNEL_BACKEND == "vxlan" and DIRECT_ROUTING %}
"DirectRouting": true,
{% endif %}
"Type": "{
{ FLANNEL_BACKEND }}"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: {
{ flanneld_image }}
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: {
{ flanneld_image }}
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---NFS PersistentVolume 挂载----------
1 ，创建pv: kubectl apply -f test-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: test-pv
namespace: test
spec:
capacity:
storage: 1Gi #PV的容量为1GB
accessModes:
- ReadWriteOnce #ReadWriteOnce表示PV能以read-write模式mount到单个节点， ReadOnlyMany表示PV能以read-only模式mount到多个节点， ReadWriteMany表示PV能以read-write模式mount到多个节点
persistentVolumeReclaimPolicy: Recycle #PV的回收策略：Retain表示需要管理员手工回收； Recycle表示清除PV中的数据，效果相当于执行rm -rf /thevolume/*； Delete表示删除Storage Provider上的对应存储资源，例如AWS EBS、 GCE PD、Azure Disk、OpenStack Cinder Volume等。
storageClassName: nfs #指定PV的class为nfs
nfs:
path: /nfsdata/pv1
server: 172.17.16.148
查看： kubectl get pv
2，创建pvc: kubectl apply -f test-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mypvc1
namespace: test
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: nfs
查看： kubectl get pvc
3，使用PVC
kubectl edit deployment -n test util-app
volumes:
- name: mydata
persistentVolumeClaim:
claimName: mypvc1
----------configmap/挂载-----------
--创建：
文件方式：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: kube-ops
data:
prometheus.yml: | #这种会生成一个文件
global:
scrape_interval: 15s
scrape_timeout: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
...
变量方式：
apiVersion: v1
data:
KEY: value #直接传递变量
....
kind: ConfigMap
metadata:
...
--引用：
文件方式：
...
ports:
- containerPort: 9090 #pod中的端口
name: http #名字，此处和svc中要对应
volumeMounts:
- mountPath: "/prometheus" #要挂载至pod中的哪个目录
subPath: prometheus
name: data #name，需要和volumes中对应
- mountPath: "/etc/prometheus"
name: config
....
volumes:
- name: config
configMap: #挂载cm类型
name: prometheus-config #引用cm名字
- name: data
persistentVolumeClaim: #挂载PVC
claimName: prometheus
变量方式：
......
containers:
- image: ${app_image}
name : tomcat-xxx
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef: #使用env方式引用CM，会加载cluster-config中的变量到container中
name: cluster-config
--------------主机目录挂载-------------------------------------
.....
volumeMounts:
- mountPath: "/app/dynatrace-6.5" #挂载至pod中的哪个目录
name: dtagent #name，需要和volumes中对应
- mountPath: "/app/logs"
name: applog
- name: initscript-volume
mountPath: "/app/bin"
...
volumes:
- name: applog
hostPath: #挂载主机目录
path: /app/logs
- name: dtagent
hostPath:
path: /app/dynatrace-6.5
- name: initscript-volume
configMap: #挂载cm类型
name: init-script #引用cm名字
----指定节点-----
--通过节点名指定：
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: tomcat-xxx
spec:
nodeName: 172.17.1.4 ##
--通过标签指定：
1，kubectl label nodes 107 type=backEndNode1 ##
2，kubectl apply -f tomcat.yml ：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myweb
spec:
replicas: 2
template:
metadata:
labels:
app: myweb
spec:
nodeSelector: ##
type: backEndNode1
-------------端口映射deployment/svc--------------
deployment:
spec:
containers:
- env:
...
ports:
- containerPort: 8080
name: http-port
protocol: TCP
svc:
apiVersion: v1
kind: Service
metadata:
.....
spec:
selector:
...
clusterIP: ...
externalTrafficPolicy: Cluster
ports:
- nodePort: 30948
port: 30948
protocol: TCP
targetPort: 8080
.......
type: NodePort
------------一个pod多个container(例：filebeat)----
--tomcat.yml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: tomcat-xxx
namespace: default #default可以省略
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: tomcat-xxx
spec:
nodeName: 172.17.1.4 #指定node节点（只有这台手动导入了镜像……）
containers: #指定了两个container
- image: filebeat:7.2.1
imagePullPolicy: IfNotPresent #本地不存在则拉取镜像
name: filebeat
volumeMounts:
- name: app-logs
mountPath: /log # A: filebeat容器中挂载到/log/目录
- name: filebeat-config
mountPath: /etc/filebeat # B: 挂载configMap，至container中/etc/filebeat目录（注：因为是挂载，原目录里的东西会被清空，filebeat重新bulid了一个-c指定配置文件版本）
- image: ${app_image}
name : tomcat-xxx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
volumeMounts:
- name: app-logs
mountPath: /app/software/apache-tomcat-9.0.31/logs # A: tomcat容器中挂载到/app/software/apache-tomcat-9.0.31/logs目录
volumes:
- name: app-logs # A: 创建一个共同挂载目录
emptyDir: {}
- name: filebeat-config # B: 挂载configMap
configMap:
name: filebeat-config
--configmap_filebeat.yml:
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config #引用CM时，要使用此名字
data:
filebeat.yml: | #创建filebeat.yml文件，后面是文件内容
max_procs: 2
........
-- Dockerfile
FROM filebeat:7.2.0
CMD ["filebeat","-e","-c","/etc/filebeat/filebeat.yml"]
-------------etcdctl--------------
etcd2:
export ETCDCTL_API=2
配置etcd内网信息:
etcdctl set /k8s/network/config '{"Network": "10.255.0.0/16"}'
这个配置，与FLANNEL_ETCD_PREFIX的配置一致
更换etcd版本：
export ETCDCTL_API=3
etcdctl version
查看集群：
etcdctl member list
etcdctl --cert-file /etc/etcd/ssl/etcd.pem --key-file /etc/etcd/ssl/etcd-key.pem --ca-file /etc/kubernetes/ssl/ca.pem member list
etcdctl endpoint status
etcdctl endpoint health
使用证书：
etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem member list
etcdctl --endpoints=https://172.17.1.50:2379,https://172.17.1.51:2379,https://172.17.1.52:2379 --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --ca-file=/etc/kubernetes/ssl/ca.pem member list
增删改：
get Gets the key or a range of keys
put Puts the given key into the store
del Removes the specified key or range of keys [key, range_end)
etcd3:
3和2版本不兼容，数据无法互相查看
#更换版本为3
export ETCDCTL_API=3
#指定证书，指定endpoints访问
etcdctl --cert="/etc/etcd/ssl/etcd.pem" --key="/etc/etcd/ssl/etcd-key.pem" --cacert="/etc/kubernetes/ssl/ca.pem" --endpoints=https://172.17.1.50:2379,https://172.17.1.51:2379,https://172.17.1.52:2379 member list
#3无ls命令，可以通过这种方式查看
etcdctl --cert="/etc/etcd/ssl/etcd.pem" --key="/etc/etcd/ssl/etcd-key.pem" --cacert="/etc/kubernetes/ssl/ca.pem" --endpoints=https://172.17.1.50:2379,https://172.17.1.51:2379,https://172.17.1.52:2379 get / --prefix --keys-only|less
----------其他----------
重启dashborad（k8s集群的WebUI）：
kubectl get pod -n kube-system |grep dashboard
kubectl delete pod -n kube-system kubernetes-dashboard-podname #上面查出的pod name
重启master服务：
systemctl restart etcd
systemctl restart kube-apiserver
systemctl restart kube-controller-manager
systemctl restart kube-scheduler
重启node节点服务：
systemctl restart docker.service ; systemctl restart kubelet.service ; systemctl restart kube-proxy.service; systemctl restart flannel.service
for i in $(kubectl get node|grep NotReady|awk '{print $1}');do ssh -t $i -C 'systemctl restart docker.service;systemctl restart flannel.service;systemctl restart kubelet.service;systemctl restart kube-proxy.service';done
重启非running状态pod：
kubectl get pods -n test -o wide|grep -v Running|awk '{print $1}'|xargs kubectl delete pod -n test
--删除节点
#master上执行:
kubectl drain 172.17.1.6 --delete-local-data --force --ignore-daemonsets #排干node上的pod
kubectl delete node 172.17.1.6 #删除node
#node上执行：
systemctl stop docker
systemctl stop kubelet
systemctl stop flannel
systemctl stop kube-proxy
systemctl disable docker
systemctl disable kubelet
systemctl disable flannel
systemctl disable kube-proxy
-------错误------------
报错：
kube-apiserver启动时报错
查看：/var/log/messages
kube-apiserver: unable to load server certificate: open /srv/kubernetes/server.cert: no such file or directory
解决：
缺少证书，复制之前备份证书至日志中目录
cp -a xxx/kube_bak/ca-cert/* /srv/kubernetes/*
报错：
"DON'T BIND ON ANY IP..."
解决：
清理docker数据目录
mkdir /var/lib/docker/bak && mv /var/lib/docker/* /var/lib/docker/bak
报错：
unable to configure the Docker daemon with file /etc/docker/daemon.json
问题：
最终发现是/etc/docker/daemon.json 和/lib/systemd/system/docker.service 对insecure-registries的设置冲突。
解决：
保留insecure-registries的设置，删除/etc/docker/daemon.json的文件
报错：
]kubectl logs -n kube-system kube-dns-xxxx
Error from server (BadRequest): a container name must be specified for pod kube-dns-xxx, choose one of: [kube2sky skydns]
解决这个pod内有两个容器，需要选择一个：
]kubectl logs -n kube-system kube-dns-xxx kube2sky
**************kubeasz****************
1 个配置：/etc/ansible/hosts
.....
1 键安装：ansible-playbook /etc/ansible/90.setup.yml
命令行工具 easzctl
#清理集群
easzctl checkout test
easzctl destroy
报错：
ERROR! Unexpected Exception, this is probably a bug: (cryptography 0.8.2 (/usr/lib64/python2.7/site-packages), Requirement.parse('cryptography>=1.1'))
yum install python2-cryptography.x86_64
sysctl -p /etc/sysctl.d/95-k8s-sysctl.conf 报错：sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
modprobe bridge
验证：ls /proc/sys/net/bridge
缺少net-addr:
yum install -y python-netaddr.noarch