一、前言
使用kops,可以在赋予kops特定AWS权限的情况下,为你自动在AWS部署EC2虚拟机并在虚拟机上安装kubernetes。
而使用Kubernetes从1.9版本开始的对于NLB的支持,可以直接创建基于AWS NLB的kubernetes service,获取AWS分配的外部IP对外进行访问。
转载自https://blog.csdn.net/cloudvtech
二、使用kops安装kubernetes
2.1 配置环境
yum install wget wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm yum install ./epel-release-latest-*.noarch.rpm yum -y update yum -y install python-pip
2.2 安装aws-cli
pip install --upgrade pip pip install awscli --upgrade —user export PATH=~/.local/bin:$PATH
2.3 安装kops
wget https://github.com/kubernetes/kops/releases/download/1.9.0/kops-linux-amd64 mv kops-linux-amd64 /bin/kops
2.4 登陆原来的AWS账号
aws configure AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXX AWS Secret Access Key [None]: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY Default region name [None]: us-west-1 Default output format [None]:
2.5 建立一个新的AWS group
aws iam create-group --group-name kops { "Group": { "Path": "/", "CreateDate": "2018-05-22T05:25:51.653Z", "GroupId": "XXXXXXXXXXXXXXXX", "Arn": "arn:aws:iam::1234567890:group/kops", "GroupName": "kops" } }
2.6 给AWS group赋予权限
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops
2.7 建立一个AWS用户并且将用户加入权限group
aws iam create-user --user-name kops { "User": { "UserName": "kops", "Path": "/", "CreateDate": "2018-05-22T05:26:16.080Z", "UserId": "XXXXXXXXXXXXXXXX", "Arn": "arn:aws:iam::1234567890:user/kops" } } aws iam add-user-to-group --user-name kops --group-name kops2.8 建立kops这个用户的access key
aws iam create-access-key --user-name kops { "AccessKey": { "UserName": "kops", "Status": "Active", "CreateDate": "2018-05-22T05:26:26.089Z", "SecretAccessKey": "xxxxxxxxxxxxxxxxxxxxx", "AccessKeyId": "yyyyyyyyyyyyyyyyyyyyyyy" } }
2.9 使用aws-cli登陆这个新用户
aws configure AWS Access Key ID [****************7LHA]: xxxxxxxxxxxxxx AWS Secret Access Key [****************2dyW]: yyyyyyyyyyyyyyyy Default region name [us-west-1]: us-west-1a Default output format [None]:
2.10 建立S3
export NAME=cluster.k8s.local aws s3api create-bucket --bucket ${NAME}-state-store --create-bucket-configuration LocationConstraint=$AWS_REGION { "Location": "http://cluster.k8s.local-state-store.s3.amazonaws.com/" } export KOPS_STATE_STORE=s3://cluster.k8s.local-state-store
2.11 使用kops建立kubernetes cluster
/usr/local/bin/kops create cluster \ --name=${NAME} \ --image=ami-18726478 \ --zones=us-west-1a \ --master-count=1 \ --master-size="t2.xlarge" \ --node-count=2 \ --node-size="t2.xlarge" \ --vpc=vpc-bbbbbbbbb \ --networking=calico \ --ssh-public-key="~/.ssh/id_rsa.pub"
最后的log:
Must specify --yes to apply changes Cluster configuration has been created. Suggestions: * list clusters with: kops get cluster * edit this cluster with: kops edit cluster cluster.k8s.local * edit your node instance group: kops edit ig --name=cluster.k8s.local nodes * edit your master instance group: kops edit ig --name=cluster.k8s.local master-us-west-1a Finally configure your cluster with: kops update cluster cluster.k8s.local --yes
2.12 启动cluster
kops update cluster cluster.k8s.local --yes
logs:
I0522 05:44:52.534291 20089 apply_cluster.go:456] Gossip DNS: skipping DNS validation W0522 05:44:52.563080 20089 firewall.go:249] Opening etcd port on masters for access from the nodes, for calico. This is unsafe in untrusted environments. I0522 05:44:52.761542 20089 executor.go:91] Tasks: 0 done / 79 total; 32 can run I0522 05:44:53.160191 20089 vfs_castore.go:731] Issuing new certificate: "apiserver-aggregator-ca" I0522 05:44:53.270980 20089 vfs_castore.go:731] Issuing new certificate: "ca" I0522 05:44:53.418228 20089 executor.go:91] Tasks: 32 done / 79 total; 23 can run I0522 05:44:53.741454 20089 vfs_castore.go:731] Issuing new certificate: "kubecfg" I0522 05:44:53.805113 20089 vfs_castore.go:731] Issuing new certificate: "kubelet" I0522 05:44:53.885277 20089 vfs_castore.go:731] Issuing new certificate: "apiserver-proxy-client" I0522 05:44:53.972077 20089 vfs_castore.go:731] Issuing new certificate: "kubelet-api" I0522 05:44:54.096859 20089 vfs_castore.go:731] Issuing new certificate: "kube-controller-manager" I0522 05:44:54.117068 20089 vfs_castore.go:731] Issuing new certificate: "kube-scheduler" I0522 05:44:54.141624 20089 vfs_castore.go:731] Issuing new certificate: "apiserver-aggregator" I0522 05:44:54.165143 20089 vfs_castore.go:731] Issuing new certificate: "kube-proxy" I0522 05:44:54.630570 20089 vfs_castore.go:731] Issuing new certificate: "kops" I0522 05:44:54.741924 20089 executor.go:91] Tasks: 55 done / 79 total; 20 can run I0522 05:44:54.950987 20089 launchconfiguration.go:341] waiting for IAM instance profile "nodes.cluster.k8s.local" to be ready I0522 05:44:54.988838 20089 launchconfiguration.go:341] waiting for IAM instance profile "masters.cluster.k8s.local" to be ready I0522 05:45:05.329692 20089 executor.go:91] Tasks: 75 done / 79 total; 3 can run I0522 05:45:05.923202 20089 vfs_castore.go:731] Issuing new certificate: "master" I0522 05:45:06.306295 20089 executor.go:91] Tasks: 78 done / 79 total; 1 can run W0522 05:45:06.492131 20089 executor.go:118] error running task "LoadBalancerAttachment/api-master-us-west-1a" (9m59s remaining to succeed): error attaching autoscaling group to ELB: ValidationError: Provided Load Balancers may not be valid. Please ensure they exist and try again. status code: 400, request id: 47ee7a1b-5d83-11e8-8909-f9b2d6aabfec I0522 05:45:06.492161 20089 executor.go:133] No progress made, sleeping before retrying 1 failed task(s) I0522 05:45:16.492436 20089 executor.go:91] Tasks: 78 done / 79 total; 1 can run W0522 05:45:16.662368 20089 executor.go:118] error running task "LoadBalancerAttachment/api-master-us-west-1a" (9m49s remaining to succeed): error attaching autoscaling group to ELB: ValidationError: Provided Load Balancers may not be valid. Please ensure they exist and try again. status code: 400, request id: 4e013103-5d83-11e8-bdfc-4f93318e3eef I0522 05:45:16.662398 20089 executor.go:133] No progress made, sleeping before retrying 1 failed task(s) I0522 05:45:26.662694 20089 executor.go:91] Tasks: 78 done / 79 total; 1 can run I0522 05:45:27.058456 20089 executor.go:91] Tasks: 79 done / 79 total; 0 can run I0522 05:45:27.058645 20089 kubectl.go:134] error running kubectl config view --output json I0522 05:45:27.058656 20089 kubectl.go:135] I0522 05:45:27.058662 20089 kubectl.go:136] W0522 05:45:27.058678 20089 update_cluster.go:279] error reading kubecfg: error getting config from kubectl: error running kubectl: exec: "kubectl": executable file not found in $PATH I0522 05:45:27.098105 20089 update_cluster.go:291] Exporting kubecfg for cluster kops has set your kubectl context to cluster.k8s.local Cluster changes have been applied to the cloud. Changes may require instances to restart: kops rolling-update cluster
2.13 查看信息
[root@ip-10-0-103-135 .ssh]# kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-41-218.us-west-1.compute.internal Ready master 15m v1.9.3 ip-10-0-49-17.us-west-1.compute.internal Ready node 10m v1.9.3 ip-10-0-53-149.us-west-1.compute.internal Ready node 9m v1.9.3
2.14 ssh到k8s node
[root@ip-10-0-103-135 ~]# ssh [email protected] Last login: Tue May 22 06:04:58 2018 from ip-10-0-103-135.us-west-1.compute.internal [ec2-user@ip-10-0-53-149 ~]$ sudo su [root@ip-10-0-53-149 ec2-user]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2d34f2985513 gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:535d108a4951f0c9c479949ff96878f353403458ec908266db36a98e0449c8b6 "/sidecar --v=2 --..." 10 minutes ago Up 10 minutes k8s_sidecar_kube-dns-7785f4d7dc-xpx5v_kube-system_c5f02eac-5d83-11e8-8bf9-02ef9dce711c_0 be2d5fbf37d6 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:4278a5d5fae8ad1612402eae4e5aea40aad7af3c63fbfbc1ca23e14c6a8dcd71 "/dnsmasq-nanny -v..." 10 minutes ago Up 10 minutes k8s_dnsmasq_kube-dns-7785f4d7dc-xpx5v_kube-system_c5f02eac-5d83-11e8-8bf9-02ef9dce711c_0 84277b8995ea gcr.io/google_containers/cluster-proportional-autoscaler-amd64@sha256:003f98d9f411ddfa6ff6d539196355e03ddd69fa4ed38c7ffb8fec6f729afe2d "/cluster-proporti..." 10 minutes ago Up 10 minutes k8s_autoscaler_kube-dns-autoscaler-787d59df8f-gtr5n_kube-system_c5f0169d-5d83-11e8-8bf9-02ef9dce711c_0 918de60b761b gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:7d3d06a0c5577f6f546d34d4bbde5b495157ee00b55d83052d68f723421827da "/kube-dns --domai..." 10 minutes ago Up 10 minutes k8s_kubedns_kube-dns-7785f4d7dc-xpx5v_kube-system_c5f02eac-5d83-11e8-8bf9-02ef9dce711c_0 6ee8444f5c88 gcr.io/google_containers/pause-amd64:3.0 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-autoscaler-787d59df8f-gtr5n_kube-system_c5f0169d-5d83-11e8-8bf9-02ef9dce711c_0 358977e7ff85 gcr.io/google_containers/pause-amd64:3.0 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-7785f4d7dc-xpx5v_kube-system_c5f02eac-5d83-11e8-8bf9-02ef9dce711c_0 69ecd8e47993 quay.io/calico/cni@sha256:3a23e093b1e98cf232a226fedff591d33919f5297f016a41d8012efc83b23a84 "/install-cni.sh" 10 minutes ago Up 10 minutes k8s_install-cni_calico-node-5wqch_kube-system_968f8fae-5d84-11e8-90f4-02ef9dce711c_0 8bb93c57e11e quay.io/calico/node@sha256:7758c25549fcfe677699bbcd3c279b3a174e7cbbbf9d16f3d71713d68f695dfb "start_runit" 11 minutes ago Up 11 minutes k8s_calico-node_calico-node-5wqch_kube-system_968f8fae-5d84-11e8-90f4-02ef9dce711c_0 511c7fa8a052 gcr.io/google_containers/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_calico-node-5wqch_kube-system_968f8fae-5d84-11e8-90f4-02ef9dce711c_0 039f90d7ce35 protokube:1.9.0 "/usr/bin/protokub..." 18 minutes ago Up 18 minutes loving_payne b271e18c1c64 gcr.io/google_containers/kube-proxy@sha256:19277373ca983423c3ff82dbb14f079a2f37b84926a4c569375314fa39a4ee96 "/bin/sh -c 'mkfif..." 18 minutes ago Up 18 minutes k8s_kube-proxy_kube-proxy-ip-10-0-53-149.us-west-1.compute.internal_kube-system_634b68e811d24cbe6f22fc91a57ebe53_0 564bae247807 gcr.io/google_containers/pause-amd64:3.0 "/pause" 18 minutes ago Up 18 minutes k8s_POD_kube-proxy-ip-10-0-53-149.us-west-1.compute.internal_kube-system_634b68e811d24cbe6f22fc91a57ebe53_0
2.15 AWS web console的信息
转载自https://blog.csdn.net/cloudvtech
三、部署nginx deployment和external load balancer
3.1 部署nginx deployment
nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
[root@ip-10-0-103-135 ec2-user]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-6c54bd5869-8rnsb 1/1 Running 0 8m nginx-deployment-6c54bd5869-h6d9t 1/1 Running 0 8m nginx-deployment-6c54bd5869-zdz28 1/1 Running 0 8m
3.2 部署基于NLB的k8s负载均衡service
nlb.yml apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" spec: externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
3.3 查看service状态
kubectl describe svc nginx Name: nginx Namespace: default Labels: app=nginx Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"labels":{"app":"nginx"},"nam... service.beta.kubernetes.io/aws-load-balancer-type=nlb Selector: app=nginx Type: LoadBalancer IP: 100.65.228.231 LoadBalancer Ingress: a6000e4955d8811e8a77e02ef9dce711-6e78fd3ea87bc9af.elb.us-west-1.amazonaws.com Port: http 80/TCP TargetPort: 80/TCP NodePort: http 31973/TCP Endpoints: 100.113.75.194:80,100.113.75.195:80,100.99.250.67:80 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 30353 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 4m service-controller Ensuring load balancer Normal EnsuredLoadBalancer 4m service-controller Ensured load balancer
3.4 查看web console
load balancer
target group
3.5 访问NLB