离线安装spinnaker到K8S集群

前言

Spinnaker 是 Netflix 在2015年开源的一款持续交付平台，最初由 Netflix 开发，用于快速、可靠地发布软件变更。Spinnaker 使开发人员可以更轻松地专注于编写代码，而无需担心底层的云基础设施，它可以和 Jenkins 以及其他流行的构建工具无缝集成。但是由于 GFW 的原因导致在国内部署Spinnaker非常困难，当然，你可以使用代理，但这就与本文的初衷不符。经过多次尝试，终于在不使用代理的情况下安装spinnaker，废话不多说，直接进入正题。

安装Halyard

使用docker的方式安装Halyard。

# useradd spinnaker
# cd /home/spinnaker
$ mkdir ~/.hal
$ docker run -d -p 8084:8084 -p 9000:9000 --name halyard --rm  -v ~/.hal:/home/spinnaker/.hal -v ~/.kube:/home/spinnaker/.kube -it gcr.azk8s.cn/spinnaker-marketplace/halyard:1.31.1

启动参数说明：/home/spinnaker/.hal目录为halyard配置文件，/home/spinnaker/.kube为k8s集群认证目录。
官网给出的镜像地址 gcr.io 在国内是不能访问的，需要将镜像地址修改为gcr.azk8s.cn。

Spinnaker安装前准备

spinnaker部署环境一共有三种:

• 在k8s分布式安装(推荐使用)
• 本地单台机器安装
• 本地git安装
  本文使用K8S方式。

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0",GitCommit:"xxx", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-xx.1", GitCommit:"51888f5", GitTreeState:"", BuildDate:"2019-10-16T08:29:13Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl create namespace spinnaker

配置spinnaker配置文件
主要设置docker镜像仓库及存储,以下为配置文件示例。

currentDeployment: default
deploymentConfigurations:
- name: default
  version: local:1.17.6
  providers:
    appengine:
      enabled: false
      accounts: []
    aws:
      enabled: false
      accounts: []
      bakeryDefaults:
        baseImages: []
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: huabei2
      defaults:
        iamRole: BaseIAMRole
    ecs:
      enabled: false
      accounts: []
    azure:
      enabled: false
      accounts: []
      bakeryDefaults:
        templateFile: azure-linux.json
        baseImages: []
    dcos:
      enabled: false
      accounts: []
      clusters: []
    dockerRegistry:
      enabled: true
      accounts:
      # 镜像仓库地址
      - name: dockerhub
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        # 你的镜像仓库地址
        address: xxxx
        # 镜像仓库认证
        username: xxxxx
        password: xxxxx
        email: [email protected]
        sortTagsByDate: true
        repositories:
        # 此处是你的namespace下的项目
        - namespace/appname
      # 镜像仓库地址
      primaryAccount: dockerhub
    google:
      enabled: false
      accounts: []
      bakeryDefaults:
        templateFile: gce.json
        baseImages: []
        zone: us-central1-f
        network: default
        useInternalIp: false
    huaweicloud:
      enabled: false
      accounts: []
      bakeryDefaults:
        baseImages: []
    kubernetes:
      enabled: true
      accounts:
      # 此处为k8s集群名称
      - name: k8s
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces:
        # 此处把你的namespace 加进去
        - yournamespace  
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        # 此处为k8s集群认证文件地址
        kubeconfigFile: /home/spinnaker/.kube/config
        oAuthScopes: []
        onlySpinnakerManaged: false
      # 与k8s集群名称一致
      primaryAccount: k8s
    oracle:
      enabled: false
      accounts: []
      bakeryDefaults:
        templateFile: oci.json
        baseImages: []
    cloudfoundry:
      enabled: false
      accounts: []
  deploymentEnvironment:
    size: SMALL
    type: Distributed
    # k8s集群名称
    accountName: k8s
    imageVariant: SLIM
    updateVersions: false
    consul:
      enabled: false
    vault:
      enabled: false
    customSizing: {}
    sidecars: {}
    initContainers: {}
    hostAliases: {}
    affinity: {}
    tolerations: {}
    nodeSelectors: {}
    gitConfig:
      upstreamUser: spinnaker
    livenessProbeConfig:
      enabled: false
    haServices:
      clouddriver:
        enabled: false
        disableClouddriverRoDeck: false
      echo:
        enabled: false
  persistentStorage:
    # 此处配置存储，示例为s3
    persistentStoreType: s3
    azs: {}
    gcs: {} 
    redis: {}
    # s3存储的认证信息
    s3:
      bucket: xxx
      rootFolder: xxx
      region: xxx
      pathStyleAccess: false
      accessKeyId: xxx
      secretAccessKey:  xxx
    oracle: {}
  features:
    auth: false
    fiat: false
    chaos: false
    entityTags: false
    pipelineTemplates: true
    artifacts: true
  metricStores:
    datadog:
      enabled: false
      tags: []
    prometheus:
      enabled: false
      add_source_metalabels: true
    stackdriver:
      enabled: false
    newrelic:
      enabled: false
      tags: []
    period: 30
    enabled: false
  notifications:
    slack:
      enabled: false
    twilio:
      enabled: false
      baseUrl: https://api.twilio.com/
    github-status:
      enabled: false
  timezone: Asia/Shanghai
  ci:
    jenkins:
      enabled: false
      masters: []
    travis:
      enabled: false
      masters: []
    wercker:
      enabled: false
      masters: []
    concourse:
      enabled: false
      masters: []
    gcb:
      enabled: false
      accounts: []
  repository:
    artifactory:
      enabled: false
      searches: []
  security:
    apiSecurity:
      ssl:
        enabled: false
      # gate地址  
      overrideBaseUrl: https://spin-gate.xxx.com
    uiSecurity:
      ssl:
        enabled: false
      # deck地址  
      overrideBaseUrl: https://spinnaker.xxx.com
    authn:
      oauth2:
        enabled: false
        client: {}
        resource: {}
        userInfoMapping: {}
      saml:
        enabled: false
        userAttributeMapping: {}
      ldap:
        enabled: false
      x509:
        enabled: false
      iap:
        enabled: false
      enabled: false
    authz:
      groupMembership:
        service: FILE
        google:
          roleProviderType: GOOGLE
        github:
          roleProviderType: GITHUB
        file:
          roleProviderType: FILE
        ldap:
          roleProviderType: LDAP
      enabled: true
  artifacts:
    bitbucket:
      enabled: false
      accounts: []
    gcs:
      enabled: false
      accounts: []
    oracle:
      enabled: false
      accounts: []
    github:
      enabled: false
      accounts: []
    gitlab:
      enabled: true
      accounts:
      - name: gitlab
        token: xxx
    gitrepo:
      enabled: false
      accounts: []
    http:
      enabled: false
      accounts: []
    helm:
      enabled: false
      accounts: []
    s3:
      enabled: false
      accounts: []
    maven:
      enabled: false
      accounts: []
    templates: []
  pubsub:
    enabled: false
    google:
      enabled: false
      pubsubType: GOOGLE
      subscriptions: []
      publishers: []
  canary:
    enabled: false
    serviceIntegrations:
    - name: google
      enabled: false
      accounts: []
      gcsEnabled: false
      stackdriverEnabled: false
    - name: prometheus
      enabled: false
      accounts: []
    - name: datadog
      enabled: false
      accounts: []
    - name: signalfx
      enabled: false
      accounts: []
    - name: aws
      enabled: false
      accounts: []
      s3Enabled: false
    - name: newrelic
      enabled: false
      accounts: []
    reduxLoggerEnabled: true
    defaultJudge: NetflixACAJudge-v1.0
    stagesEnabled: true
    templatesEnabled: true
    showAllConfigsEnabled: true
  plugins:
    plugins: []
    enabled: false
    downloadingEnabled: false
  spinnaker:
    extensibility:
      repositories: {}
  webhook:
    trust:
      enabled: false
  telemetry:
    enabled: false
    endpoint: https://stats.spinnaker.io
    instanceId: xxx
    connectionTimeoutMillis: 3000
    readTimeoutMillis: 5000

本地安装spinnaker
接下来的操作尤为重要，我们采用本地安装的方式，需要将bom设置为local，安装版本1.17.6.

$ pwd
/home/spinnaker
$ mkdir ~/.hal/.boms/bom -p
$ cd  ~/.hal/.boms/bom
$ more 1.17.6.yml
version: 1.17.6
timestamp: '2020-01-14 08:44:42'
services:
  echo:
    version: local:2.9.1-20191216151527
    commit: 771a15b2b7bd8d78f77caf3c3ecff950e187c1ae
  clouddriver:
    version: local:6.4.5-20200114034416
    commit: 5f272cd8d3911423dfcf7e9448c31fe4aa045e2e
  deck:
    version: local:2.13.5-20200114034416
    commit: 75cecc4cf1d52ff78fb1fb5e057b516c51be10fb
  fiat:
    version: local:1.8.3-20191202102650
    commit: c62d038c2a9531042ff33c5992384184b1370b27
  front50:
    version: local:0.20.1-20191107034416
    commit: 9415a443b0d6bf800ccca8c2764d303eb4d29366
  gate:
    version: local:1.13.0-20191029172246
    commit: a453541b47c745a283712bb240ab392ad7319e8d
  igor:
    version: local:1.7.0-20191029183208
    commit: 37fe1ed0c463bdaa87996a4d4dd81fee2325ec8e
  kayenta:
    version: local:0.12.0-20191023142816
    commit: 5dcec805b7533d0406f1e657a62122f4278d665d
  orca:
    version: local:2.11.2-20191212093351
    commit: b88f62a1b2b1bdee0f45d7f9491932f9c51371d9
  rosco:
    version: local:0.15.1-20191202163249
    commit: 269dc830cf7ea2ee6c160163e30d6cbd099269c2
  defaultArtifact: {}
  monitoring-third-party:
    version: local:0.16.0-20191007112816
    commit: 59cbbec589f982864cee45d20c99c32d39c75f7f
  monitoring-daemon:
    version: local:0.16.0-20191007112816
    commit: 59cbbec589f982864cee45d20c99c32d39c75f7f
dependencies:
  redis:
    version: 2:2.8.4-2
  consul:
    version: 0.7.5
  vault:
    version: 0.7.0
artifactSources:
  debianRepository: https://dl.bintray.com/spinnaker-releases/debians
  dockerRegistry: gcr.azk8s.cn/spinnaker-marketplace
  googleImageProject: marketplace-spinnaker-release
  gitPrefix: https://github.com/spinnaker

bom配置文件需要注意两点，版本前需要加local，镜像地址更换为国内地址！
配置完成之后，hal在deploy的时候不会去google拉取版本配置，而是从本地读取，以echo为例，本地读取目录为~/.hal/.boms/echo/echo.yml，所以我们需要将相应的yml文件下载到本地相应目录。
下载地址：https://github.com/spinnaker
以echo为例，下载文件为https://github.com/spinnaker/echo/tree/master/halconfig 内的文件，并放在本地目录~/.hal/.boms/echo下。
下载完成后目录结构如下：

├── bom/
│   ├── 1.17.6.yml
├── clouddriver/
│   ├── clouddriver.yml
│   └── ...
├── deck/
│   ├──    settings.js
.......

正式部署spinnaker

完成了这些基本工作，就可以愉快的（无需代理）的部署spinnaker了。

# 设置hal使用本地文件系统的bom斑斑
$ hal config version edit --version local:1.17.6
$hal deploy apply
+ Get current deployment
  Success
+ Prep deployment
  Success
Problems in default.security:
- WARNING Your UI or API domain does not have override base URLs
  set even though your Spinnaker deployment is a Distributed deployment on a
  remote cloud provider. As a result, you will need to open SSH tunnels against
  that deployment to access Spinnaker.
? We recommend that you instead configure an authentication
  mechanism (OAuth2, SAML2, or x509) to make it easier to access Spinnaker
  securely, and then register the intended Domain and IP addresses that your
  publicly facing services will be using.

+ Preparation complete... deploying Spinnaker
+ Get current deployment
  Success
+ Apply deployment
  Success
+ Deploy spin-redis
  Success
+ Deploy spin-clouddriver
  Success
+ Deploy spin-front50
  Success
+ Deploy spin-orca
  Success
+ Deploy spin-deck
  Success
+ Deploy spin-echo
  Success
+ Deploy spin-gate
  Success
+ Deploy spin-rosco
  Success
+ Deploy spin-igor
  Success
+ Run `hal deploy connect` to connect to Spinnaker.

查看spinnaker在k8s中的部署情况：

# kubens spinnaker
Context "xxxx" modified.
Active namespace is "spinnaker".
# kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
spin-clouddriver-7fb74cf5c6-c7g87   1/1     Running   0          10m
spin-deck-76b4df7b48-nnj8x          1/1     Running   0          10m
spin-echo-cccf9f7db-bn6t5           1/1     Running   0          10m
spin-fiat-7dc9547ff5-jqw7l          1/1     Running   0          10m
spin-front50-5744455478-qmg2x       1/1     Running   0          10m
spin-gate-6bbb57846c-tplqg          1/1     Running   0          10m
spin-igor-74bfbc6c57-xtbb8          1/1     Running   0          10m
spin-orca-78dc9844cb-6w4ww          1/1     Running   0          10m
spin-redis-6bf56c789c-lbq6l         1/1     Running   0          10m
spin-rosco-664b65f8d6-6n58m         1/1     Running   0          10m

配置ingress
配置ingress访问spinnaker。

$ more spinnaker-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: spinnaker
  namespace: spinnaker
  annotations:
    kubernetes.io/ingress.class: web
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

spec:
  tls:
    - secretName: ingress-niucache
      hosts:
        - spinnaker.xxx.com
        - spin-gate.xxx.com
  rules:
    - host: spinnaker.xxx.com
      http:
        paths:
        - path: /
          backend:
            serviceName: spin-deck
            servicePort: 9000
    - host: spin-gate.xxx.com
      http:
        paths:
        - path: /
          backend:
            serviceName: spin-gate
            servicePort: 8084
# kubectl get ingress
NAME        HOSTS                                           ADDRESS                         PORTS     AGE
spinnaker   spinnaker.xxx.com,spin-gate.xxx.com   xx  80, 443   10m            

访问spinnaker
将域名做好解析之后就可以访问spinnaker了。

至此，spinnaker安装完成！

后记

为了绕过GW安装spinnaker，真是煞费苦心，为了做个记录，同时分享给有需要的朋友。标题写的离线安装，不是真正的离线，意思是无需代理，但是还需要可以连接互联网下载镜像的，但是如果镜像都在本地的镜像仓库中，那就可以做到真正离线了。本文只是介绍了spinnaker的安装，并没有提及使用，个人觉得spinnaker最难的地方就是安装，使用的话，大家可以参考官方文档，已经很详细了，如果以后有需要，再补充使用的文档吧。