Helm部署Ingress使用DaemonSet+Taint/Tolerations+NodeSelector

  • A+
所属分类:k8s

kubernetes集群中需要在指定的几个节点上只部署Nginx Ingress Controller实例,不会跑其他业务容器。

Helm部署Ingress使用DaemonSet+Taint/Tolerations+NodeSelector

环境说明

教程基于有k8s集群,并安装好helm部署环境。强烈推荐使用helm发布您的代码!

执行helm version出现如下证明环境已经就绪。

  1. [root@oneinstack ~]# helm version
  2. Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
  3. Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

helm下载ingress chart

搜索ingress charts

  1. helm search ingress

下载nginx ingress

  1. helm fetch stable/nginx-ingress

fetch之后得到nginx-ingress-1.4.0.tgz

修改ingress helm chart

解压nginx-ingress-1.4.0.tgz

  1. tar xzf nginx-ingress-1.4.0.tgz

注意: 解压过程出现implausibly old time stamp 1970-01-01 08:00:00可忽略

修改values.yaml,下面是我的修改好的:

  1. ## nginx configuration
  2. ## Ref: https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md
  3. ##
  4. controller:
  5.   name: controller
  6.   image:
  7.     repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller  # 建议将镜像拖到自己私有参考,修改私有仓库地址
  8.     tag: "0.24.1"
  9.     pullPolicy: IfNotPresent
  10.     # www-data -> uid 33
  11.     runAsUser: 33
  12.   config: {}
  13.   # Will add custom header to Nginx https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/customization/custom-headers
  14.   headers: {}
  15.   # Required for use with CNI based kubernetes installations (such as ones set up by kubeadm),
  16.   # since CNI and hostport don't mix yet. Can be deprecated once https://github.com/kubernetes/kubernetes/issues/23920
  17.   # is merged
  18.   hostNetwork: true   # 80 443 暴露到宿主机
  19.   # Optionally change this to ClusterFirstWithHostNet in case you have 'hostNetwork: true'.
  20.   # By defaultwhile using host network, name resolution uses the host's DNS. If you wish nginx-controller
  21.   # to keep resolving names inside the k8s network, use ClusterFirstWithHostNet.
  22.   dnsPolicy: ClusterFirst
  23.   ## Use host ports 80 and 443
  24.   daemonset:
  25.     useHostPort: false
  26.     hostPorts:
  27.       http: 80
  28.       https: 443
  29.       ## healthz endpoint
  30.       stats: 18080
  31.   ## Required only if defaultBackend.enabled = false
  32.   ## Must be <namespace>/<service_name>
  33.   ##
  34.   defaultBackendService: ""
  35.   ## Election ID to use for status update
  36.   ##
  37.   electionID: ingress-controller-leader
  38.   ## Name of the ingress class to route through this controller
  39.   ##
  40.   ingressClass: nginx # 后续ingress.yaml annotations指定kubernetes.io/ingress.class: nginx
  41.   # labels to add to the pod container metadata
  42.   podLabels: {}
  43.   #  key: value
  44.   ## Allows customization of the external service
  45.   ## the ingress will be bound to via DNS
  46.   publishService:
  47.     enabled: false
  48.     ## Allows overriding of the publish service to bind to
  49.     ## Must be <namespace>/<service_name>
  50.     ##
  51.     pathOverride: ""
  52.   ## Limit the scope of the controller
  53.   ##
  54.   scope:
  55.     enabled: false
  56.     namespace: ""   # defaults to .Release.Namespace
  57.   ## Additional command line arguments to pass to nginx-ingress-controller
  58.   ## E.g. to specify the default SSL certificate you can use
  59.   ## extraArgs:
  60.   ##   default-ssl-certificate: "<namespace>/<secret_name>"
  61.   extraArgs: {}
  62.   ## Additional environment variables to set
  63.   extraEnvs: []
  64.   # extraEnvs:
  65.   #   - name: FOO
  66.   #     valueFrom:
  67.   #       secretKeyRef:
  68.   #         key: FOO
  69.   #         name: secret-resource
  70.   ## DaemonSet or Deployment
  71.   ##
  72.   kind: DaemonSet   #DaemonSet模式
  73.   # The update strategy to apply to the Deployment or DaemonSet
  74.   ##
  75.   updateStrategy: {}
  76.   #  rollingUpdate:
  77.   #    maxUnavailable: 1
  78.   #  type: RollingUpdate
  79.   # minReadySeconds to avoid killing pods before we are ready
  80.   ##
  81.   minReadySeconds: 0
  82.   ## Node tolerations for server scheduling to nodes with taints
  83.   ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  84.   ##
  85.   tolerations:    #在节点上打污点,此处是容忍的key value effect
  86.     - key: "nginx-ingress"
  87.       operator: "Equal"
  88.       value: "true"
  89.       effect: "NoSchedule"
  90.   affinity: {}
  91.   ## Node labels for controller pod assignment
  92.   ## Ref: https://kubernetes.io/docs/user-guide/node-selection/
  93.   ##
  94.   nodeSelector:
  95.     nginx-ingress: "true"    #使用节点标签选择器,访问在所有节点运行ingress
  96.   ## Liveness and readiness probe values
  97.   ## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
  98.   ##
  99.   livenessProbe:
  100.     failureThreshold: 3
  101.     initialDelaySeconds: 10
  102.     periodSeconds: 10
  103.     successThreshold: 1
  104.     timeoutSeconds: 1
  105.     port: 10254
  106.   readinessProbe:
  107.     failureThreshold: 3
  108.     initialDelaySeconds: 10
  109.     periodSeconds: 10
  110.     successThreshold: 1
  111.     timeoutSeconds: 1
  112.     port: 10254
  113.   ## Annotations to be added to controller pods
  114.   ##
  115.   podAnnotations:    # 支持prometheus抓取数据
  116.     prometheus.io/scrape: "true"
  117.     prometheus.io/port: "10254"
  118.   replicaCount: 1
  119.   minAvailable: 1
  120.   resources: {}
  121.   #  limits:
  122.   #    cpu: 100m
  123.   #    memory: 64Mi
  124.   #  requests:
  125.   #    cpu: 100m
  126.   #    memory: 64Mi
  127.   autoscaling:
  128.     enabled: false
  129.     minReplicas: 1
  130.     maxReplicas: 11
  131.     targetCPUUtilizationPercentage: 50
  132.     targetMemoryUtilizationPercentage: 50
  133.   ## Override NGINX template
  134.   customTemplate:
  135.     configMapName: ""
  136.     configMapKey: ""
  137.   service:
  138.     annotations: {}
  139.     labels: {}
  140.     clusterIP: ""
  141.     ## List of IP addresses at which the controller services are available
  142.     ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
  143.     ##
  144.     externalIPs: []
  145.     loadBalancerIP: ""
  146.     loadBalancerSourceRanges: []
  147.     enableHttp: true
  148.     enableHttps: true
  149.     ## Set external traffic policy to: "Local" to preserve source IP on
  150.     ## providers supporting it
  151.     ## Ref: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer
  152.     externalTrafficPolicy: ""
  153.     healthCheckNodePort: 0
  154.     targetPorts:
  155.       http: http
  156.       https: https
  157.     type: ClusterIP
  158.     # type: NodePort
  159.     # nodePorts:
  160.     #   http: 32080
  161.     #   https: 32443
  162.     nodePorts:
  163.       http: ""
  164.       https: ""
  165.   extraContainers: []
  166.   ## Additional containers to be added to the controller pod.
  167.   ## See https://github.com/lemonldap-ng-controller/lemonldap-ng-controller as example.
  168.   #  - name: my-sidecar
  169.   #    image: nginx:latest
  170.   #  - name: lemonldap-ng-controller
  171.   #    image: lemonldapng/lemonldap-ng-controller:0.2.0
  172.   #    args:
  173.   #      - /lemonldap-ng-controller
  174.   #      - --alsologtostderr
  175.   #      - --configmap=$(POD_NAMESPACE)/lemonldap-ng-configuration
  176.   #    env:
  177.   #      - name: POD_NAME
  178.   #        valueFrom:
  179.   #          fieldRef:
  180.   #            fieldPath: metadata.name
  181.   #      - name: POD_NAMESPACE
  182.   #        valueFrom:
  183.   #          fieldRef:
  184.   #            fieldPath: metadata.namespace
  185.   #    volumeMounts:
  186.   #    - name: copy-portal-skins
  187.   #      mountPath: /srv/var/lib/lemonldap-ng/portal/skins
  188.   extraVolumeMounts: []
  189.   ## Additional volumeMounts to the controller main container.
  190.   #  - name: copy-portal-skins
  191.   #   mountPath: /var/lib/lemonldap-ng/portal/skins
  192.   extraVolumes: []
  193.   ## Additional volumes to the controller pod.
  194.   #  - name: copy-portal-skins
  195.   #    emptyDir: {}
  196.   extraInitContainers: []
  197.   ## Containers, which are run before the app containers are started.
  198.   # - name: init-myservice
  199.   #   image: busybox
  200.   #   command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  201.   stats:
  202.     enabled: true
  203.     service:
  204.       annotations: {}
  205.       clusterIP: ""
  206.       ## List of IP addresses at which the stats service is available
  207.       ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
  208.       ##
  209.       externalIPs: []
  210.       loadBalancerIP: ""
  211.       loadBalancerSourceRanges: []
  212.       servicePort: 18080
  213.       type: ClusterIP
  214.   ## If controller.stats.enabled = true and controller.metrics.enabled = true, Prometheus metrics will be exported
  215.   ##
  216.   metrics:
  217.     enabled: true
  218.     service:
  219.       annotations:
  220.         prometheus.io/scrape: "true"
  221.         prometheus.io/port: "10254"
  222.       clusterIP: ""
  223.       ## List of IP addresses at which the stats-exporter service is available
  224.       ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
  225.       ##
  226.       externalIPs: []
  227.       loadBalancerIP: ""
  228.       loadBalancerSourceRanges: []
  229.       servicePort: 9913
  230.       type: ClusterIP
  231.     serviceMonitor:
  232.       enabled: false
  233.       additionalLabels: {}
  234.       namespace: ""
  235.   lifecycle: {}
  236.   priorityClassName: ""
  237. ## Rollback limit
  238. ##
  239. revisionHistoryLimit: 10
  240. ## Default 404 backend
  241. ##
  242. defaultBackend:
  243.   ## If false, controller.defaultBackendService must be provided
  244.   ##
  245.   enabled: true
  246.   name: default-backend
  247.   image:
  248.     repository: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend
  249.     tag: "1.4"
  250.     pullPolicy: IfNotPresent
  251.   extraArgs: {}
  252.   port: 8080
  253.   ## Node tolerations for server scheduling to nodes with taints
  254.   ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  255.   ##
  256.   tolerations: []
  257.   #  - key: "key"
  258.   #    operator: "Equal|Exists"
  259.   #    value: "value"
  260.   #    effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
  261.   affinity: {}
  262.   # labels to add to the pod container metadata
  263.   podLabels: {}
  264.   #  key: value
  265.   ## Node labels for default backend pod assignment
  266.   ## Ref: https://kubernetes.io/docs/user-guide/node-selection/
  267.   ##
  268.   nodeSelector: {}
  269.   ## Annotations to be added to default backend pods
  270.   ##
  271.   podAnnotations: {}
  272.   replicaCount: 1
  273.   minAvailable: 1
  274.   resources: {}
  275.   # limits:
  276.   #   cpu: 10m
  277.   #   memory: 20Mi
  278.   # requests:
  279.   #   cpu: 10m
  280.   #   memory: 20Mi
  281.   service:
  282.     annotations: {}
  283.     clusterIP: ""
  284.     ## List of IP addresses at which the default backend service is available
  285.     ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
  286.     ##
  287.     externalIPs: []
  288.     loadBalancerIP: ""
  289.     loadBalancerSourceRanges: []
  290.     servicePort: 80
  291.     type: ClusterIP
  292.   priorityClassName: ""
  293. ## Enable RBAC as per https://github.com/kubernetes/ingress/tree/master/examples/rbac/nginx and https://github.com/kubernetes/ingress/issues/266
  294. rbac:
  295.   create: true
  296. # If true, create & use Pod Security Policy resources
  297. # https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  298. podSecurityPolicy:
  299.   enabled: false
  300. serviceAccount:
  301.   create: true
  302.   name:
  303. ## Optional array of imagePullSecrets containing private registry credentials
  304. ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  305. imagePullSecrets: []
  306. # - name: secretName
  307. # TCP service key:value pairs
  308. # Ref: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/tcp
  309. ##
  310. tcp: {}
  311. #  8080"default/example-tcp-svc:9000"
  312. # UDP service key:value pairs
  313. # Ref: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/udp
  314. ##
  315. udp: {}
  316. #  53"kube-system/kube-dns:53"

NODE打标签,设置污点

打标签

  1. kubectl label nodes 10.0.10.7 nginx-ingress=true
  2. kubectl label nodes 10.0.10.8 nginx-ingress=true
  3. kubectl label nodes 10.0.10.9 nginx-ingress=true

设置污点

  1. kubectl taint nodes 10.0.10.7 nginx-ingress=true:NoSchedule
  2. kubectl taint nodes 10.0.10.8 nginx-ingress=true:NoSchedule
  3. kubectl taint nodes 10.0.10.9 nginx-ingress=true:NoSchedule

安装nginx-ingress

  1. helm upgrade nginx-ingress ./nginx-ingress --install --namespace nginx-ingress --dry-run  # 测试运行
  1. helm upgrade nginx-ingress ./nginx-ingress --install --namespace nginx-ingress

注意:

  1. 需要先在NODE节点打污点、标签
  2. helm名字和命名空间请使用nginx-ingress, 和直接用yaml文件(ingress-nginx)有区别。否则DaemonSet、pod名字比较奇怪

Sun Apr 14 15:29:38 CST 2019

avatar

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: