Helm部署Ingress使用DaemonSet+Taint/Tolerations+NodeSelector

    kubernetes集群中需要在指定的几个节点上只部署Nginx Ingress Controller实例,不会跑其他业务容器。

    Helm部署Ingress使用DaemonSet+Taint/Tolerations+NodeSelector

    环境说明

    教程基于有k8s集群,并安装好helm部署环境。强烈推荐使用helm发布您的代码!

    执行helm version出现如下证明环境已经就绪。

    1. [root@oneinstack ~]# helm version
    2. Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
    3. Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

    helm下载ingress chart

    搜索ingress charts

    1. helm search ingress

    下载nginx ingress

    1. helm fetch stable/nginx-ingress

    fetch之后得到nginx-ingress-1.4.0.tgz

    修改ingress helm chart

    解压nginx-ingress-1.4.0.tgz

    1. tar xzf nginx-ingress-1.4.0.tgz

    注意: 解压过程出现implausibly old time stamp 1970-01-01 08:00:00可忽略

    修改values.yaml,下面是我的修改好的:

    1. ## nginx configuration
    2. ## Ref: https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md
    3. ##
    4. controller:
    5.   name: controller
    6.   image:
    7.     repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller  # 建议将镜像拖到自己私有参考,修改私有仓库地址
    8.     tag: "0.24.1"
    9.     pullPolicy: IfNotPresent
    10.     # www-data -> uid 33
    11.     runAsUser: 33
    12.   config: {}
    13.   # Will add custom header to Nginx https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/customization/custom-headers
    14.   headers: {}
    15.   # Required for use with CNI based kubernetes installations (such as ones set up by kubeadm),
    16.   # since CNI and hostport don't mix yet. Can be deprecated once https://github.com/kubernetes/kubernetes/issues/23920
    17.   # is merged
    18.   hostNetwork: true   # 80 443 暴露到宿主机
    19.   # Optionally change this to ClusterFirstWithHostNet in case you have 'hostNetwork: true'.
    20.   # By defaultwhile using host network, name resolution uses the host's DNS. If you wish nginx-controller
    21.   # to keep resolving names inside the k8s network, use ClusterFirstWithHostNet.
    22.   dnsPolicy: ClusterFirst
    23.   ## Use host ports 80 and 443
    24.   daemonset:
    25.     useHostPort: false
    26.     hostPorts:
    27.       http: 80
    28.       https: 443
    29.       ## healthz endpoint
    30.       stats: 18080
    31.   ## Required only if defaultBackend.enabled = false
    32.   ## Must be <namespace>/<service_name>
    33.   ##
    34.   defaultBackendService: ""
    35.   ## Election ID to use for status update
    36.   ##
    37.   electionID: ingress-controller-leader
    38.   ## Name of the ingress class to route through this controller
    39.   ##
    40.   ingressClass: nginx # 后续ingress.yaml annotations指定kubernetes.io/ingress.class: nginx
    41.   # labels to add to the pod container metadata
    42.   podLabels: {}
    43.   #  key: value
    44.   ## Allows customization of the external service
    45.   ## the ingress will be bound to via DNS
    46.   publishService:
    47.     enabled: false
    48.     ## Allows overriding of the publish service to bind to
    49.     ## Must be <namespace>/<service_name>
    50.     ##
    51.     pathOverride: ""
    52.   ## Limit the scope of the controller
    53.   ##
    54.   scope:
    55.     enabled: false
    56.     namespace: ""   # defaults to .Release.Namespace
    57.   ## Additional command line arguments to pass to nginx-ingress-controller
    58.   ## E.g. to specify the default SSL certificate you can use
    59.   ## extraArgs:
    60.   ##   default-ssl-certificate: "<namespace>/<secret_name>"
    61.   extraArgs: {}
    62.   ## Additional environment variables to set
    63.   extraEnvs: []
    64.   # extraEnvs:
    65.   #   - name: FOO
    66.   #     valueFrom:
    67.   #       secretKeyRef:
    68.   #         key: FOO
    69.   #         name: secret-resource
    70.   ## DaemonSet or Deployment
    71.   ##
    72.   kind: DaemonSet   #DaemonSet模式
    73.   # The update strategy to apply to the Deployment or DaemonSet
    74.   ##
    75.   updateStrategy: {}
    76.   #  rollingUpdate:
    77.   #    maxUnavailable: 1
    78.   #  type: RollingUpdate
    79.   # minReadySeconds to avoid killing pods before we are ready
    80.   ##
    81.   minReadySeconds: 0
    82.   ## Node tolerations for server scheduling to nodes with taints
    83.   ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
    84.   ##
    85.   tolerations:    #在节点上打污点,此处是容忍的key value effect
    86.     - key: "nginx-ingress"
    87.       operator: "Equal"
    88.       value: "true"
    89.       effect: "NoSchedule"
    90.   affinity: {}
    91.   ## Node labels for controller pod assignment
    92.   ## Ref: https://kubernetes.io/docs/user-guide/node-selection/
    93.   ##
    94.   nodeSelector:
    95.     nginx-ingress: "true"    #使用节点标签选择器,访问在所有节点运行ingress
    96.   ## Liveness and readiness probe values
    97.   ## Ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
    98.   ##
    99.   livenessProbe:
    100.     failureThreshold: 3
    101.     initialDelaySeconds: 10
    102.     periodSeconds: 10
    103.     successThreshold: 1
    104.     timeoutSeconds: 1
    105.     port: 10254
    106.   readinessProbe:
    107.     failureThreshold: 3
    108.     initialDelaySeconds: 10
    109.     periodSeconds: 10
    110.     successThreshold: 1
    111.     timeoutSeconds: 1
    112.     port: 10254
    113.   ## Annotations to be added to controller pods
    114.   ##
    115.   podAnnotations:    # 支持prometheus抓取数据
    116.     prometheus.io/scrape: "true"
    117.     prometheus.io/port: "10254"
    118.   replicaCount: 1
    119.   minAvailable: 1
    120.   resources: {}
    121.   #  limits:
    122.   #    cpu: 100m
    123.   #    memory: 64Mi
    124.   #  requests:
    125.   #    cpu: 100m
    126.   #    memory: 64Mi
    127.   autoscaling:
    128.     enabled: false
    129.     minReplicas: 1
    130.     maxReplicas: 11
    131.     targetCPUUtilizationPercentage: 50
    132.     targetMemoryUtilizationPercentage: 50
    133.   ## Override NGINX template
    134.   customTemplate:
    135.     configMapName: ""
    136.     configMapKey: ""
    137.   service:
    138.     annotations: {}
    139.     labels: {}
    140.     clusterIP: ""
    141.     ## List of IP addresses at which the controller services are available
    142.     ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
    143.     ##
    144.     externalIPs: []
    145.     loadBalancerIP: ""
    146.     loadBalancerSourceRanges: []
    147.     enableHttp: true
    148.     enableHttps: true
    149.     ## Set external traffic policy to: "Local" to preserve source IP on
    150.     ## providers supporting it
    151.     ## Ref: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer
    152.     externalTrafficPolicy: ""
    153.     healthCheckNodePort: 0
    154.     targetPorts:
    155.       http: http
    156.       https: https
    157.     type: ClusterIP
    158.     # type: NodePort
    159.     # nodePorts:
    160.     #   http: 32080
    161.     #   https: 32443
    162.     nodePorts:
    163.       http: ""
    164.       https: ""
    165.   extraContainers: []
    166.   ## Additional containers to be added to the controller pod.
    167.   ## See https://github.com/lemonldap-ng-controller/lemonldap-ng-controller as example.
    168.   #  - name: my-sidecar
    169.   #    image: nginx:latest
    170.   #  - name: lemonldap-ng-controller
    171.   #    image: lemonldapng/lemonldap-ng-controller:0.2.0
    172.   #    args:
    173.   #      - /lemonldap-ng-controller
    174.   #      - --alsologtostderr
    175.   #      - --configmap=$(POD_NAMESPACE)/lemonldap-ng-configuration
    176.   #    env:
    177.   #      - name: POD_NAME
    178.   #        valueFrom:
    179.   #          fieldRef:
    180.   #            fieldPath: metadata.name
    181.   #      - name: POD_NAMESPACE
    182.   #        valueFrom:
    183.   #          fieldRef:
    184.   #            fieldPath: metadata.namespace
    185.   #    volumeMounts:
    186.   #    - name: copy-portal-skins
    187.   #      mountPath: /srv/var/lib/lemonldap-ng/portal/skins
    188.   extraVolumeMounts: []
    189.   ## Additional volumeMounts to the controller main container.
    190.   #  - name: copy-portal-skins
    191.   #   mountPath: /var/lib/lemonldap-ng/portal/skins
    192.   extraVolumes: []
    193.   ## Additional volumes to the controller pod.
    194.   #  - name: copy-portal-skins
    195.   #    emptyDir: {}
    196.   extraInitContainers: []
    197.   ## Containers, which are run before the app containers are started.
    198.   # - name: init-myservice
    199.   #   image: busybox
    200.   #   command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
    201.   stats:
    202.     enabled: true
    203.     service:
    204.       annotations: {}
    205.       clusterIP: ""
    206.       ## List of IP addresses at which the stats service is available
    207.       ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
    208.       ##
    209.       externalIPs: []
    210.       loadBalancerIP: ""
    211.       loadBalancerSourceRanges: []
    212.       servicePort: 18080
    213.       type: ClusterIP
    214.   ## If controller.stats.enabled = true and controller.metrics.enabled = true, Prometheus metrics will be exported
    215.   ##
    216.   metrics:
    217.     enabled: true
    218.     service:
    219.       annotations:
    220.         prometheus.io/scrape: "true"
    221.         prometheus.io/port: "10254"
    222.       clusterIP: ""
    223.       ## List of IP addresses at which the stats-exporter service is available
    224.       ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
    225.       ##
    226.       externalIPs: []
    227.       loadBalancerIP: ""
    228.       loadBalancerSourceRanges: []
    229.       servicePort: 9913
    230.       type: ClusterIP
    231.     serviceMonitor:
    232.       enabled: false
    233.       additionalLabels: {}
    234.       namespace: ""
    235.   lifecycle: {}
    236.   priorityClassName: ""
    237. ## Rollback limit
    238. ##
    239. revisionHistoryLimit: 10
    240. ## Default 404 backend
    241. ##
    242. defaultBackend:
    243.   ## If false, controller.defaultBackendService must be provided
    244.   ##
    245.   enabled: true
    246.   name: default-backend
    247.   image:
    248.     repository: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend
    249.     tag: "1.4"
    250.     pullPolicy: IfNotPresent
    251.   extraArgs: {}
    252.   port: 8080
    253.   ## Node tolerations for server scheduling to nodes with taints
    254.   ## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
    255.   ##
    256.   tolerations: []
    257.   #  - key: "key"
    258.   #    operator: "Equal|Exists"
    259.   #    value: "value"
    260.   #    effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
    261.   affinity: {}
    262.   # labels to add to the pod container metadata
    263.   podLabels: {}
    264.   #  key: value
    265.   ## Node labels for default backend pod assignment
    266.   ## Ref: https://kubernetes.io/docs/user-guide/node-selection/
    267.   ##
    268.   nodeSelector: {}
    269.   ## Annotations to be added to default backend pods
    270.   ##
    271.   podAnnotations: {}
    272.   replicaCount: 1
    273.   minAvailable: 1
    274.   resources: {}
    275.   # limits:
    276.   #   cpu: 10m
    277.   #   memory: 20Mi
    278.   # requests:
    279.   #   cpu: 10m
    280.   #   memory: 20Mi
    281.   service:
    282.     annotations: {}
    283.     clusterIP: ""
    284.     ## List of IP addresses at which the default backend service is available
    285.     ## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
    286.     ##
    287.     externalIPs: []
    288.     loadBalancerIP: ""
    289.     loadBalancerSourceRanges: []
    290.     servicePort: 80
    291.     type: ClusterIP
    292.   priorityClassName: ""
    293. ## Enable RBAC as per https://github.com/kubernetes/ingress/tree/master/examples/rbac/nginx and https://github.com/kubernetes/ingress/issues/266
    294. rbac:
    295.   create: true
    296. # If true, create & use Pod Security Policy resources
    297. # https://kubernetes.io/docs/concepts/policy/pod-security-policy/
    298. podSecurityPolicy:
    299.   enabled: false
    300. serviceAccount:
    301.   create: true
    302.   name:
    303. ## Optional array of imagePullSecrets containing private registry credentials
    304. ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    305. imagePullSecrets: []
    306. # - name: secretName
    307. # TCP service key:value pairs
    308. # Ref: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/tcp
    309. ##
    310. tcp: {}
    311. #  8080"default/example-tcp-svc:9000"
    312. # UDP service key:value pairs
    313. # Ref: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/udp
    314. ##
    315. udp: {}
    316. #  53"kube-system/kube-dns:53"

    NODE打标签,设置污点

    打标签

    1. kubectl label nodes 10.0.10.7 nginx-ingress=true
    2. kubectl label nodes 10.0.10.8 nginx-ingress=true
    3. kubectl label nodes 10.0.10.9 nginx-ingress=true

    设置污点

    1. kubectl taint nodes 10.0.10.7 nginx-ingress=true:NoSchedule
    2. kubectl taint nodes 10.0.10.8 nginx-ingress=true:NoSchedule
    3. kubectl taint nodes 10.0.10.9 nginx-ingress=true:NoSchedule

    安装nginx-ingress

    1. helm upgrade nginx-ingress ./nginx-ingress --install --namespace nginx-ingress --dry-run  # 测试运行
    1. helm upgrade nginx-ingress ./nginx-ingress --install --namespace nginx-ingress

    注意:

    1. 需要先在NODE节点打污点、标签
    2. helm名字和命名空间请使用nginx-ingress, 和直接用yaml文件(ingress-nginx)有区别。否则DaemonSet、pod名字比较奇怪

    Sun Apr 14 15:29:38 CST 2019

    • 本文由 发表于 2019-04-14
    • 转载请务必保留本文链接:https://linuxeye.com/476.html
    Filebeat收集K8S日志 k8s

    Filebeat收集K8S日志

    Kubernetes 中比较流行的日志收集解决方案是 Elasticsearch、Logstash和 Kibana(ELK)技术栈,今天来推荐EFK,即Logstash换成filebeat。 切换到E...
    k8s基于etcd的CoreDNS动态域名解析 k8s

    k8s基于etcd的CoreDNS动态域名解析

    本文基于已经搭建好的k8s集群,集群搭建参考:《Kubernetes集群搭建》 下载coredns yaml部署脚本 注意:如果已经部署coredns可忽略 wget https://github.c...
    Kubernetes集群搭建 k8s

    Kubernetes集群搭建

    环境说明 操作系统:CentOS7.4 64bit 软件版本:kubernetes-v1.9.9、etcd-v3.3.8、flannel-v0.10.0 下载地址: https://dl.k8s.io...
    匿名

    发表评论

    匿名网友