何宇泽

k8s部署treafik2.x

发表于 更新于 分类于
本文字数： 9.7k 阅读时长 ≈ 9 分钟

1. k8s部署treafik2.x

https://blog.51cto.com/u_15329153/3371086?xiangguantuijian

1.1. 创建 CRD 资源

在Traefik v2.0版本后，开始使用 CRD（Custom Resource Definition）来完成路由配置等，所以需要提前创建CRD资源。

cat traefik-crd.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
---
## IngressRouteTCP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
---
## Middleware
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
---
## TraefikService
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
---
## TraefikTLSStore
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
---
## IngressRouteUDP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp

创建traefik crd 资源

1
kubectl apply -f traefik-crd.yaml

1.2. 创建 RBAC 权限

Kubernetes 在 1.6 版本中引入了基于角色的访问控制（RBAC）策略，方便对 Kubernetes 资源和 API 进行细粒度控制。Traefik 需要一定的权限，所以，这里提前创建好Traefik ServiceAccount并分配一定的权限。

cat traefik-rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# cat traefik-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system

创建 Traefik RBAC 资源

1
kubectl apply -f traefik-rbac.yaml

1.3. 创建 Traefik 配置文件

由于 Traefik 配置很多，通过 CLI 定义不是很方便，一般时候都会通过配置文件配置 Traefik 参数，然后存入ConfigMap，将其挂入 Traefik 中。

下面配置中可以通过配置kubernetesCRD与kubernetesIngress和kubernetesGateway三项参数，让 Traefik 支持 CRD、Ingress与kubernetesGateway三种路由配置方式。

cat traefik-config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# cat traefik-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
  namespace: kube-system
data:
  traefik.yaml: |-
    serversTransport:
      insecureSkipVerify: true  ## Traefik 忽略验证代理服务的 TLS 证书
    api:
      insecure: true            ## 允许 HTTP 方式访问 API
      dashboard: true           ## 启用 Dashboard
      debug: true               ## 启用 Debug 调试模式
    metrics:
      prometheus: metrics       ## 配置 Prometheus 监控指标数据，并使用默认配置
    entryPoints:
      web:
        address: ":80"          ## 配置 80 端口，并设置入口名称为 web
      websecure:
        address: ":443"         ## 配置 443 端口，并设置入口名称为 websecure
      traefik:
        address: ":8090"        ## 配置 8090 端口，并设置入口名称为 dashboard
      metrics:
        address: ":8082"        ## 配置 8082 端口，作为metrics收集入口
      tcpep:
        address: ":8000"        ## 配置 8000 端口，作为tcp入口
      udpep:
        address: ":9000/udp"    ## 配置 9000 端口，作为udp入口
    providers:
      kubernetescrd:            ## 启用 Kubernetes CRD 方式来配置路由规则
        ingressclass: traefik-v2.3
      kubernetesingress:        ## 启动 Kubernetes Ingress 方式来配置路由规则
        ingressclass: traefik-v2.3
    log:
      filePath: "/etc/traefik/logs/traefik.log"              ## 设置调试日志文件存储路径，如果为空则输出到控制台
      level: error              ## 设置调试日志级别
      format: json                ## 设置调试日志格式
    accessLog:
      filePath: "/etc/traefik/logs/access.log"              ## 设置访问日志文件存储路径，如果为空则输出到控制台
      format: json                ## 设置访问调试日志格式
      bufferingSize: 0          ## 设置访问日志缓存行数
      filters:
        #statusCodes: ["200"]   ## 设置只保留指定状态码范围内的访问日志
        retryAttempts: true     ## 设置代理访问重试失败时，保留访问日志
        minDuration: 20         ## 设置保留请求时间超过指定持续时间的访问日志
      fields:                   ## 设置访问日志中的字段是否保留（keep 保留、drop 不保留）
        defaultMode: keep       ## 设置默认保留访问日志字段
        names:                  ## 针对访问日志特别字段特别配置保留模式
          ClientUsername: drop  
        headers:                ## 设置 Header 中字段是否保留
          defaultMode: keep     ## 设置默认保留 Header 中字段
          names:                ## 针对 Header 中特别字段特别配置保留模式
            User-Agent: redact
            Authorization: drop
            Content-Type: keep

创建Traefik configmap 资源

1
kubectl apply -f traefik-config.yaml

1.4. 设置节点 Label 标签

1
2
3
4
5
6
7
8
[root@k8s-master traefik]# kubectl label nodes k8s-master IngressProxy=traefik2.3
node/k8s-master labeled

[root@k8s-master traefik]# kubectl label nodes k8s-node1 IngressProxy=traefik2.3
node/k8s-node1 labeled

[root@k8s-master traefik]# kubectl label nodes k8s-node2 IngressProxy=traefik2.3
node/k8s-node2 labeled

验证节点标签是否成功

1
2
3
4
5
[root@k8s-master traefik]# kubectl get node --show-labels
NAME         STATUS   ROLES    AGE   VERSION   LABELS
k8s-master   Ready    master   46h   v1.18.0   IngressProxy=traefik2.3,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-node1    Ready    <none>   46h   v1.18.0   IngressProxy=traefik2.3,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux
k8s-node2    Ready    <none>   46h   v1.18.0   IngressProxy=traefik2.3,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node2,kubernetes.io/os=linux

节点删除Label标签

1
2
3
# kubectl label nodes k8s-master IngressProxy-
# kubectl label nodes k8s-node1 IngressProxy-
# kubectl label nodes k8s-node2 IngressProxy-

1.5. 创建 Traefik

下面将用DaemonSet方式部署 Traefik，便于在多服务器间扩展，用 hostport 方式绑定服务器 80、443 端口，方便流量通过物理机进入 Kubernetes 内部。

cat traefik-deploy.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
# cat traefik-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-v2
  namespace: kube-system
  labels:
    app: traefik-v2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: traefik-v2
  template:
    metadata:
      labels:
        app: traefik-v2
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 1
      containers:
        - name: traefik-v2
          image: traefik:v2.3
          args:
            - --configfile=/config/traefik.yaml
          ports:
            - name: web
              containerPort: 80
              hostPort: 80           #hostPort方式，将端口暴露到集群节点
            - name: websecure
              containerPort: 443
              hostPort: 443          #hostPort方式，将端口暴露到集群节点
            - name: admin
              containerPort: 8090
            - name: tcpep
              containerPort: 8000
            - name: udpep
              containerPort: 9000
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 300m
              memory: 1024Mi
          securityContext:
            capabilities:              ## 只开放网络权限    
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          volumeMounts:
          - mountPath: "/config"
            name: "config"
          - mountPath: /etc/traefik/logs
            name: logdir
          - mountPath: /etc/localtime
            name: timezone
            readOnly: true
      volumes:
        - name: config
          configMap:
            name: traefik-config 
        - name: logdir
          hostPath:
            path: /data/traefik/logs
            type: "DirectoryOrCreate"
        - name: timezone
          hostPath:
            path: /etc/localtime
            type: File
      tolerations:            
        - operator: "Exists"        ## 设置容忍所有污点，防止节点被设置污点
      hostNetwork: true             ## 开启host网络，提高网络入口的网络性能
      nodeSelector:                 ## 设置node筛选器，在特定label的节点上启动
        IngressProxy: "traefik2.3"
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-v2
  namespace: kube-system
spec:
  type: LoadBalancer
  selector:
    app: traefik-v2
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 443
      name: websecure
      targetPort: 443
    - protocol: TCP
      port: 8090
      name: admin
      targetPort: 8090
    - protocol: TCP
      port: 8000
      name: tcpep
      targetPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: traefikudp-v2
  namespace: kube-system
spec:
  type: LoadBalancer
  selector:
    app: traefik-v2
  ports:
    - protocol: UDP
      port: 9000
      name: udpep
      targetPort: 9000

创建 Traefik

1
kubectl apply -f traefik-deploy.yaml

使用Deployment类型部署，以便于在多服务器间扩展，使用 hostport 方式占用服务器 80、443 端口，方便流量进入。

1
2
3
4
5
# 部署 Traefik
# kubectl apply -f traefik-deploy.yaml 
deployment.apps/traefik-v2 created
service/traefik-v2 created
service/traefikudp-v2 created

到此 Traefik v2.3 应用已经部署完成。
这时候就可以通过节点http://IP:8090,可以看到dashboard相关信息

traefik

1.6. 路由配置

1.6.1. 配置 HTTP 路由规则 （Traefik Dashboard 为例）

Traefik 应用已经部署完成，但是想让外部访问 Kubernetes 内部服务，还需要配置路由规则，这里开启了 Traefik Dashboard 配置，所以首先配置 Traefik Dashboard 看板的路由规则，使外部能够访问 Traefik Dashboard。

创建 Traefik Dashboard 路由规则文件 traefik-dashboard-route.yaml

因为静态配置文件指定了ingressclass，所以这里的annotations 要指定，否则访问会404

traefik

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# cat traefik-dashboard-route.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik-v2.3     
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`traefik.heyonggs.com`) 
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

部署Traefik Dashboard 路由规则对象

1
2
# kubectl apply -f traefik-dashboard-route.yaml
ingressroute.traefik.containo.us/traefik-dashboard created

客户端通过域名访问服务，必须要进行 DNS 解析，可以通过 DNS 服务器进行域名解析，也可以修改 hosts 文件将 Traefik 指定节点的 IP 和自定义 host 绑定

1
2
# cat hosts
192.168.101.202 traefik.heyonggs.com

打开任意浏览器输入地址： http://traefik.heyonggs.com进行访问，打开 Traefik Dashboard.

此处没有配置验证登录，如果想配置验证登录，使用middleware即可。

traefik

-------------本文结束感谢您的阅读-------------