^-^
【go云原生】k8s日志采集与监控发现
最近更新：2025-01-08   |   字数总计：4.1k   |   阅读估时：20分钟   |   阅读量：次
  1. k8s日志采集与监控发现
    1. 日志管理
      1. 写文件与写日志中心
      2. filebeat从文件中采集日志
      3. filebeat从文件中采集日志(守护进程\sidecar模式)
    2. 监控
      1. prometheus业务数据采集
      2. prometheus单节点部署
      3. prometheus节点发现
      4. prometheus指标计算

k8s日志采集与监控发现

日志管理

写文件与写日志中心

  • 场景，已经有一个可向腾讯云日志服务输出日志的go应用程序，现在需要将该程序部署到k8s上，该程序将直接写到cls中。
  • 准备私人注册中心的密钥
    1
    2
    3
    kubectl create secret docker-registry dockerregistry --docker-username lin --docker-password 111111 --docker-server 192.168.239.142:5000

    kubectl get secrets dockerregistry -o yaml # 复制这一段
  • Deployment.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    # 定义应用程序命名空间
    apiVersion: v1
    kind: Namespace # 命名空间用于隔离应用程序，默认是default
    metadata:
        name: management 
    --- # 粘贴密钥到这里
    apiVersion: v1
    data:
        .dockerconfigjson: qoienopgqeurgniuegrqwe
    kind: Secret
    metadata:
        name: dockerregistry
        namespace: management # 为了隔离性，这里也要写命名空间
    type: kubernetes.io/dockerconfigjson
    --- # 定义应用程序configmap
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: management-config
        namespace: management
    data:
        config.yaml: | # 将原先程序中使用的config.yaml的内容作为configMap的数据，注意这里是定义一种资源，实际的数据还是需要从存储中映射过来的
        cls:
            topicId: 55d523ba-1795-4457-053-c88100a2076
            endpoint: ap-guangzhou.cs.tencentcs.com
            secretId: "AKIDKZf1Guiosrx6VndYuJ7mhkgbPXON"
            secretKey: "FTbbv6EC4udUEvV9fAYLmfZEmLoAXp9"
        log:
            path: "runtime/logs"
    --- # 定义应用程序deployment
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        namespace: management
        name: management-deploy
        labels:
            lname: management-deploy
    spec:
        replicas: 10
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  management-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: management-pod
            spec:
                imagePullSecrets:
                    - name: dockerregistry
                containers:
                    - name: management-c
                        image: 192.168.239.142:5000/management:1.0.0 # 使用了我们自己的镜像注册中心
                        # 镜像拉取策略：Always（适合开发环境）, IfNotPresent（适合生产环境）, Never 
                        imagePullPolicy: IfNotPresent
                        workingDir: /app
                        ports:
                            - containerPort: 8080
                        livenessProbe: # 存活探针
                            httpGet:
                                path: /health # 我们的程序实现了这个接口
                                port: 8080
                            initialDelaySeconds: 5
                            periodSeconds: 5
                            failureThreshold: 3
                            successThreshold: 1
                            timeoutSeconds: 1
                        volumeMounts:
                            - name: management-conf-v
                                mountPath: "/app/conf"
                                readOnly: true
                volumes:
                    - name: management-conf-v
                        configMap:
                            name: management-config
                            items:
                                - key: "config.yaml" # 挂载config.yaml到pod中
                                    path: "config.yaml"
    --- # 定义应用程序的service，没有service无法访问
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: management
        name: management-svc
    spec:
        type: NodePort
        selector:
            name: management-pod
        ports:
            - protocol: TCP
                name: http8080
                port: 8081 # 服务端口
                targetPort: 8080 # 容器端口
                nodePort: 30080 # 节点端口
    1
    2
    3
    4
    5
    6
    kubectl apply -f Deployment.yaml
    kubectl get -f Deployment.yaml # 等到副本都启动完毕，会告知我们CLUSTER-IP
    curl http://10.110.145.209:8081/health 
    kubectl describe -n management pods management-deploy-nqownrgpqug # 找到容器ip
    curl http://10.244.2.6:8080/health
    curl http://192.168.239.141:30080/write/cls

filebeat从文件中采集日志

  • 场景，已经有一个可向腾讯云日志服务输出日志的go应用程序，需要使用filebeat（与app是两个容器）进行日志采集并发送到腾讯云上，该应用将日志写道文件里由filebeat进行采集再发送到腾讯云。
  • 先在腾讯云日志中心，创建一个filebeat，复制配置文件。
  • Deployment-filebeat.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    # 定义应用程序命名空间
    apiVersion: v1
    kind: Namespace # 命名空间用于隔离应用程序，默认是default
    metadata:
        name: management 
    --- # 粘贴密钥到这里
    apiVersion: v1
    data:
        .dockerconfigjson: qoienopgqeurgniuegrqwe
    kind: Secret
    metadata:
        name: dockerregistry
        namespace: management # 为了隔离性，这里也要写命名空间
    type: kubernetes.io/dockerconfigjson
    --- # 定义应用程序configmap
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: management-config
        namespace: management
    data:
        config.yaml: | # 将原先程序中使用的config.yaml的内容作为configMap的数据，注意这里是定义一种资源，实际的数据还是需要从存储中映射过来的
        cls:
            topicId: 55d523ba-1795-4457-053-c88100a2076
            endpoint: ap-guangzhou.cs.tencentcs.com
            secretId: "AKIDKZf1Guiosrx6VndYuJ7mhkgbPXON"
            secretKey: "FTbbv6EC4udUEvV9fAYLmfZEmLoAXp9"
        log:
            path: "runtime/logs"
    --- # 添加filebeat配置
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: filebeat-config
        namespace: management
    data:
        filebeat.yaml: | # 这一段在应用程序中copy了一个新的filebeat.yml文件
        fliebeat.inputs
         - type：filestream
           id: myapp-management-id
           paths:
             - "/app/runtime/logs/*.log"
        output.kafka:
            enabed: true
            hosts: ["gz-producer.cs.tencentcs.com:9096"]
            topic: "55d523ba-1795-4457-053-ac881002076"
            version: "0.11.0.2"
            compression: "gzip"
            username："742b399-7454-423b-8079-5afb2f4f3413"
            password: "AKIDKZf1GUiosrX6ndYuJ7mhkgbPOXON#FTbbv6EC4UdUEvV9fAYmfZuEmLoAXp9"
            codec.format:
                string: "${{message}}"
    --- # 定义应用程序deployment
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        namespace: management
        name: management-deploy
        labels:
            lname: management-deploy
    spec:
        replicas: 10
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  management-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: management-pod
            spec:
                imagePullSecrets:
                    - name: dockerregistry
                containers:
                    - name: management-c
                        image: 192.168.239.142:5000/management:1.0.0 # 使用了我们自己的镜像注册中心
                        # 镜像拉取策略：Always（适合开发环境）, IfNotPresent（适合生产环境）, Never 
                        imagePullPolicy: IfNotPresent
                        workingDir: /app
                        ports:
                            - containerPort: 8080
                        livenessProbe: # 存活探针
                            httpGet:
                                path: /health # 我们的程序实现了这个接口
                                port: 8080
                            initialDelaySeconds: 5
                            periodSeconds: 5
                            failureThreshold: 3
                            successThreshold: 1
                            timeoutSeconds: 1
                        volumeMounts:
                            - name: management-conf-v
                                mountPath: "/app/conf" # 映射到目录的写法
                                readOnly: true
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                    - name: filebeat-c
                        image: docker.elastic.co/beats/filebeat:8.9.0
                        volumeMounts:
                            - name: filebeat-conf-v
                                mountPath: "/usr/share/filebeat/filebeat.yml" # 映射到文件的写法
                                subPath: "filebeat.yml" # 要写这个
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                volumes:
                    - name: management-conf-v
                        configMap:
                            name: management-config
                            items:
                                - key: "config.yaml" # 挂载config.yaml到pod中
                                    path: "config.yaml"
                    - name: filebeat-conf-v
                        configMap:
                            name: filebeat-config # 挂载新的配置文件，因为文件名和路径一样，所以可以不用写items 
                    - name: management-log-v # 创建一个新的目录，因为需要两个容器都能访问到这个位置，一个app去写日志，一个
                        emptyDir:
                            sizeLimit: 500Mi 
    --- # 定义应用程序的service，没有service无法访问
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: management
        name: management-svc
    spec:
        type: NodePort
        selector:
            name: management-pod
        ports:
            - protocol: TCP
                name: http8080
                port: 8081 # 服务端口
                targetPort: 8080 # 容器端口
                nodePort: 30080 # 节点端口
    1
    2
    3
    4
    kubectl delete -f Deployment.yaml
    kubectl apply -f Deployment-filebeat.yaml
    curl http://192.168.239.141:30080/write/cls
    curl http://192.168.239.141:30080/write/file # 看能不能被同步到腾讯云

filebeat从文件中采集日志(守护进程\sidecar模式)

  • 场景，已经有一个可向腾讯云日志服务输出日志的go应用程序，需要使用filebeat（filebeat作为守护进程存在）进行日志采集并发送到腾讯云上，该应用将日志写道文件里由filebeat进行采集再发送到腾讯云。
  • Deployment-filebeat-ds.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    # 定义应用程序命名空间
    apiVersion: v1
    kind: Namespace # 命名空间用于隔离应用程序，默认是default
    metadata:
        name: management 
    --- # 粘贴密钥到这里
    apiVersion: v1
    data:
        .dockerconfigjson: qoienopgqeurgniuegrqwe
    kind: Secret
    metadata:
        name: dockerregistry
        namespace: management # 为了隔离性，这里也要写命名空间
    type: kubernetes.io/dockerconfigjson
    --- # 定义应用程序configmap
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: management-config
        namespace: management
    data:
        config.yaml: | # 将原先程序中使用的config.yaml的内容作为configMap的数据，注意这里是定义一种资源，实际的数据还是需要从存储中映射过来的
        cls:
            topicId: 55d523ba-1795-4457-053-c88100a2076
            endpoint: ap-guangzhou.cs.tencentcs.com
            secretId: "AKIDKZf1Guiosrx6VndYuJ7mhkgbPXON"
            secretKey: "FTbbv6EC4udUEvV9fAYLmfZEmLoAXp9"
        log:
            path: "runtime/logs"
    --- # 添加filebeat配置
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: filebeat-config
        namespace: management
    data:
        filebeat.yaml: | # 这一段在应用程序中copy了一个新的filebeat.yml文件
        fliebeat.inputs
         - type：filestream
           id: myapp-management-id
           paths:
             - "/app/runtime/logs/*.log"
        output.kafka:
            enabed: true
            hosts: ["gz-producer.cs.tencentcs.com:9096"]
            topic: "55d523ba-1795-4457-053-ac881002076"
            version: "0.11.0.2"
            compression: "gzip"
            username："742b399-7454-423b-8079-5afb2f4f3413"
            password: "AKIDKZf1GUiosrX6ndYuJ7mhkgbPOXON#FTbbv6EC4UdUEvV9fAYmfZuEmLoAXp9"
            codec.format:
                string: "${{message}}"
    --- # 定义应用程序deployment
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        namespace: management
        name: management-deploy
        labels:
            lname: management-deploy
    spec:
        replicas: 10
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  management-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: management-pod
            spec:
                imagePullSecrets:
                    - name: dockerregistry
                containers:
                    - name: management-c
                        image: 192.168.239.142:5000/management:1.0.0 # 使用了我们自己的镜像注册中心
                        # 镜像拉取策略：Always（适合开发环境）, IfNotPresent（适合生产环境）, Never 
                        imagePullPolicy: IfNotPresent
                        workingDir: /app
                        ports:
                            - containerPort: 8080
                        livenessProbe: # 存活探针
                            httpGet:
                                path: /health # 我们的程序实现了这个接口
                                port: 8080
                            initialDelaySeconds: 5
                            periodSeconds: 5
                            failureThreshold: 3
                            successThreshold: 1
                            timeoutSeconds: 1
                        volumeMounts:
                            - name: management-conf-v
                                mountPath: "/app/conf" # 映射到目录的写法
                                readOnly: true
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                volumes:
                    - name: management-conf-v
                        configMap:
                            name: management-config
                            items:
                                - key: "config.yaml" # 挂载config.yaml到pod中
                                    path: "config.yaml"
                    - name: management-log-v # 以守护进程的方式运行filebeat，那么需要使用hostpath
                        hostPath:
                            path: /var/log/management
                            type: DirectoryOrCreate
    --- # 定义守护进程
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
        namespace: management
        name: filebeat-ds
        labels:
            lname: filebeat-ds
    spec:
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  filebeat-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: filebeat-pod 
            spec:
                containers:
                    - name: filebeat-c
                        image: docker.elastic.co/beats/filebeat:8.9.0
                        volumeMounts:
                            - name: filebeat-conf-v
                                mountPath: "/usr/share/filebeat/filebeat.yml" # 映射到文件的写法
                                subPath: "filebeat.yml" # 要写这个
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                volumes:
                    - name: filebeat-conf-v
                        configMap:
                            name: filebeat-config 
                    - name: management-log-v # 以守护进程的方式运行filebeat，那么需要使用hostpath
                        hostPath:
                            path: /var/log/management
                            type: DirectoryOrCreate
    --- # 定义应用程序的service，没有service无法访问
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: management
        name: management-svc
    spec:
        type: NodePort
        selector:
            name: management-pod
        ports:
            - protocol: TCP
                name: http8080
                port: 8081 # 服务端口
                targetPort: 8080 # 容器端口
                nodePort: 30080 # 节点端口
    1
    2
    3
    4
    5
    kubectl delete -f  Deployment-filebeat.yaml
    kubectl apply -f Deployment-filebeat-ds.yaml
    # 注意哈，使用守护进程的方式，是所有pod都访问了同一个节点的同一个位置，我们的应用程序对日志使用的是随机uuid作名称，如果不是，有可能会有资源竞争问题。
    curl http://192.168.239.141:30080/write/file # 看能不能被同步到腾讯云
    ls /var/log/management # 我们可以直接在节点的目录中看到日志

监控

  • 场景，接入prometheus对业务数据进行采集。

prometheus业务数据采集

1
go get go.opentelemetry.io/otel/exporters/prometheus@v0.39.0
  • main.go
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    func main(){
        shutdown,err := initProvider()
        if err != nil{
            log.Fatal(err)
        }
        defer shutdown(context.Backgroud())
        writeToFileCounter := initCounter("management-metrics", "WriteToFile", "写入文件次数")
        writeToClsCounter := initCounter("management-metrics", "WriteToCls", "写入cls次数")
        ...
        app1 := app.NewApp(conf, initLogger(fileWriter), initLogger(clsWriter))
        middleApp := app.NewMiddleApp(writeToFileCounter, writeToClsCounter, app1)
        r:= gin.Default()
        initRouter(r, middleApp)
        go serveMetrics() // 普罗米修斯端点需要用协程启动
        ...
    }

    func initProvider() (func(ctx content.Context) error,error){
        ctx := context.background()
        // 初始化资源信息和属性
        res,err := resource.New(
            ctx, resource.WithOS(), resource.WithHost(),
            resource.WithAttributes(
                semconv.ServiceName("management"),
                semconv.ServiceVersion("1.0.0"),
                attribute.String("env", "dev"),
                attribute.String("author", "lin"),
            ),
        )
        if err != nil {
            log.Fatal(err)
        }
        // 初始化Exporter与Reader
        var metricReader metric.Reader
        {
            metricReader, err = prometheus.New()
            if err != nil{
                log.Fatal(err)
            }
        }
        // 初始化Provider
        provider := metric.NewMeterProvider(metric.WithResource(res), metric.WithReader(metricReader))
        // 设置全局provider
        otel.SetMeterProvider(provider)
        return provider.Shutdown, nil
    }

    // 统计接口调用次数
    func initCounter(meterName string, counterName string, desc string) metric.Int64Counter {
        meter := otel.Meter(meterName)
        counter, err := meter.Int64Counter(counterName, metric.WithDescription(desc))
        if err != nil{
            log.Fatal(err)
        }
        return counter
    }

    // 启动普罗米修斯指标输出端点
    func serveMetrics(){
        http.Handle("/metrics", promhttp.Handler())
        err := http.ListenAndServe(":2223", nil)
        if err != nil{
            log.Fatal(err)
        }
    }
  • app.go
    1
    2
    3
    4
    5
    6
    7
    // 以中间件的形式，为原来的接口添加计数
    type AppInterface interface{
        WriteLogsToFile(c *gin.Context)
        WriteLogsToCls(c *gin.Context)
        Ping(c *gin.Context)
        Health(c *gin.Context)
    }
  • middleware.go
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    type middleApp struct{
        writeToFileCounter metric.Int64Counter
        writeToClsCounter metric.Int64Counter
        next AppInterface
    }
    func NewMiddleApp(writeToFileCounter metric.Int64Counter, writeToClsCounter metric.Int64Counter, next AppInterface) AppInterface{
        return &middleApp{
            writeToFileCounter: writeToFileCounter,
            writeToClsCounter: writeToClsCounter,
            next: next,
        }
    }
    func(a *middleApp) WriteLogsToFile(c *gin.Context){
        a.writeToFileCounter.Add(context.Backgroud(), 1)
        a.next.WriteLogsToFile(c)
    }
    func(a *middleApp) WriteLogsToCls(c *gin.Context){
        a.writeToClsCounter.Add(context.Backgroud(), 1)
        a.next.WriteLogsToCls(c)
    }
    func(a *middleApp) Ping(c *gin.Context){
        a.next.Ping(c)
    }
    func(a *middleApp) Health(c *gin.Context){
        a.next.Health(c)
    }

prometheus单节点部署

  • prometheus.yml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    global:
        scrape_interval: 20s # 20s采集一次
        scrape_timeout: 10s # 获取数据的超时时间
        evaluation_interval: 20s # 规则评估频率 
    # 采集配置
    scrape_configs:
        - job_name: k8s-cadvisor
            kubernetes_sd_configs:
                - role: node
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            scheme: https
            relabel_configs:
                - action: replace
                    replacement: kubernetes.default.svc:443
                    target_label: __address__
                - action: replace
                    source_label: 
                        - __meta_kubernetes_node_name
                    regex: (.*)
                    replacement: /api/v1/node/$1/proxy/metrics/cadvisor
                    target_label: __metrics_path__
                - action: labelmap
                    regex: __meta_kubernetes_node_label_(.*)
        - job_name: k8s-node-exporter
            kubernetes_sd_configs:
                - role: pod
                    namespaces:
                        name:
                            - monitor
                    selectors:
                        - role: pod
                            label: name=node-exporter-pod
            scheme: http
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
                - action: replace
                    source_labels: 
                        - __meta_kubernetes_pod_node_name
                    target_label: node
                - action: replace
                    source_labels:
                        - __meta_kubernetes_pod_ip
                    regex: (.*)
                    replacement: $1:9100
                    target_label: __address__
                - action: replace
                    replacement: /metrics
                    target_label: __metrics_path__   
        - job_name: bus-data-collector
            kubernetes_sd_configs:
                - role: pod
                    namespaces:
                        names:
                        - management
                    selectors：
                        - role: pod
                            label: name=management-pod
            scheme: http
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
                - action: replace
                    source_labels: 
                        - __meta_kubernetes_pod_name
                    target_label: pod_name
                - action: replace
                    source_labels:
                        - __meta_kubernetes_pod_ip
                    regex: (.*)
                    replacement: $1:2223
                    target_label: __address__
                - action: replace
                    replacement: /metrics
                    target_label: __metrics_path__
  • monitor.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    apiVersion: v1
    kind: Namespace
    metadata:
        name: monitor
    --- # Node Exporter 要以守护进程部署
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
        namespace: monitor
        name: node-exporter-ds
        labels:
            lname: node-exporter-ds
    spec:
        selector: 
            matchLabels:
                name: node-exporter-ds
        template:
            metadata: 
                labels:
                    name: node-exporter-ds
            spec:
                tolerations: # 需要容忍度，因为master节点也部署
                    - key: "node-role.kubernetes.io/control-plane"
                        operator: "Exists"
                        effect: "NoSchedule"
                containers:
                    - name: node-exporter
                        image: quay.io/prometheus/node-exporter:v1.6.1
                        args:
                            - "--path.rootfs=/host"
                        volumeMounts:
                            - name: host-root
                                mountPath: "/host"
                                readOnly: true
                volumes: 
                    - name: host-root
                        hostPath:
                            path: /
                            type: Directory
    --- 
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: monitor
        name: node-exporter-svc
    spec:
        type: NodePort
        selector:
            name: node-exporter-pod
        ports:
            - protocol: TCP
                name: http9100
                port: 9100 # 服务端口
                targetPort: 9100 # 容器端口
                nodePort: 30091 # 节点端口
    --- # 添加prometheus配置
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: prometheus-config
        namespace: monitor
    data:
        prometheus.yml: | 
        global:
        scrape_interval: 20s # 20s采集一次
        scrape_timeout: 10s # 获取数据的超时时间
        evaluation_interval: 20s # 规则评估频率 
        # 采集配置
        scrape_configs:
            - job_name: k8s-cadvisor
            kubernetes_sd_configs:
                - role: node
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            scheme: https
            relabel_configs:
                - action: replace
                    replacement: kubernetes.default.svc:443
                    target_label: __address__
                - action: replace
                    source_label: 
                        - __meta_kubernetes_node_name
                    regex: (.*)
                    replacement: /api/v1/node/$1/proxy/metrics/cadvisor
                    target_label: __metrics_path__
                - action: labelmap
                    regex: __meta_kubernetes_node_label_(.*)
        - job_name: k8s-node-exporter
            kubernetes_sd_configs:
                - role: pod
                    namespaces:
                        name:
                            - monitor
                    selectors:
                        - role: pod
                            label: name=node-exporter-pod
            scheme: http
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
                - action: replace
                    source_labels: 
                        - __meta_kubernetes_pod_node_name
                    target_label: node
                - action: replace
                    source_labels:
                        - __meta_kubernetes_pod_ip
                    regex: (.*)
                    replacement: $1:9100
                    target_label: __address__
                - action: replace
                    replacement: /metrics
                    target_label: __metrics_path__   
        - job_name: bus-data-collector
            kubernetes_sd_configs:
                - role: pod
                    namespaces:
                        names:
                        - management
                    selectors：
                        - role: pod
                            label: name=management-pod
            scheme: http
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
                - action: replace
                    source_labels: 
                        - __meta_kubernetes_pod_name
                    target_label: pod_name
                - action: replace
                    source_labels:
                        - __meta_kubernetes_pod_ip
                    regex: (.*)
                    replacement: $1:2223
                    target_label: __address__
                - action: replace
                    replacement: /metrics
                    target_label: __metrics_path__  
    --- # 为prometheus分配一个有高权限的角色
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
        name: prometheus
    rules:
        - apiGroups: [""]
            resources:
                - nodes
                - nodes/proxy
                - services
                - endpoints
                - pods
            verbs: ["get", "list", "watch"]
        - nonResourceURLs: ["/metrics"]
            verbs: ["get"]
    --- 
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
        name: prometheus
    roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: prometheus
    subjects:
        - kind: ServiceAccount
            name: default
            namespace: monitor
    --- # 部署prometheus
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        namespace: monitor
        name: prometheus-deploy
        labels:
            lname: prometheus-deploy
    spec:
        replicas: 1
        selector: 
            matchLabels:
                name:  prometheus-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: prometheus-pod
            spec:
                containers:
                    - name: prometheus-c
                        image: prom/prometheus:v2.37.9 
                        ports:
                            - containerPort: 9090
                        args:
                            - "--config.file=/etc/prometheus/prometheus.yml"
                        volumeMounts:
                            - name: prometheus-conf-v
                                mountPath: "/etc/prometheus/"
                                readOnly: true
                volumes:
                    - name: prometheus-conf-v
                        configMap:
                            name: prometheus-config
    --- # prometheus service
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: monitor
        name: prometheus-svc
    spec:
        type: NodePort
        selector:
            name: prometheus-pod
        ports:
            - protocol: TCP
                name: http9090
                port: 9090 # 服务端口
                targetPort: 9090 # 容器端口
                nodePort: 30090 # 节点端口
    1
    2
    3
    kubectl apply -f monitor.yaml
    http://192.168.239.157:30091 # Node export
    http://192.168.239.157:30090 # prometheus

prometheus节点发现

  • Deployment-filebeat-ds.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    # 定义应用程序命名空间
    apiVersion: v1
    kind: Namespace # 命名空间用于隔离应用程序，默认是default
    metadata:
        name: management 
    --- # 粘贴密钥到这里
    apiVersion: v1
    data:
        .dockerconfigjson: qoienopgqeurgniuegrqwe
    kind: Secret
    metadata:
        name: dockerregistry
        namespace: management # 为了隔离性，这里也要写命名空间
    type: kubernetes.io/dockerconfigjson
    --- # 定义应用程序configmap
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: management-config
        namespace: management
    data:
        config.yaml: | # 将原先程序中使用的config.yaml的内容作为configMap的数据，注意这里是定义一种资源，实际的数据还是需要从存储中映射过来的
        cls:
            topicId: 55d523ba-1795-4457-053-c88100a2076
            endpoint: ap-guangzhou.cs.tencentcs.com
            secretId: "AKIDKZf1Guiosrx6VndYuJ7mhkgbPXON"
            secretKey: "FTbbv6EC4udUEvV9fAYLmfZEmLoAXp9"
        log:
            path: "runtime/logs"
    --- # 添加filebeat配置
    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: filebeat-config
        namespace: management
    data:
        filebeat.yaml: | # 这一段在应用程序中copy了一个新的filebeat.yml文件
        fliebeat.inputs
         - type：filestream
           id: myapp-management-id
           paths:
             - "/app/runtime/logs/*.log"
        output.kafka:
            enabed: true
            hosts: ["gz-producer.cs.tencentcs.com:9096"]
            topic: "55d523ba-1795-4457-053-ac881002076"
            version: "0.11.0.2"
            compression: "gzip"
            username："742b399-7454-423b-8079-5afb2f4f3413"
            password: "AKIDKZf1GUiosrX6ndYuJ7mhkgbPOXON#FTbbv6EC4UdUEvV9fAYmfZuEmLoAXp9"
            codec.format:
                string: "${{message}}"
    --- # 定义应用程序deployment
    apiVersion: apps/v1
    kind: Deployment
    metadata:
        namespace: management
        name: management-deploy
        labels:
            lname: management-deploy
    spec:
        replicas: 10
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  management-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: management-pod
            spec:
                imagePullSecrets:
                    - name: dockerregistry
                containers:
                    - name: management-c
                        image: 192.168.239.142:5000/management:1.0.0 # 使用了我们自己的镜像注册中心
                        # 镜像拉取策略：Always（适合开发环境）, IfNotPresent（适合生产环境）, Never 
                        imagePullPolicy: IfNotPresent
                        workingDir: /app
                        ports:
                            - containerPort: 8080
                        livenessProbe: # 存活探针
                            httpGet:
                                path: /health # 我们的程序实现了这个接口
                                port: 8080
                            initialDelaySeconds: 5
                            periodSeconds: 5
                            failureThreshold: 3
                            successThreshold: 1
                            timeoutSeconds: 1
                        volumeMounts:
                            - name: management-conf-v
                                mountPath: "/app/conf" # 映射到目录的写法
                                readOnly: true
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                volumes:
                    - name: management-conf-v
                        configMap:
                            name: management-config
                            items:
                                - key: "config.yaml" # 挂载config.yaml到pod中
                                    path: "config.yaml"
                    - name: management-log-v # 以守护进程的方式运行filebeat，那么需要使用hostpath
                        hostPath:
                            path: /var/log/management
                            type: DirectoryOrCreate
    --- # 定义守护进程
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
        namespace: management
        name: filebeat-ds
        labels:
            lname: filebeat-ds
    spec:
        selector: # 选中哪些pod使得当前deployment来管理
            matchLabels:
                name:  filebeat-pod 
        template: # pod的模板
            metadata: 
                labels:
                    name: filebeat-pod 
            spec:
                containers:
                    - name: filebeat-c
                        image: docker.elastic.co/beats/filebeat:8.9.0
                        volumeMounts:
                            - name: filebeat-conf-v
                                mountPath: "/usr/share/filebeat/filebeat.yml" # 映射到文件的写法
                                subPath: "filebeat.yml" # 要写这个
                            - name: management-log-v
                                mountPath: "/app/runtime/logs"
                volumes:
                    - name: filebeat-conf-v
                        configMap:
                            name: filebeat-config 
                    - name: management-log-v # 以守护进程的方式运行filebeat，那么需要使用hostpath
                        hostPath:
                            path: /var/log/management
                            type: DirectoryOrCreate
    --- # 定义应用程序的service，没有service无法访问
    apiVersion: v1
    kind: Service
    metadata: 
        namesapce: management
        name: management-svc
    spec:
        type: NodePort
        selector:
            name: management-pod
        ports:
            - protocol: TCP
                name: http8080
                port: 8081 # 服务端口
                targetPort: 8080 # 容器端口
                nodePort: 30080 # 节点端口
            - protocol: TCP
                name: http2223
                port: 2223
                targetPort: 2223
                nodePort: 30023
    1
    kubectl apply -f Deployment-filebeat-ds.yaml
  • 节点过滤参见prometheus.yml
    1
    kubectl apply -f monitor.yaml

prometheus指标计算

1
2
3
4
5
6
7
8
9
10
11
12
# 节点15分钟CPU使用率
100 - sum(increase(node_cpu_seconds_total{mode="idle"}[15m])) by (instance)/sum(increase(node_cpu_seconds_total[15m])) by (instance) * 100
# 内存使用率
(node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTota_bytes * 100
# 磁盘使用率
100 - node_fiesystem_avaibytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100

# 节点容器运行的总进数（一个容器可能有多个进程），非k8spod中的进程无法采集
sum(container_processest{id!="/") by (instance)
3
# 24小时内请求WriteTocis接口的次数
sum(increase(writeToCls_totaL[24h]))
2024-12-17 该篇文章被 Cleofwine 归为分类: Go云原生

上一篇：【go分布式中间件】ElasticSearch 下一篇：【go分布式中间件】Kafka
© 2023-2026 Cleofwine / Email / GitHub / 浙ICP备2024136127号
🌙赠我弯弯一枚月，也赠予我晚星⭐ 本站由Hexo驱动｜使用Hexo-theme-A4主题