本文介绍通过elk + filebeat方式收集k8s日志，其中filebeat以sidecar方式部署。elfk最新版本：7.6.2

---

k8s日志收集方案

• 3种日志收集方案：

1. node上部署一个日志收集程序Daemonset方式部署日志收集程序，对本节点 /var/log 和 /var/lib/docker/containers 两个目录下的日志进行采集2. sidecar方式部署日志收集程序每个运行应用程序的pod中附加一个日志收集的容器，使用 emptyDir 共享日志目录让日志容器收集日志3. 应用程序直接推送日志常见的如 graylog 工具，直接修改代码推送日志到es，然后在graylog上展示出来

• 3种收集方案的优缺点：

方案	优点	缺点
1. node上部署一个日志收集程序	每个node仅需部署一个日志收集程序，消耗资源少，对应用无侵入	应用程序日志需要写到标准输出和标准错误输出，不支持多行日志
2. pod中附加一个日志收集容器	低耦合	每个pod启动一个日志收集容器，增加资源消耗
3. 应用程序直接推送日志	无需额外收集工具	侵入应用，增加应用复杂度

下面测试第1种方案：每个node上部署一个日志收集程序，注意elfk版本保持一致。

---

SideCar方式收集k8s日志

• 主机说明：

系统	ip	角色	cpu	内存	hostname
CentOS 7.8	192.168.30.128	master、deploy	>=2	>=2G	master1
CentOS 7.8	192.168.30.129	master	>=2	>=2G	master2
CentOS 7.8	192.168.30.130	node	>=2	>=2G	node1
CentOS 7.8	192.168.30.131	node	>=2	>=2G	node2
CentOS 7.8	192.168.30.132	node	>=2	>=2G	node3

• 搭建k8s集群：

搭建过程省略，具体参考：Kubeadm方式搭建k8s集群 或 二进制方式搭建k8s集群

搭建完成后，查看集群：

kubectl get nodesNAME      STATUS   ROLES    AGE     VERSION
master1   Ready    master   4d16h   v1.14.0
master2   Ready    master   4d16h   v1.14.0
node1     Ready    <none>   4d16h   v1.14.0
node2     Ready    <none>   4d16h   v1.14.0
node3     Ready    <none>   4d16h   v1.14.0

这里为了方便，直接使用之前的k8s集群，注意删除之前实验的k8s资源对象。

• 部署es集群：

mkdir /elfk & cd /elfkvim elasticsearch.yaml

apiVersion: v1
kind: Service
metadata:name: elasticsearchnamespace: defaultlabels:app: elasticsearch
spec:selector:app: elasticsearchclusterIP: Noneports:- name: apiport: 9200- name: discoveryport: 9300---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: elasticsearchnamespace: default
spec:serviceName: elasticsearchreplicas: 3selector:matchLabels:app: elasticsearchtemplate:metadata:labels:app: elasticsearchspec:containers:- name: elasticsearchimage: docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2resources:limits:cpu: 1000mrequests:cpu: 100mports:- containerPort: 9200name: apiprotocol: TCP- containerPort: 9300name: discoveryprotocol: TCPenv:- name: "http.host"value: "0.0.0.0"- name: "network.host"value: "_eth0_"- name: "cluster.name"value: "es-cluster"- name: node.namevalueFrom:fieldRef:fieldPath: metadata.name- name: "bootstrap.memory_lock"value: "false"- name: "discovery.seed_hosts"value: "elasticsearch"- name: "cluster.initial_master_nodes"value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"- name: "discovery.seed_resolver.timeout"value: "10s"- name: "discovery.zen.minimum_master_nodes"value: "2"- name: "ES_JAVA_OPTS"value: "-Xms512m -Xmx512m"volumeMounts:- name: datamountPath: /usr/share/elasticsearch/dataterminationGracePeriodSeconds: 30volumes:- name: datahostPath:path: /home/elasticsearch/data                #该路径为es数据存储目录，自动创建initContainers:- name: fix-permissionsimage: busyboxcommand: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]securityContext:privileged: truevolumeMounts:- name: datamountPath: /usr/share/elasticsearch/data- name: increase-vm-max-mapimage: busyboxcommand: ["sysctl", "-w", "vm.max_map_count=262144"]securityContext:privileged: true- name: increase-fd-ulimitimage: busyboxcommand: ["sh", "-c", "ulimit -n 65536"]securityContext:privileged: true

为了方便，建议提前在所有node节点上拉取elasticsearch镜像：docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2。

kubectl apply -f elasticsearch.yamlkubectl get podNAME              READY   STATUS    RESTARTS   AGE
elasticsearch-0   1/1     Running   0          110s
elasticsearch-1   1/1     Running   0          97s
elasticsearch-2   1/1     Running   0          83skubectl get stsNAME            READY   AGE
elasticsearch   3/3     2m13s

• 部署kibana：

这里为了方便，直接使用NodePort暴露kibana端口。

vim kibana.yaml

apiVersion: v1
kind: Service
metadata:name: kibananamespace: defaultlabels:app: kibana
spec:selector:app: kibanaports:- port: 5601nodePort: 30080type: NodePort---
apiVersion: apps/v1
kind: Deployment
metadata:name: kibananamespace: defaultlabels:app: kibana
spec:selector:matchLabels:app: kibanatemplate:metadata:labels:app: kibanaspec:containers:- name: kibanaimage: docker.elastic.co/kibana/kibana-oss:7.6.2resources:limits:cpu: 1000mrequests:cpu: 100menv:- name: ELASTICSEARCH_HOSTSvalue: "http://elasticsearch:9200"ports:- containerPort: 5601

kubectl apply -f kibana.yamlkubectl get pod |grep kibanakibana-84d7449d95-jg5nt   1/1     Running   0          29skubectl get deploy |grep kibanakibana   1/1     1            1           53s

部署没问题的话，是可以正常访问kibana页面的，浏览器访问ip:30080，

接下来部署应用程序，收集日志。

• 以sidecar方式部署filebeat收集nginx日志：

vim filebeat-nginx.yaml

apiVersion: v1
kind: Service
metadata:name: nginxnamespace: defaultlabels:app: nginx
spec:selector:app: nginxports:- port: 80nodePort: 30090type: NodePort---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-confignamespace: defaultlabels:app: filebeat
data:filebeat.yml: |-filebeat.config:inputs:path: ${path.config}/inputs.d/*.ymlreload.enabled: falsemodules:path: ${path.config}/modules.d/*.ymlreload.enabled: falsefilebeat.inputs:- type: logpaths:- /logdata/*.logtail_files: truefields:pod_name: '${pod_name}'POD_IP: '${POD_IP}'setup.template.name: "nginx-logs"setup.template.pattern: "nginx-logs-*"output.elasticsearch:hosts: ["elasticsearch:9200"]index: "nginx-logs"---
apiVersion: apps/v1
kind: Deployment
metadata:name: nginxnamespace: default
spec:replicas: 1minReadySeconds: 15strategy:rollingUpdate:maxSurge: 1maxUnavailable: 1selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:terminationGracePeriodSeconds: 30containers:- name: filebeatimage: docker.elastic.co/beats/filebeat-oss:7.6.2args: ["-c", "/etc/filebeat/filebeat.yml","-e",]env:- name: POD_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.podIP- name: pod_namevalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namesecurityContext:runAsUser: 0resources:limits:memory: 200Mirequests:cpu: 200mmemory: 200MivolumeMounts:- name: configmountPath: /etc/filebeat/- name: datamountPath: /usr/share/filebeat/data- name: logdatamountPath: /logdata- name: nginximage: nginx:1.17.0ports:- containerPort: 80volumeMounts:- name: logdatamountPath: /var/log/nginxvolumes:- name: dataemptyDir: {}- name: logdataemptyDir: {}- name: configconfigMap:name: filebeat-configitems:- key: filebeat.ymlpath: filebeat.yml

kubectl apply -f filebeat-nginx.yamlkubectl get pod |grep nginxnginx-865f745bdd-q6xwm    2/2     Running   0          16skubectl describe pod nginx-865f745bdd-q6xwmNormal  Scheduled  51s   default-scheduler  Successfully assigned default/nginx-865f745bdd-q6xwm to node2Normal  Pulled     50s   kubelet, node2     Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machineNormal  Created    50s   kubelet, node2     Created container filebeatNormal  Started    50s   kubelet, node2     Started container filebeatNormal  Pulled     50s   kubelet, node2     Container image "nginx:1.17.0" already present on machineNormal  Created    50s   kubelet, node2     Created container nginxNormal  Started    50s   kubelet, node2     Started container nginx

访问nginx页面以产生日志：ip:30090，

• kibana创建索引，查看nginx日志：

可以看到index就是在filebeat配置文件中指定的index——nginx-logs，添加可用的fields：log.file.path后，显示日志来源，

因为上面filebeat配置文件中收集的日志路径是/var/log/nginx/*.log，也可以只指定单个日志（具体的日志路径）并指定index。

• 以sidecar方式部署filebeat收集tomcat日志：

vim filebeat-tomcat.yaml

apiVersion: v1
kind: Service
metadata:name: tomcatnamespace: defaultlabels:app: tomcat
spec:selector:app: tomcatports:- port: 8080nodePort: 30100type: NodePort---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-config-tomcatnamespace: defaultlabels:app: filebeat
data:filebeat.yml: |-filebeat.config:inputs:path: ${path.config}/inputs.d/*.ymlreload.enabled: falsemodules:path: ${path.config}/modules.d/*.ymlreload.enabled: falsefilebeat.inputs:- type: logpaths:- /logdata/*.logtail_files: truefields:pod_name: '${pod_name}'POD_IP: '${POD_IP}'setup.template.name: "tomcat-logs"setup.template.pattern: "tomcat-logs-*"output.elasticsearch:hosts: ["elasticsearch:9200"]index: "tomcat-logs"---
apiVersion: apps/v1
kind: Deployment
metadata:name: tomcatnamespace: default
spec:replicas: 1minReadySeconds: 15strategy:rollingUpdate:maxSurge: 1maxUnavailable: 1selector:matchLabels:app: tomcattemplate:metadata:labels:app: tomcatspec:terminationGracePeriodSeconds: 30containers:- name: filebeatimage: docker.elastic.co/beats/filebeat-oss:7.6.2args: ["-c", "/etc/filebeat/filebeat.yml","-e",]env:- name: POD_IPvalueFrom:fieldRef:apiVersion: v1fieldPath: status.podIP- name: pod_namevalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namesecurityContext:runAsUser: 0resources:limits:memory: 200Mirequests:cpu: 200mmemory: 200MivolumeMounts:- name: configmountPath: /etc/filebeat/- name: datamountPath: /usr/share/filebeat/data- name: logdatamountPath: /logdata- name: tomcatimage: tomcat:8.0.51-alpineports:- containerPort: 8080volumeMounts:- name: logdatamountPath: /usr/local/tomcat/logsvolumes:- name: dataemptyDir: {}- name: logdataemptyDir: {}- name: configconfigMap:name: filebeat-config-tomcatitems:- key: filebeat.ymlpath: filebeat.yml

kubectl apply -f filebeat-tomcat.yamlkubectl get pod |grep tomcattomcat-5c7b6644f4-9hslh   2/2     Running   0          16skubectl describe pod tomcat-5c7b6644f4-9hslhNormal  Scheduled  34s   default-scheduler  Successfully assigned default/tomcat-5c7b6644f4-9hslh to node1Normal  Pulled     33s   kubelet, node1     Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machineNormal  Created    33s   kubelet, node1     Created container filebeatNormal  Started    33s   kubelet, node1     Started container filebeatNormal  Pulled     33s   kubelet, node1     Container image "tomcat:8.0.51-alpine" already present on machineNormal  Created    33s   kubelet, node1     Created container tomcatNormal  Started    33s   kubelet, node1     Started container tomcat

访问tomcat页面以产生日志：ip:30100，

• kibana查看tomcat日志：

可以看到index就是在filebeat配置文件中指定的index——tomcat-logs，添加可用的fields：log.file.path后，显示日志来源，

因为上面filebeat配置文件中收集的日志路径是/usr/local/tomcat/logs/*.log，也可以只指定单个日志（具体的日志路径）并指定index。

• 总结：

上面通过sidecar方式部署filebeat收集k8s日志，比使用logagent方式部署filebeat收集k8s日志更加直接，可以指定具体路径并自定义index，在查看日志时也更加方便，缺点就是会增加资源消耗，不过资源消耗在可接受的范围内。

需要注意的是，filebeat对不同应用程序日志收集的ConfigMap名称尽量不要相同，避免冲突和误删除。

上面为了方便，filebeat收集日志是直接输出到es中的，也可以在集群中再部署一个logstash，由filebeat传输到logstash之后进行日志处理再输出到es中，这里就不演示了。

logstash.yaml（参考）：

apiVersion: v1
kind: ConfigMap
metadata:name: logstash-confignamespace: default
data:logstash.yml: |http.host: "0.0.0.0"path.config: /usr/share/logstash/pipelinelogstash.conf: |input {beats {port => 5044}}filter {#multiline {#pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"#negate => true#what => "previous"#}grok {match => [ "message", "%{TIMESTAMP_ISO8601:logtime} %{LOGLEVEL:level}" ]}}output {elasticsearch {hosts => ["elasticsearch:9200"]index => "your-index-%{+YYYY.MM.dd}"}}---
kind: Service
apiVersion: v1
metadata:name: logstashnamespace: default
spec:selector:app: logstashports:- protocol: TCPport: 5044targetPort: 5044type: ClusterIP---
apiVersion: apps/v1
kind: Deployment
metadata:name: logstashnamespace: default
spec:selector:matchLabels:app: logstashreplicas: 1template:metadata:labels:app: logstashspec:containers:- name: logstashimage: docker.elastic.co/logstash/logstash-oss:7.6.2ports:- containerPort: 5044volumeMounts:- name: configmountPath: /usr/share/logstash/config- name: pipelinemountPath: /usr/share/logstash/pipelinevolumes:- name: configconfigMap:name: logstash-configitems:- key: logstash.ymlpath: logstash.yml- name: pipelineconfigMap:name: logstash-configitems:- key: logstash.confpath: logstash.conf

kubectl get pod |grep logstashlogstash-6b475db5f6-mxxsj   1/1     Running   0          26skubectl logs -f logstash-6b475db5f6-mxxsj[INFO ] 2020-05-17 04:46:22.488 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2020-05-17 04:46:22.501 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-05-17 04:46:22.532 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-05-17 04:46:22.615 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-17 04:46:22.639 [[main]<beats] Server - Starting server on port: 5044kubectl get svc |grep logstashlogstash        ClusterIP   10.96.103.207    <none>        5044/TCP            1m34s

kubectl get pod |grep nginxnginx-865f745bdd-q6xwm      2/2     Running   0          62mkubectl exec -it nginx-865f745bdd-q6xwm bash[root@nginx-865f745bdd-q6xwm filebeat]# yum install -y telnet[root@nginx-865f745bdd-q6xwm filebeat]# telnet logstash 5044Trying 10.96.103.207...
Connected to logstash.
Escape character is '^]'.

简单测试部署没问题，filebeat容器测试端口连接也没问题，此处省略进一步配置logstash过滤处理日志的过程。

前面提过，sidecar方式部署filebeat的话，elk组件是否部署在k8s集群内无关紧要，只需要连接没问题即可。上面部署的elk组件是在k8s集群内，接下来将elk部署于k8s集群外测试日志收集。

---