部署 EFK 插件

官方文件目录：kubernetes/cluster/addons/fluentd-elasticsearch

  1. $ ls *.yaml
  2. es-controller.yaml es-rbac.yaml es-service.yaml  fluentd-es-ds.yaml  kibana-controller.yaml  kibana-service.yaml fluentd-es-rbac.yaml
  • 新加了 es-rbac.yaml 和 fluentd-es-rbac.yaml 文件，定义了 elasticsearch 和 fluentd 使用的 Role 和 RoleBinding；

已经修改好的 yaml 文件见：EFK。

配置 es-controller.yaml

  1. $ diff es-controller.yaml.orig es-controller.yaml
  2. 22a23
  3. >       serviceAccountName: elasticsearch
  4. 24c25
  5. <       - image: gcr.io/google_containers/elasticsearch:v2.4.1-2
  6. ---
  7. >       - image: onlyerich/elasticsearch:v2.4.1-2

配置 es-service.yaml

无需配置；

配置 fluentd-es-ds.yaml

  1. $ diff fluentd-es-ds.yaml.orig fluentd-es-ds.yaml
  2. 23a24
  3. >       serviceAccountName: fluentd
  4. 26c27
  5. <         image: gcr.io/google_containers/fluentd-elasticsearch:1.22
  6. ---
  7. >         image: onlyerich/fluentd-elasticsearch:1.22

配置 kibana-controller.yaml

  1. $ diff kibana-controller.yaml.orig kibana-controller.yaml
  2. 22c22
  3. <         image: gcr.io/google_containers/kibana:v4.6.1-1
  4. ---
  5. >         image: onlyerich/kibana:v4.6.1-1

给 Node 设置标签

DaemonSet fluentd-es-v1.22 只会调度到设置了标签 beta.kubernetes.io/fluentd-ds-ready=true 的 Node，需要在期望运行 fluentd 的 Node 上设置该标签；

  1. $ kubectl get nodes
  2. NAME        STATUS    AGE       VERSION
  3. 10.64.3.7   Ready     1d        v1.6.2
  5. $ kubectl label nodes 10.64.3.7 beta.kubernetes.io/fluentd-ds-ready=true
  6. node "10.64.3.7" labeled

执行定义文件

  1. $ pwd
  2. /root/kubernetes/cluster/addons/fluentd-elasticsearch
  3. $ ls *.yaml
  4. es-controller.yaml es-rbac.yaml es-service.yaml  fluentd-es-ds.yaml  kibana-controller.yaml  kibana-service.yaml fluentd-es-rbac.yaml
  5. $ kubectl create -f .
  6. $

检查执行结果

  1. $ kubectl get deployment -n kube-system|grep kibana
  2. kibana-logging         1         1         1            1           2m
  4. $ kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'
  5. elasticsearch-logging-v1-kwc9w          1/1       Running   0          4m
  6. elasticsearch-logging-v1-ws9mk          1/1       Running   0          4m
  7. fluentd-es-v1.22-g76x0                  1/1       Running   0          4m
  8. kibana-logging-324921636-ph7sn          1/1       Running   0          4m
  10. $ kubectl get service  -n kube-system|grep -E 'elasticsearch|kibana'
  11. elasticsearch-logging   10.254.128.156   <none>        9200/TCP        3m
  12. kibana-logging          10.254.88.109    <none>        5601/TCP        3m

kibana Pod 第一次启动时会用较长时间(10-20分钟)来优化和 Cache 状态页面，可以 tailf 该 Pod 的日志观察进度：

  1. $ kubectl logs kibana-logging-324921636-ph7sn -n kube-system -f
  2. ELASTICSEARCH_URL=http://elasticsearch-logging:9200
  3. server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging
  4. {"type":"log","@timestamp":"2017-04-08T09:30:30Z","tags":["info","optimize"],"pid":7,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"}
  5. {"type":"log","@timestamp":"2017-04-08T09:44:01Z","tags":["info","optimize"],"pid":7,"message":"Optimization of bundles for kibana and statusPage complete in 811.00 seconds"}
  6. {"type":"log","@timestamp":"2017-04-08T09:44:02Z","tags":["status","plugin:kibana@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

访问 kibana

  1. 通过 kube-apiserver 访问：

    获取 monitoring-grafana 服务 URL

    1.  $ kubectl cluster-info
    2.  Kubernetes master is running at https://10.64.3.7:6443
    3.  Elasticsearch is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
    4.  Heapster is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/heapster
    5.  Kibana is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging
    6.  KubeDNS is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns
    7.  kubernetes-dashboard is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
    8.  monitoring-grafana is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
    9.  monitoring-influxdb is running at https://10.64.3.7:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

    由于 kube-apiserver 开启了 RBAC 授权，而浏览器访问 kube-apiserver 的时候使用的是匿名证书，所以访问安全端口会导致授权失败。这里需要使用非安全端口访问 kube-apiserver：

    浏览器访问 URL： http://10.64.3.7:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging

  2. 通过 kubectl proxy 访问：

    创建代理

    1.  $ kubectl proxy --address='10.64.3.7' --port=8086 --accept-hosts='^*$'
    2.  Starting to serve on 10.64.3.7:8086

    浏览器访问 URL：http://10.64.3.7:8086/api/v1/proxy/namespaces/kube-system/services/kibana-logging

在 Settings -> Indices 页面创建一个 index（相当于 mysql 中的一个 database），选中 Index contains time-based events，使用默认的 logstash-* pattern，点击 Create ;

es-setting

创建Index后，稍等几分钟就可以在 Discover 菜单下看到 ElasticSearch logging 中汇聚的日志；

es-home