日志服务未收集到els中

  • Choerodon平台版本: 0.20.0

  • 遇到问题的执行步骤:

    日志部署完之后,在kibana中未找到相关索引

  • 原因分析:

    choerodon-logging创建出来的fluend-bit的pod是正常的
    fluend-bit的configmap里面的配置的目录下能看到fluend-bit收集到日志信息如下图


    但是在kibana里面找不到相关的索引

  • 疑问:

    尝试使用外部的els,发现修改部署参数之后,choerodon-logging生成的fluend-bit的配置文件中的ES_HOST参数未修改

  1. 请你查看一下 /etc/dodcker/daemon.json 的 “data-root”: “/var/lib/docker”,看是否修改过。

  2. 看一下fluent-bit 的configmap 是否有你需要收集应用的规则。

@Vista
1.data-root有改过


2.fluent-bit的configmap的收集规则应该有

目前 choerodon-logging 默认配置 data-root 为 /var/lib/docker,我需要修改chart支持可配置 data-root ,修改完之后通知你。

啥时候能修改好?我这边比较着急
另外
尝试使用外部的els,发现修改部署参数之后,choerodon-logging生成的fluend-bit的配置文件中的ES_HOST参数未修改

我尽量今天修改完。

好的, 谢谢, 修改完成之后麻烦通知我,
另外, 参数fluent-bit.elasticsearch.host未生效,配置外部els之后,生成的fluend-bit的pod的环境变量ES_HOST还是默认值

你可以通过软链接将 /u01/system/docker 链接到 /var/lib/docker

建了软连接之后好像还是收集不到?

安装日志服务时,指定docker 的data-root 路径

   helm install c7n/choerodon-logging \
    --set fluent-bit.elasticsearch.host="elasticsearch.logging" \
    --set fluent-bit.docker.data="<you docker data-root path>" \
    --name=choerodon-logging \
    --namespace=logging \
    --version=0.8.1

@Vista
新安装之后, 还是没有收集到?

你贴一下 fluent-bit 的日志

@Vista

[2020/02/13 01:31:15] [error] [out_es] could not pack/validate JSON response {"took":15,"errors":true,"items":[{"index":{"_index":"hzero-asgard","_type":"flb_type","_id":"qpkrPHABgwFb7NHH6L0f","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@5b765f43 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 205, completed tasks = 1837]]"}}},{"index":{"_index":"hzero-asgard","_type":"flb_type","_id":"q5krPHABgwFb7NHH6L0f","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@1411a7b6 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads [2020/02/13 01:31:15] [ warn] [out_es] Elasticsearch error {"took":15,"errors":true,"items":[{"index":{"_index":"hzero-asgard","_type":"flb_type","_id":"qpkrPHABgwFb7NHH6L0f","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@5b765f43 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 205, completed tasks = 1837]]"}}},{"index":{"_index":"hzero-asgard","_type":"flb_type","_id":"q5krPHABgwFb7NHH6L0f","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@1411a7b6 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks [2020/02/13 01:31:16] [error] [out_es] could not pack/validate JSON response {"took":18,"errors":true,"items":[{"index":{"_index":"hzero-file","_type":"flb_type","_id":"yrYrPHABIQwU6_NY6xJS","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@77785de6 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks = 207, completed tasks = 1998]]"}}},{"index":{"_index":"hzero-file","_type":"flb_type","_id":"y7YrPHABIQwU6_NY6xJS","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@2d8324 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, q [2020/02/13 01:31:16] [ warn] [out_es] Elasticsearch error {"took":18,"errors":true,"items":[{"index":{"_index":"hzero-file","_type":"flb_type","_id":"yrYrPHABIQwU6_NY6xJS","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@77785de6 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks = 207, completed tasks = 1998]]"}}},{"index":{"_index":"hzero-file","_type":"flb_type","_id":"y7YrPHABIQwU6_NY6xJS","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@2d8324 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 205, [2020/02/13 01:31:16] [ warn] [engine] Task cannot be retried: task_id=5 thread_id=2 output=es.0 [2020/02/13 01:31:17] [error] [out_es] could not pack/validate JSON response {"took":14,"errors":true,"items":[{"index":{"_index":"hzero-file","_type":"flb_type","_id":"7JkrPHABgwFb7NHH78Ul","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@33de1367 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 207, completed tasks = 1838]]"}}},{"index":{"_index":"hzero-file","_type":"flb_type","_id":"7ZkrPHABgwFb7NHH78Ul","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@13e9903c on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, [2020/02/13 01:31:17] [ warn] [out_es] Elasticsearch error {"took":14,"errors":true,"items":[{"index":{"_index":"hzero-file","_type":"flb_type","_id":"7JkrPHABgwFb7NHH78Ul","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@33de1367 on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 207, completed tasks = 1838]]"}}},{"index":{"_index":"hzero-file","_type":"flb_type","_id":"7ZkrPHABgwFb7NHH78Ul","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@13e9903c on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks = 20 [2020/02/13 01:31:19] [ warn] [out_es] Elasticsearch error {"took":2,"errors":true,"items":[{"index":{"_index":"kibana","_type":"flb_type","_id":"KZkrPHABgwFb7NHH9tBc","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@59c7f06 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks = 204, completed tasks = 2003]]"}}},{"index":{"_index":"kibana","_type":"flb_type","_id":"KpkrPHABgwFb7NHH9tBc","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@4cc923ad on EsThreadPoolExecutor[name = elasticsearch-data-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6aa6f3ae[Running, pool size = 1, active threads = 1, queued tasks = 208, complet [2020/02/13 01:31:19] [ warn] [engine] Task cannot be retried: task_id=9 thread_id=2 output=es.0 [2020/02/13 01:31:20] [ warn] [out_es] Elasticsearch error {"took":1,"errors":true,"items":[{"index":{"_index":"kibana","_type":"flb_type","_id":"PrYrPHABIQwU6_NY-htD","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@29f3c3b8 on EsThreadPoolExecutor[name = elasticsearch-data-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51e1f1e3[Running, pool size = 1, active threads = 1, queued tasks = 202, completed tasks = 2006]]"}}}]}

这是你 elasticsearch 的问题

现在应该是有了, 不过fluent-bit.elasticsearch.host="elasticsearch.logging"这个参数好像没用, 生成的fluent-bit的pod里面的env还是没变

这个问题已经在最新的文档中修复了。