通过kafka进行无限日志logging

概观

我正在尝试使用Docker容器来使Logstash与Kafka 一起工作。

对于我的卡夫卡容器，我正在使用Spotify构build 。 我也使用官方的Logstash容器 。

我想要达到如下所示：

这里是我用于Logstash的configuration：

file_kafka.conf

input { file { path => ["/data/*.log"] start_position => "beginning" } } output { kafka { topic_id => "test" bootstrap_servers => "<host>:9092" } } 

kafka_stdout.conf

 input { kafka { topics => ["test"] bootstrap_servers => "<host>:9092" auto_offset_reset => "latest" } } output { stdout {} } 

这个设置对我来说似乎完全合理。 事实上，它起初实际上是一种工作。

问题

我启动了所涉及的容器，Logstash正确地拾取了我的test.log文件中的单行并将其打印到屏幕上。

但是，我然后运行以下：

echo 'blah' >> test.log

我从logstash收到这个输出：

2017-05-11T18:36:49.090Z %{host} 2017-05-11T18:36:49.082Z %{host} 2017-05-11T18:36:49.076Z %{host} 2017-05-11T18:36:49.069Z %{host} 2017-05-11T18:36:49.062Z %{host} 2017-05-11T18:36:49.053Z %{host} 2017-05-11T18:36:49.044Z %{host} 2017-05-11T18:36:49.034Z %{host} 2017-05-11T18:36:49.025Z %{host} 2017-05-11T18:36:49.016Z %{host} 2017-05-11T18:36:49.009Z %{host} 2017-05-11T18:36:49.003Z %{host} 2017-05-11T18:36:48.995Z %{host} 2017-05-11T18:36:48.988Z %{host} 2017-05-11T18:36:48.982Z %{host} 2017-05-11T18:36:48.975Z %{host} 2017-05-11T18:36:48.968Z %{host} 2017-05-11T18:36:48.948Z %{host} 2017-05-11T18:36:48.927Z %{host} 2017-05-11T18:36:48.911Z %{host} 2017-05-11T18:36:48.896Z %{host} 2017-05-11T18:36:48.881Z %{host} 2017-05-11T18:36:48.868Z %{host} 2017-05-11T18:36:48.857Z %{host} 2017-05-11T18:36:48.849Z %{host} 2017-05-11T18:36:48.841Z %{host} 2017-05-11T18:36:48.830Z %{host} 2017-05-11T18:36:48.820Z %{host} 2017-05-11T18:36:48.805Z %{host} 2017-05-11T18:36:48.791Z %{host} 2017-05-11T18:36:48.779Z %{host} 2017-05-11T18:36:48.492Z %{host} 2017-05-11T18:36:48.483Z %{host} 2017-05-11T18:36:48.471Z %{host} 2017-05-11T18:36:48.463Z %{host} 2017-05-11T18:36:48.454Z %{host} 2017-05-11T18:36:48.445Z %{host} 2017-05-11T18:36:48.437Z %{host} 2017-05-11T18:36:48.427Z %{host} 2017-05-11T18:36:48.420Z %{host} 2017-05-11T18:36:48.412Z %{host} 2017-05-11T18:36:48.406Z %{host} 2017-05-11T18:36:48.399Z %{host} 2017-05-11T18:36:48.391Z %{host} 2017-05-11T18:36:48.385Z %{host} 2017-05-11T18:36:48.378Z %{host} 2017-05-11T18:36:48.371Z %{host} 2017-05-11T18:36:48.356Z %{host} 2017-05-11T18:36:48.286Z %{host} 2017-05-11T18:36:48.278Z %{host} 2017-05-11T18:36:48.269Z %{host} 2017-05-11T18:36:48.261Z %{host} 2017-05-11T18:36:48.246Z %{host} 2017-05-11T18:36:48.238Z %{host} 2017-05-11T18:36:48.229Z %{host} 2017-05-11T18:36:48.219Z %{host} 2017-05-11T18:36:48.209Z %{host} 2017-05-11T18:36:48.201Z %{host} 2017-05-11T18:36:48.191Z %{host} 2017-05-11T18:36:48.182Z %{host} 2017-05-11T18:36:48.175Z %{host} 2017-05-11T18:36:48.167Z %{host} 2017-05-11T18:36:48.112Z %{host} 2017-05-11T18:36:48.103Z %{host} 2017-05-11T18:36:48.095Z %{host} 2017-05-11T18:36:48.086Z %{host} 2017-05-11T18:36:48.077Z %{host} 2017-05-11T18:36:48.068Z %{host} 2017-05-11T18:36:48.059Z %{host} 2017-05-11T18:36:48.048Z %{host} 2017-05-11T18:36:48.031Z %{host} 2017-05-11T18:36:48.016Z %{host} 2017-05-11T18:36:48.009Z %{host} 2017-05-11T18:36:48.002Z %{host} 2017-05-11T18:36:47.994Z %{host} 2017-05-11T18:36:47.986Z %{host} 2017-05-11T18:36:47.972Z %{host} 2017-05-11T18:36:47.947Z %{host} 2017-05-11T18:36:47.923Z %{host} 2017-05-11T18:36:47.904Z %{host} 2017-05-11T18:36:47.879Z %{host} 2017-05-11T18:36:47.864Z %{host} 2017-05-11T18:36:47.853Z %{host} 2017-05-11T18:36:47.842Z %{host} 2017-05-11T18:36:47.834Z %{host} 2017-05-11T18:36:47.820Z %{host} 2017-05-11T18:36:47.810Z %{host} 2017-05-11T18:36:47.799Z %{host} 2017-05-11T18:36:47.792Z %{host} 2017-05-11T18:36:47.785Z %{host} 2017-05-11T18:36:47.778Z %{host} 2017-05-11T18:36:47.768Z %{host} 2017-05-11T18:36:47.757Z %{host} 2017-05-11T18:36:47.741Z %{host} 2017-05-11T18:36:47.722Z %{host} 2017-05-11T18:36:47.702Z %{host} 2017-05-11T18:36:47.690Z %{host} 2017-05-11T18:36:47.674Z %{host} 2017-05-11T18:36:47.667Z %{host} 2017-05-11T18:36:47.659Z %{host} 2017-05-11T18:36:47.649Z %{host} 2017-05-11T18:36:47.639Z %{host} 2017-05-11T18:36:47.632Z %{host} 2017-05-11T18:36:47.626Z %{host} 2017-05-11T18:36:47.619Z %{host} 2017-05-11T18:36:47.612Z %{host} 2017-05-11T18:36:47.603Z %{host} 2017-05-11T18:36:47.592Z %{host} 2017-05-11T18:36:47.555Z %{host} 2017-05-11T18:36:47.537Z %{host} 2017-05-11T18:36:47.523Z %{host} 2017-05-11T18:36:47.513Z %{host} 2017-05-11T18:36:47.505Z %{host} 2017-05-11T18:36:47.492Z %{host} 2017-05-11T18:36:47.484Z %{host} 2017-05-11T18:36:47.476Z %{host} 2017-05-11T18:36:47.468Z %{host} 2017-05-11T18:36:47.456Z %{host} 2017-05-11T18:36:47.447Z %{host} 2017-05-11T18:36:47.437Z %{host} 2017-05-11T18:36:47.425Z %{host} 2017-05-11T18:36:47.416Z %{host} 2017-05-11T18:36:47.403Z %{host} 2017-05-11T18:36:47.395Z %{host} 2017-05-11T18:36:47.388Z %{host} 2017-05-11T18:36:47.379Z %{host} 2017-05-11T18:36:47.372Z %{host} 2017-05-11T18:36:47.364Z %{host} 2017-05-11T18:36:47.351Z %{host} 2017-05-11T18:36:47.318Z %{host} 2017-05-11T18:36:47.308Z %{host} 2017-05-11T18:36:47.293Z %{host} 2017-05-11T18:36:47.274Z %{host} 2017-05-11T18:36:47.255Z %{host} 2017-05-11T18:36:47.244Z %{host} 2017-05-11T18:36:47.232Z %{host} 2017-05-11T18:36:47.215Z %{host} 2017-05-11T18:36:47.205Z %{host} 2017-05-11T18:36:47.197Z %{host} 2017-05-11T18:36:47.188Z %{host} 2017-05-11T18:36:47.180Z %{host} 2017-05-11T18:36:47.170Z %{host} 2017-05-11T18:36:47.162Z %{host} 2017-05-11T18:36:47.155Z %{host} 2017-05-11T18:36:47.147Z %{host} 2017-05-11T18:36:47.141Z %{host} 2017-05-11T18:36:47.134Z %{host} 2017-05-11T18:36:47.127Z %{host} 2017-05-11T18:36:47.119Z %{host} 2017-05-11T18:36:47.111Z %{host} 2017-05-11T18:36:47.096Z %{host} 2017-05-11T18:36:47.082Z %{host} 2017-05-11T18:36:47.064Z %{host} 2017-05-11T18:36:47.047Z %{host} 2017-05-11T18:36:47.034Z %{host} 2017-05-11T18:36:47.026Z %{host} 2017-05-11T18:36:47.014Z %{host} 2017-05-11T18:36:47.006Z %{host} 2017-05-11T18:36:46.996Z %{host} 2017-05-11T18:36:46.975Z %{host} 2017-05-11T18:36:46.965Z %{host} 2017-05-11T18:36:46.935Z %{host} 2017-05-11T18:36:46.928Z %{host} 2017-05-11T18:36:46.921Z %{host} 2017-05-11T18:36:46.914Z %{host} 2017-05-11T18:36:46.907Z %{host} 2017-05-11T18:36:46.899Z %{host} 2017-05-11T18:36:46.888Z %{host} 2017-05-11T18:36:46.867Z %{host} 2017-05-11T18:36:46.845Z %{host} 2017-05-11T18:36:46.833Z %{host} 2017-05-11T18:36:46.817Z %{host} 2017-05-11T18:36:46.798Z %{host} 2017-05-11T18:36:46.783Z %{host} 2017-05-11T18:36:46.759Z %{host} 2017-05-11T18:36:46.743Z %{host} 2017-05-11T18:36:46.725Z %{host} 2017-05-11T18:36:46.712Z %{host} 2017-05-11T18:36:46.695Z %{host} 2017-05-11T18:36:46.685Z %{host} 2017-05-11T18:36:46.677Z %{host} 2017-05-11T18:36:46.666Z %{host} 2017-05-11T18:36:46.659Z %{host} 2017-05-11T18:36:46.642Z %{host} 2017-05-11T18:36:46.631Z %{host} 2017-05-11T18:36:46.620Z %{host} 2017-05-11T18:36:46.606Z %{host} 2017-05-11T18:36:46.593Z %{host} 2017-05-11T18:36:46.577Z %{host} 2017-05-11T18:36:46.568Z %{host} 2017-05-11T18:36:46.549Z %{host} 2017-05-11T18:36:46.541Z %{host} 2017-05-11T18:36:46.532Z %{host} 2017-05-11T18:36:46.524Z %{host} 2017-05-11T18:36:46.507Z %{host} 2017-05-11T18:36:46.491Z %{host} 2017-05-11T18:36:46.474Z %{host} 2017-05-11T18:36:46.429Z %{host} 2017-05-11T18:36:46.420Z %{host} 2017-05-11T18:36:46.408Z %{host} 2017-05-11T18:36:46.397Z %{host} 2017-05-11T18:36:46.382Z %{host} 2017-05-11T18:36:46.353Z %{host} 2017-05-11T18:36:46.345Z %{host} 2017-05-11T18:36:46.337Z %{host} 2017-05-11T18:36:46.330Z %{host} 2017-05-11T18:36:46.323Z %{host} 2017-05-11T18:36:46.314Z %{host} 2017-05-11T18:36:46.302Z %{host} 2017-05-11T18:36:46.293Z %{host} 2017-05-11T18:36:46.279Z %{host} 2017-05-11T18:36:46.265Z %{host} 2017-05-11T18:36:46.252Z %{host} 2017-05-11T18:36:46.181Z %{host} 2017-05-11T18:36:46.152Z %{host} 2017-05-11T18:36:46.128Z %{host} 2017-05-11T18:36:46.110Z %{host} 2017-05-11T18:36:46.088Z %{host} 2017-05-11T18:36:46.075Z %{host} 2017-05-11T18:36:46.067Z %{host} 2017-05-11T18:36:46.058Z %{host} 2017-05-11T18:36:46.045Z %{host} 2017-05-11T18:36:46.025Z %{host} 2017-05-11T18:36:46.016Z %{host} 2017-05-11T18:36:46.008Z %{host} 2017-05-11T18:36:45.995Z %{host} 2017-05-11T18:36:45.986Z %{host} 2017-05-11T18:36:45.972Z %{host} 2017-05-11T18:36:45.934Z %{host} 2017-05-11T18:36:45.915Z %{host} 2017-05-11T18:36:45.898Z %{host} 2017-05-11T18:36:45.887Z %{host} 2017-05-11T18:36:45.855Z dc6258001c01 blah

这段文本被无限重复，直到我终止了logstash进程。 就好像它进入了某种非终止循环。 在这种情况发生的时候，我也看到我的话题抵消了卡夫卡事件的增加。

所以这给我留下了两个问题：

1. 为什么logstash会一遍又一遍地重复读取同一行？
2. 这个消息的格式是什么？ 是不是应该是timestamp host line ？ 它最终看起来是正确的，但在此之前， timestamp %{host}模式被无用地重复了约232次。 为什么？

Answer 1

我自己也遇到了Kafka和RabbitMQ的类似问题。

Logstash并不在意你将configuration分割成单独的文件，最终这样做的能力是供人类使用的（为了使configuration更易于查找和修改）。 在运行时，所有Logstashconfiguration都在stream水线中合并在一起，并被视为单个configuration。 虽然我不一定知道订单是如何应用的，但它可以在一台主机上运行：

 input { file { path => ["/data/*.log"] start_position => "beginning" } kafka { topics => ["test"] bootstrap_servers => "<host>:9092" auto_offset_reset => "latest" } } output { kafka { topic_id => "test" bootstrap_servers => "<host>:9092" } stdout {} } 

考虑到你的一个input被设置为读取最后写入的偏移量，它将只读取，不应用filter，并将其输出到相同的主题。 卡夫卡经纪人没有关于消息本身的概念，但卡夫卡（最终是ZooKeeper）跟踪发行人写入的消息的偏移量和订户读取的消息的偏移量。

您需要运行一个单独的Logstash实例。 要validation没有第二个configuration的input（使用stdout），我build议使用kafkacat来读取你的Kafka主题（它也可以用来发布消息）。

 kafkacat -C -b <broker>:9092 -t <topic> 

唯一的select（我知道的作品），允许您使用一个Logstash实例是，你必须在filter步骤中使用mutate方法，但这会使逻辑复杂化，最好遵守托运人/索引模式与Logstash。