configurationfluentd以正确parsing并将使用docker json-file日志logging驱动程序格式化的java stacktrace转换为弹性的单个消息
我们的服务作为docker实例运行。 鉴于限制是泊坞窗日志logging驱动程序不能被更改为任何不同于默认的json文件驱动程序。 （scala micro）服务输出一个看起来像这样的日志
{"log":"10:30:12.375 [application-akka.actor.default-dispatcher-13] [WARN] [rulekeepr-615239361-v5mtn-7]- cvrslogic.RulekeeprLogicProvider(91) - decision making have failed unexpectedly\n","stream":"stdout","time":"2017-05-08T10:30:12.376485994Z"} {"log":"java.lang.RuntimeException: Error extracting fields to make a lookup for a rule at P2: [failed calculating amount/amountEUR/directive: [failed getting accountInfo of companyId:3303 from deadcart: unexpected status returned: 500]]\n","stream":"stdout","time":"2017-05-08T10:30:12.376528449Z"} {"log":"\u0009at org.assbox.rulekeepr.services.BasicRuleService$$anonfun$lookupRule$2.apply(BasicRuleService.scala:53)\n","stream":"stdout","time":"2017-05-08T10:30:12.376537277Z"} {"log":"\u0009at org.assbox.rulekeepr.services.BasicRuleService$$anonfun$lookupRule$2.apply(BasicRuleService.scala:53)\n","stream":"stdout","time":"2017-05-08T10:30:12.376542826Z"} {"log":"\u0009at scala.concurrent.Future$$anonfun$transform$1$$anonfun$apply$2.apply(Future.scala:224)\n","stream":"stdout","time":"2017-05-08T10:30:12.376548224Z"} {"log":"Caused by: java.lang.RuntimeException: failed calculating amount/amountEUR/directive: [failed getting accountInfo of companyId:3303 from deadcart: unexpected status returned: 500]\n","stream":"stdout","time":"2017-05-08T10:30:12.376674554Z"} {"log":"\u0009at org.assbox.rulekeepr.services.logic.TlrComputedFields$$anonfun$calculatedFields$1.applyOrElse(AbstractComputedFields.scala:39)\n","stream":"stdout","time":"2017-05-08T10:30:12.376680922Z"} {"log":"\u0009at org.assbox.rulekeepr.services.logic.TlrComputedFields$$anonfun$calculatedFields$1.applyOrElse(AbstractComputedFields.scala:36)\n","stream":"stdout","time":"2017-05-08T10:30:12.376686377Z"} {"log":"\u0009at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)\n","stream":"stdout","time":"2017-05-08T10:30:12.376691228Z"} {"log":"\u0009... 19 common frames omitted\n","stream":"stdout","time":"2017-05-08T10:30:12.376720255Z"} {"log":"Caused by: java.lang.RuntimeException: failed getting accountInfo of companyId:3303 from deadcart: unexpected status returned: 500\n","stream":"stdout","time":"2017-05-08T10:30:12.376724303Z"} {"log":"\u0009at org.assbox.rulekeepr.services.mixins.DCartHelper$$anonfun$accountInfo$1.apply(DCartHelper.scala:31)\n","stream":"stdout","time":"2017-05-08T10:30:12.376729945Z"} {"log":"\u0009at org.assbox.rulekeepr.services.mixins.DCartHelper$$anonfun$accountInfo$1.apply(DCartHelper.scala:24)\n","stream":"stdout","time":"2017-05-08T10:30:12.376734254Z"} {"log":"\u0009... 19 common frames omitted\n","stream":"stdout","time":"2017-05-08T10:30:12.37676087Z"} 如何利用fluentd指令正确地组合包含堆栈跟踪的以下日志事件，以便将它们作为单个消息发送到弹性域？
我完全控制了所使用的logback appender模式，所以我可以改变日志值的出现顺序，甚至改变appender类。
我们正在使用k8s，事实certificate，它不是直接改变docker日志logging驱动程序，所以我们正在寻找一个解决scheme，将能够处理给定的例子。
我不太在意将日志级别，线程，日志logging提取到特定的密钥中，所以我可以稍后在kibana中轻松地进行筛选。 有，但不太重要。 重要的是准确地parsing时间戳，直到毫秒，并将其用作实际的日志甚至是时间戳，因为它是弹性的。
你可以尝试使用fluentd-plugin-grok-parser – 但是我遇到同样的问题 – 看起来\ u0009 tab字符不被识别，所以使用fluentd-plugin-detect-exceptions不会检测到多行exception – 至less在我的尝试中还没有…。
你可以使用fluent-plugin-concat 。
例如在Fluentd v0.14.x中，
<source> @type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true <parse> @type json </parse> @label @INPUT </source> <label @INPUT> <filter kubernetes.**> @type concat key log multiline_start_regexp ^\d{2}:\d{2}:\d{2}\.\d+ continuous_line_regexp ^(\s+|java.lang|Caused by:) separator "" flush_interval 3s timeout_label @PARSE </filter> <match kubernetes.**> @type relabel @label @PARSE </match> </label> <label @PARSE> <filter kubernetes.**> @type parser key_name log inject_key_prefix log. <parse> @type multiline_grok grok_failure_key grokfailure <grok> pattern YOUR_GROK_PATTERN </grok> </parse> </filter> <match kubernetes.**> @type relabel @label @OUTPUT </match> </label> <label @OUTPUT> <match kubernetes.**> @type stdout </match> </label>
类似的问题：