使用stream畅和水槽无限增加的文件描述符的数量
在Docker容器中运行stream畅的代理和stream畅的插件。 一段时间后,我的容器内的一些程序失败,出现下一个错误:
Check failed: _s.ok() Bad status: Runtime error: Could not create thread: Resource temporarily unavailable (error 11)
我发现Docker容器中的文件描述符的数量无限增加:
sudo lsof| grep flume | wc -l 469292
过了一段时间,描述符的数量变大了,那么文件描述符就会受到限制。 看起来,节点连接是以错误的方式处理的:文件描述符在事务结束后仍然存在。
Fluendconfiguration:
<source> @type forward tag forward_1 bind 0.0.0.0 port 24224 linger_timeout 0 </source> <match forward_1> @type copy <store> @type flume timeout 15 host localhost port 33333 </store> </match>
烟气configuration:
forward_1.sources = source1 forward_1.channels = channel1 forward_1.sinks = sink1 # forward_1.sources.source1.type = thrift forward_1.sources.source1.bind = localhost forward_1.sources.source1.port = 33333 forward_1.sources.source1.channels = channel1 # forward_1.channels.channel1.type = memory forward_1.channels.channel1.capacity = 10000 forward_1.channels.channel1.transactionCapacity = 1000 # forward_1.sinks.sink1.type = org.apache.kudu.flume.sink.KuduSink forward_1.sinks.sink1.channel = channel1 forward_1.sinks.sink1.masterAddresses = 10.1.0.1:7051 forward_1.sinks.sink1.tableName = shop_logs forward_1.sinks.sink1.batchSize = 50 forward_1.sinks.sink1.producer = KuduJsonProducer
Docker版本:17.03 Fluentd版本:0.14.19 Flume版本:1.7.0
有没有办法限制文件描述符的数量?
尝试添加“线程”来源到您的水槽configuration。 喜欢这个:
forward_1.sources.source1.threads = number_of_threads
请阅读https://flume.apache.org/FlumeUserGuide.html#thrift-source