如何configurationLogstash来parsingAWS ELB日志？

我想从Logstash中parsing设置在dockerised ELK堆栈中的AWS ELB日志[存储在S3存储桶中]。

我克隆了这个回购。 这是它的文档。

我添加了这样的logstashconfiguration文件[并注释掉所有其他]：

# AWS ELB configuration file ADD ./aws_elb_logs.conf /etc/logstash/conf.d/aws_elb_logs.conf

configuration文件如下： 

 input { s3 { # Logging_user AWS creds access_key_id => "fjnsdfjnsdjfnjsdn" secret_access_key => "asdfsdfsdfsdfsdfsdfsdfsd" bucket => "elb-access-logs" region => "us-west-2" # keep track of the last processed file sincedb_path => "./last-s3-file" codec => "json" type => "elb" } } filter { if [type] == "elb" { grok { match => [ 'message', '%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} (?:%{IP:backend_ip}:%{NUMBER:backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{NUMBER:elb_status_code:int}|-) (?:%{NUMBER:backend_status_code:int}|-) %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} "(?:%{WORD:verb}|-) (?:%{GREEDYDATA:request}|-) (?:HTTP/%{NUMBER:httpversion}|-( )?)" "%{DATA:userAgent}"( %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol})?' ] } grok { match => [ "request", "%{URIPROTO:http_protocol}" ] } geoip { source => "client_ip" target => "geoip" add_tag => [ "geoip" ] } useragent { source => "userAgent" } date { match => ["timestamp", "ISO8601"] } } } output { elasticsearch { hosts => localhost port => "9200" index => "logstash-%{+YYYY.MM.dd}" } stdout { debug => true } }

当我创build容器时，我从Logstash获得以下错误日志： 

 ==> /var/log/logstash/logstash.log <== {:timestamp=>"2016-10-18T13:04:40.798000+0000", :message=>"Pipeline aborted due to error", :exception=>"LogStash::ConfigurationError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in `config_init'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:72:in `config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error} {:timestamp=>"2016-10-18T13:04:43.801000+0000", :message=>"stopping pipeline", :id=>"main"}

我不明白什么是/正在做什么错误！

任何指针将被欢迎..

编辑：

现在是这样的： ==> /var/log/logstash/logstash.log <== {:timestamp=>"2016-10-18T14:26:50.492000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::S3 access_key_id=>\"gsfgdfgdfgdfgdfg\", secret_access_key=>\"dsfgsdfgsdgsdfgsdfg\", bucket=>\"elb-access-logs-dr\", region=>\"us-west-2\", sincedb_path=>\"./last-s3-file\", codec=><LogStash::Codecs::JSON charset=>\"UTF-8\">, type=>\"elb\", use_ssl=>true, delete=>false, interval=>60, temporary_directory=>\"/opt/logstash/logstash\">\n Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.", :level=>:error}

如果您正在使用logstash版本大于2的容器之一，那么弹性search输出插件的configuration就是错误来自的地方。 使用logstash版本2，configuration选项port被删除，端口在hosts选项（ cf doc ）中被configuration为主机。

Interesting Posts