为什么kubelet无法从docker获取统计信息(错误500)?

我刚刚注意到我的kubelet日志上有很多错误,如下所示:

Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.010282 6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "d3c1a6936a84028bd420f75c44a13a6ad7c816b651d33877352f2708dccbd943" Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.015338 6851 server.go:944] GET /stats/kube-system/kubedash-lm97r/cfa615f2-a405-11e5-8e30-525400ad92ba/kubedash: (10.337027ms) 500 Dec 16 15:21:00 kubenode4 kubelet[6851]: goroutine 2288 [running]: Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc20953cfc0, 0x1f4) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.Error(0x7f74e8150938, 0xc20953cfc0, 0xc2088c9e00, 0x72, 0x1f4) Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1274 +0x9a Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).error(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0x7f74ebf63dc0, 0xc2089665f0) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:282 +0x213 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).serveStats(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:1034 +0x78f Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).handleStats(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:440 +0x46 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.*Server.(k8s.io/kubernetes/pkg/kubelet.handleStats)  fm(0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:154 +0x45 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*httpHandler).ServeHTTP(0xc2081de558, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:275 +0x45 Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.(*ServeMux).ServeHTTP(0xc2081f5350, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1541 +0x17d Dec 16 15:21:00 kubenode4 kubelet[6851]: github.com/emicklei/go-restful.Container.ServeHTTP(0x0, 0x0, 0x0, 0xc20824b3e0, 0x2, 0x2, 0xc2081f5350, 0x0, 0x1fc09a0, 0x0, ...) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).ServeHTTP(0xc2080df340, 0x7f74e8150900, 0xc208c38000, 0xc2086c6d00) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:943 +0x162 Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.serverHandler.ServeHTTP(0xc208045f2 [[Go 1.1 package http] 10.233.64.1:57859] Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.010309 6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "534ae8d2b4f4e2a111b7c38f9c67b9189f001f72a26fb6e76c3d71e3ec790a01" Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.015509 6851 server.go:944] GET /stats/kube-system/influxdb-kj5j2/c92a2514-a405-11e5-8e30-525400ad92ba/influxdb: (10.195837ms) 500 Dec 16 15:21:00 kubenode4 kubelet[6851]: goroutine 2306 [running]: Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc20953d140, 0x1f4) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.Error(0x7f74e8150938, 0xc20953d140, 0xc20938f400, 0x72, 0x1f4) Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1274 +0x9a Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).error(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0x7f74ebf63dc0, 0xc208e9c240) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:282 +0x213 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).serveStats(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:1034 +0x78f Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).handleStats(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:440 +0x46 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.*Server.(k8s.io/kubernetes/pkg/kubelet.handleStats)  fm(0x7f74e8150938, 0xc20953d140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:154 +0x45 Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*httpHandler).ServeHTTP(0xc2081de558, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:275 +0x45 Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.(*ServeMux).ServeHTTP(0xc2081f5350, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1541 +0x17d Dec 16 15:21:00 kubenode4 kubelet[6851]: github.com/emicklei/go-restful.Container.ServeHTTP(0x0, 0x0, 0x0, 0xc20824b3e0, 0x2, 0x2, 0xc2081f5350, 0x0, 0x1fc09a0, 0x0, ...) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).ServeHTTP(0xc2080df340, 0x7f74e8150900, 0xc208c38140, 0xc2086c7040) Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:943 +0x162 Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.serverHandler.ServeHTTP(0xc208045f2 [[Go 1.1 package http] 10.233.64.1:57861] Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.016198 6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "9db4584cdd04a376db5c43197073e7e1e014d8c3213dbe6ace1e8abf54fe4de2" Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.016272 6851 server.go:944] GET /stats/kube-system/influxdb-kj5j2/c92a2514-a405-11e5-8e30-525400ad92ba/grafana: (131.759  s) 500 

我使用heapster来监视集群,看起来'stats'的调用得到了答案,但是由于不明原因,它被认为是500错误。

我在裸机服务器上运行一个集群。

kubernetes版本:v1.1.3
docker工人:1.9.1
操作系统:debian 8.1

对我来说似乎是一个错误。 我已经提交了https://github.com/kubernetes/kubernetes/issues/18868