无法在etcd3中findKubernetes apiserver的数据

我使用kargo脚本来构build我的集群。 我无法find数据存储在etcd3中的位置，尽pipe查看了apiserver的详细日志。

以下是我看到的hyperkube apiserver日志：

 $ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e [[ SNIP ]] I0127 23:31:55.871267 1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} I0127 23:31:55.875975 1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} I0127 23:31:55.876169 1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215 I0127 23:31:55.877950 1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] I0127 23:31:55.878148 1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>} I0127 23:31:55.879372 1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] 

hyperkube apiserver是以这些参数开始的：

 $ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366 [ { "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b", "Created": "2017-01-27T23:35:10.691147667Z", "Path": "/hyperkube", "Args": [ "apiserver", "--advertise-address=10.60.68.23", "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379", "--etcd-quorum-read=true", "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem", "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem", "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem", "--insecure-bind-address=127.0.0.1", "--apiserver-count=3", "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota", "--service-cluster-ip-range=10.233.0.0/18", "--service-node-port-range=30000-32767", "--client-ca-file=/etc/kubernetes/ssl/ca.pem", "--basic-auth-file=/etc/kubernetes/users/known_users.csv", "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem", "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem", "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv", "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem", "--secure-port=443", "--insecure-port=8080", "--v=4", "--allow-privileged=true", "--cloud-provider=openstack", "--cloud-config=/etc/kubernetes/cloud_config", "--anonymous-auth=False" ], 

它没有覆盖/registry的默认etcd前缀。 我不知道apiserver在哪里存储数据。

 $ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry Error: 100: Key not found (/registry) [163] 

Answer 1

如果您的系统设置正确，请注意有一些从etcd2到etcd3的更改。 例如etcd3中的关键空间现在是平坦的，所以与etcd2不同，不再有任何目录。 如上所示，不要使用etcdctl'ls'命令，而应使用“get”命令（并确保etcdctl命令通过设置ETCDCTL_API = 3环境variables来使用v3 api）。

Answer 2

要通过kubernetes获取存储在etcd v3中的密钥和值：

 ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry 

从etcd v3获得1个指定的密钥和值，例如：

 ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes 

基于： https ： //github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md

Answer 3

 ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get / --prefix --keys-only