我的书签
添加书签
移除书签Run etcd clusters as a Kubernetes StatefulSet
Running etcd as a Kubernetes StatefulSet
Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.
Example Manifest
This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.
If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.
1. $ kubectl apply --filename etcd.yaml
Upon being applied, wait for the pods to become ready.
1. $ kubectl get pods
2. NAME READY STATUS RESTARTS AGE
3. etcd-0 1/1 Running 0 24m
4. etcd-1 1/1 Running 0 24m
5. etcd-2 1/1 Running 0 24m
The container used in the example includes etcdctl and can be called directly inside the pods.
1. $ kubectl exec -it etcd-0 -- etcdctl member list -wtable
2. +------------------+---------+--------+-------------------------+-------------------------+------------+
3. | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
4. +------------------+---------+--------+-------------------------+-------------------------+------------+
5. | 4f98c3545405a0b0 | started | etcd-2 | http://etcd-2.etcd:2380 | http://etcd-2.etcd:2379 | false |
6. | a394e0ee91773643 | started | etcd-0 | http://etcd-0.etcd:2380 | http://etcd-0.etcd:2379 | false |
7. | d10297b8d2f01265 | started | etcd-1 | http://etcd-1.etcd:2380 | http://etcd-1.etcd:2379 | false |
8. +------------------+---------+--------+-------------------------+-------------------------+------------+
To deploy with a self-signed certificate, refer to the commented configuration headings starting with ## TLS to find values that you can uncomment. Additional instructions for generating a cert with cert-manager is included in a section below.
1. # file: etcd.yaml
2. ---
3. apiVersion: v1
4. kind: Service
5. metadata:
6. name: etcd
7. namespace: default
8. spec:
9. type: ClusterIP
10. clusterIP: None
11. selector:
12. app: etcd
13. ##
14. ## Ideally we would use SRV records to do peer discovery for initialization.
15. ## Unfortunately discovery will not work without logic to wait for these to
16. ## populate in the container. This problem is relatively easy to overcome by
17. ## making changes to prevent the etcd process from starting until the records
18. ## have populated. The documentation on statefulsets briefly talk about it.
19. ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id
20. publishNotReadyAddresses: true
21. ##
22. ## The naming scheme of the client and server ports match the scheme that etcd
23. ## uses when doing discovery with SRV records.
24. ports:
25. - name: etcd-client
26. port: 2379
27. - name: etcd-server
28. port: 2380
29. - name: etcd-metrics
30. port: 8080
31. ---
32. apiVersion: apps/v1
33. kind: StatefulSet
34. metadata:
35. namespace: default
36. name: etcd
37. spec:
38. ##
39. ## The service name is being set to leverage the service headlessly.
40. ## https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
41. serviceName: etcd
42. ##
43. ## If you are increasing the replica count of an existing cluster, you should
44. ## also update the --initial-cluster-state flag as noted further down in the
45. ## container configuration.
46. replicas: 3
47. ##
48. ## For initialization, the etcd pods must be available to eachother before
49. ## they are "ready" for traffic. The "Parallel" policy makes this possible.
50. podManagementPolicy: Parallel
51. ##
52. ## To ensure availability of the etcd cluster, the rolling update strategy
53. ## is used. For availability, there must be at least 51% of the etcd nodes
54. ## online at any given time.
55. updateStrategy:
56. type: RollingUpdate
57. ##
58. ## This is label query over pods that should match the replica count.
59. ## It must match the pod template's labels. For more information, see the
60. ## following documentation:
61. ## https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
62. selector:
63. matchLabels:
64. app: etcd
65. ##
66. ## Pod configuration template.
67. template:
68. metadata:
69. ##
70. ## The labeling here is tied to the "matchLabels" of this StatefulSet and
71. ## "affinity" configuration of the pod that will be created.
72. ##
73. ## This example's labeling scheme is fine for one etcd cluster per
74. ## namespace, but should you desire multiple clusters per namespace, you
75. ## will need to update the labeling schema to be unique per etcd cluster.
76. labels:
77. app: etcd
78. annotations:
79. ##
80. ## This gets referenced in the etcd container's configuration as part of
81. ## the DNS name. It must match the service name created for the etcd
82. ## cluster. The choice to place it in an annotation instead of the env
83. ## settings is because there should only be 1 service per etcd cluster.
84. serviceName: etcd
85. spec:
86. ##
87. ## Configuring the node affinity is necessary to prevent etcd servers from
88. ## ending up on the same hardware together.
89. ##
90. ## See the scheduling documentation for more information about this:
91. ## https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
92. affinity:
93. ## The podAntiAffinity is a set of rules for scheduling that describe
94. ## when NOT to place a pod from this StatefulSet on a node.
95. podAntiAffinity:
96. ##
97. ## When preparing to place the pod on a node, the scheduler will check
98. ## for other pods matching the rules described by the labelSelector
99. ## separated by the chosen topology key.
100. requiredDuringSchedulingIgnoredDuringExecution:
101. ## This label selector is looking for app=etcd
102. - labelSelector:
103. matchExpressions:
104. - key: app
105. operator: In
106. values:
107. - etcd
108. ## This topology key denotes a common label used on nodes in the
109. ## cluster. The podAntiAffinity configuration essentially states
110. ## that if another pod has a label of app=etcd on the node, the
111. ## scheduler should not place another pod on the node.
112. ## https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetesiohostname
113. topologyKey: "kubernetes.io/hostname"
114. ##
115. ## Containers in the pod
116. containers:
117. ## This example only has this etcd container.
118. - name: etcd
119. image: quay.io/coreos/etcd:v3.6.0
120. imagePullPolicy: IfNotPresent
121. ports:
122. - name: etcd-client
123. containerPort: 2379
124. - name: etcd-server
125. containerPort: 2380
126. - name: etcd-metrics
127. containerPort: 8080
128. ##
129. ## These probes will fail over TLS for self-signed certificates, so etcd
130. ## is configured to deliver metrics over port 8080 further down.
131. ##
132. ## As mentioned in the "Monitoring etcd" page, /readyz and /livez were
133. ## added in v3.5.12. Prior to this, monitoring required extra tooling
134. ## inside the container to make these probes work.
135. ##
136. ## The values in this readiness probe should be further validated, it
137. ## is only an example configuration.
138. readinessProbe:
139. httpGet:
140. path: /readyz
141. port: 8080
142. initialDelaySeconds: 10
143. periodSeconds: 5
144. timeoutSeconds: 5
145. successThreshold: 1
146. failureThreshold: 30
147. ## The values in this liveness probe should be further validated, it
148. ## is only an example configuration.
149. livenessProbe:
150. httpGet:
151. path: /livez
152. port: 8080
153. initialDelaySeconds: 15
154. periodSeconds: 10
155. timeoutSeconds: 5
156. failureThreshold: 3
157. env:
158. ##
159. ## Environment variables defined here can be used by other parts of the
160. ## container configuration. They are interpreted by Kubernetes, instead
161. ## of in the container environment.
162. ##
163. ## These env vars pass along information about the pod.
164. - name: K8S_NAMESPACE
165. valueFrom:
166. fieldRef:
167. fieldPath: metadata.namespace
168. - name: HOSTNAME
169. valueFrom:
170. fieldRef:
171. fieldPath: metadata.name
172. - name: SERVICE_NAME
173. valueFrom:
174. fieldRef:
175. fieldPath: metadata.annotations['serviceName']
176. ##
177. ## Configuring etcdctl inside the container to connect to the etcd node
178. ## in the container reduces confusion when debugging.
179. - name: ETCDCTL_ENDPOINTS
180. value: $(HOSTNAME).$(SERVICE_NAME):2379
181. ##
182. ## TLS client configuration for etcdctl in the container.
183. ## These files paths are part of the "etcd-client-certs" volume mount.
184. # - name: ETCDCTL_KEY
185. # value: /etc/etcd/certs/client/tls.key
186. # - name: ETCDCTL_CERT
187. # value: /etc/etcd/certs/client/tls.crt
188. # - name: ETCDCTL_CACERT
189. # value: /etc/etcd/certs/client/ca.crt
190. ##
191. ## Use this URI_SCHEME value for non-TLS clusters.
192. - name: URI_SCHEME
193. value: "http"
194. ## TLS: Use this URI_SCHEME for TLS clusters.
195. # - name: URI_SCHEME
196. # value: "https"
197. ##
198. ## If you're using a different container, the executable may be in a
199. ## different location. This example uses the full path to help remove
200. ## ambiguity to you, the reader.
201. ## Often you can just use "etcd" instead of "/usr/local/bin/etcd" and it
202. ## will work because the $PATH includes a directory containing "etcd".
203. command:
204. - /usr/local/bin/etcd
205. ##
206. ## Arguments used with the etcd command inside the container.
207. args:
208. ##
209. ## Configure the name of the etcd server.
210. - --name=$(HOSTNAME)
211. ##
212. ## Configure etcd to use the persistent storage configured below.
213. - --data-dir=/data
214. ##
215. ## In this example we're consolidating the WAL into sharing space with
216. ## the data directory. This is not ideal in production environments and
217. ## should be placed in it's own volume.
218. - --wal-dir=/data/wal
219. ##
220. ## URL configurations are parameterized here and you shouldn't need to
221. ## do anything with these.
222. - --listen-peer-urls=$(URI_SCHEME)://0.0.0.0:2380
223. - --listen-client-urls=$(URI_SCHEME)://0.0.0.0:2379
224. - --advertise-client-urls=$(URI_SCHEME)://$(HOSTNAME).$(SERVICE_NAME):2379
225. ##
226. ## This must be set to "new" for initial cluster bootstrapping. To scale
227. ## the cluster up, this should be changed to "existing" when the replica
228. ## count is increased. If set incorrectly, etcd makes an attempt to
229. ## start but fail safely.
230. - --initial-cluster-state=new
231. ##
232. ## Token used for cluster initialization. The recommendation for this is
233. ## to use a unique token for every cluster. This example parameterized
234. ## to be unique to the namespace, but if you are deploying multiple etcd
235. ## clusters in the same namespace, you should do something extra to
236. ## ensure uniqueness amongst clusters.
237. - --initial-cluster-token=etcd-$(K8S_NAMESPACE)
238. ##
239. ## The initial cluster flag needs to be updated to match the number of
240. ## replicas configured. When combined, these are a little hard to read.
241. ## Here is what a single parameterized peer looks like:
242. ## etcd-0=$(URI_SCHEME)://etcd-0.$(SERVICE_NAME):2380
243. - --initial-cluster=etcd-0=$(URI_SCHEME)://etcd-0.$(SERVICE_NAME):2380,etcd-1=$(URI_SCHEME)://etcd-1.$(SERVICE_NAME):2380,etcd-2=$(URI_SCHEME)://etcd-2.$(SERVICE_NAME):2380
244. ##
245. ## The peer urls flag should be fine as-is.
246. - --initial-advertise-peer-urls=$(URI_SCHEME)://$(HOSTNAME).$(SERVICE_NAME):2380
247. ##
248. ## This avoids probe failure if you opt to configure TLS.
249. - --listen-metrics-urls=http://0.0.0.0:8080
250. ##
251. ## These are some configurations you may want to consider enabling, but
252. ## should look into further to identify what settings are best for you.
253. # - --auto-compaction-mode=periodic
254. # - --auto-compaction-retention=10m
255. ##
256. ## TLS client configuration for etcd, reusing the etcdctl env vars.
257. # - --client-cert-auth
258. # - --trusted-ca-file=$(ETCDCTL_CACERT)
259. # - --cert-file=$(ETCDCTL_CERT)
260. # - --key-file=$(ETCDCTL_KEY)
261. ##
262. ## TLS server configuration for etcdctl in the container.
263. ## These files paths are part of the "etcd-server-certs" volume mount.
264. # - --peer-client-cert-auth
265. # - --peer-trusted-ca-file=/etc/etcd/certs/server/ca.crt
266. # - --peer-cert-file=/etc/etcd/certs/server/tls.crt
267. # - --peer-key-file=/etc/etcd/certs/server/tls.key
268. ##
269. ## This is the mount configuration.
270. volumeMounts:
271. - name: etcd-data
272. mountPath: /data
273. ##
274. ## TLS client configuration for etcdctl
275. # - name: etcd-client-tls
276. # mountPath: "/etc/etcd/certs/client"
277. # readOnly: true
278. ##
279. ## TLS server configuration
280. # - name: etcd-server-tls
281. # mountPath: "/etc/etcd/certs/server"
282. # readOnly: true
283. volumes:
284. ##
285. ## TLS client configuration
286. # - name: etcd-client-tls
287. # secret:
288. # secretName: etcd-client-tls
289. # optional: false
290. ##
291. ## TLS server configuration
292. # - name: etcd-server-tls
293. # secret:
294. # secretName: etcd-server-tls
295. # optional: false
296. ##
297. ## This StatefulSet will uses the volumeClaimTemplate field to create a PVC in
298. ## the cluster for each replica. These PVCs can not be easily resized later.
299. volumeClaimTemplates:
300. - metadata:
301. name: etcd-data
302. spec:
303. accessModes: ["ReadWriteOnce"]
304. ##
305. ## In some clusters, it is necessary to explicitly set the storage class.
306. ## This example will end up using the default storage class.
307. # storageClassName: ""
308. resources:
309. requests:
310. storage: 1Gi
Generating Certificates
In this section, we use Helm to install an operator called cert-manager.
With cert-manager installed in the cluster, self-signed certificates can be generated in the cluster. These generated certificates get placed inside a secret object that can be attached as files in containers.
This is the helm command to install cert-manager.
1. $ helm upgrade --install --create-namespace --namespace cert-manager cert-manager cert-manager --repo https://charts.jetstack.io --set crds.enabled=true
This is an example ClusterIssuer configuration for generating self-signed certificates.
1. # file: issuer.yaml
2. apiVersion: cert-manager.io/v1
3. kind: ClusterIssuer
4. metadata:
5. name: selfsigned
6. spec:
7. selfSigned: {}
This manifest creates Certificate objects for the client and server certs, referencing the ClusterIssuer “selfsigned”. The dnsNames should be an exhaustive list of valid hostnames for the certificates that cert-manager creates.
1. # file: certificates.yaml
2. ---
3. apiVersion: cert-manager.io/v1
4. kind: Certificate
5. metadata:
6. name: etcd-server
7. namespace: default
8. spec:
9. secretName: etcd-server-tls
10. issuerRef:
11. name: selfsigned
12. kind: ClusterIssuer
13. commonName: etcd
14. dnsNames:
15. - etcd
16. - etcd.default
17. - etcd.default.svc.cluster.local
18. - etcd-0
19. - etcd-0.etcd
20. - etcd-0.etcd.default
21. - etcd-0.etcd.default.svc
22. - etcd-0.etcd.default.svc.cluster.local
23. - etcd-1
24. - etcd-1.etcd
25. - etcd-1.etcd.default
26. - etcd-1.etcd.default.svc
27. - etcd-1.etcd.default.svc.cluster.local
28. - etcd-2
29. - etcd-2.etcd
30. - etcd-2.etcd.default
31. - etcd-2.etcd.default.svc
32. - etcd-2.etcd.default.svc.cluster.local
33. ---
34. apiVersion: cert-manager.io/v1
35. kind: Certificate
36. metadata:
37. name: etcd-client
38. namespace: default
39. spec:
40. secretName: etcd-client-tls
41. issuerRef:
42. name: selfsigned
43. kind: ClusterIssuer
44. commonName: etcd
45. dnsNames:
46. - etcd
47. - etcd.default
48. - etcd.default.svc.cluster.local
49. - etcd-0
50. - etcd-0.etcd
51. - etcd-0.etcd.default
52. - etcd-0.etcd.default.svc
53. - etcd-0.etcd.default.svc.cluster.local
54. - etcd-1
55. - etcd-1.etcd
56. - etcd-1.etcd.default
57. - etcd-1.etcd.default.svc
58. - etcd-1.etcd.default.svc.cluster.local
59. - etcd-2
60. - etcd-2.etcd
61. - etcd-2.etcd.default
62. - etcd-2.etcd.default.svc
63. - etcd-2.etcd.default.svc.cluster.local
Last modified January 16, 2025: Remove refs to
latestcontainer tags (9f41e14)
