我的书签
添加书签
移除书签Health check and resource monitoring
In the MatrixOne distributed cluster, there are multiple components and objects. To ensure its regular operation and troubleshooting, we must perform a series of health checks and resource monitoring.
The health check and resource monitoring environment introduced in this document will be based on the MatrixOne Distributed Cluster Deployment environment.
The upgraded environment introduced in this document will be based on the environment of MatrixOne Distributed Cluster Deployment.
Objects to check
- Physical resource layer: including the three virtual machines’ CPU, memory, and disk resources. For a mature solution to monitor these resources, see Monitoring Solution.
- Logical resource layer: including the capacity usage of MinIO, the CPU and memory resource usage of each node and Pod of Kubernetes, the overall status of MatrixOne, and the status of each component (such as LogService, CN, DN).
Resource monitoring
MinIO capacity usage monitoring
MinIO has a management interface through which we can visually monitor its capacity usage, including the remaining space. For more information, see official documentation.
Node/Pod resource monitoring
To determine whether the MatrixOne service needs to be scaled up or down, users often need to monitor the resources used by the Node where the MatrixOne cluster resides and the pods corresponding to the components.
You can use the kubectl top command to complete it. For more information, see Kubernetes official website document of
Node Monitoring
- Use the following command to view the details of MatrixOne cluster nodes:
1. kubectl get node
1. [root@master0 ~]# kubectl get node
2. NAME STATUS ROLES AGE VERSION
3. master0 Ready control-plane,master 22h v1.23.17
4. node0 Ready <none> 22h v1.23.17
2. According to the above-returned results, use the following command to view the resource usage of a specific node. According to the previous deployment scheme, it can be seen that the MatrixOne cluster is located on the node named node0:
1. NODE="[node to be monitored]" # According to the above results, it may be IP, hostname, or alias, such as 10.0.0.1, host-10-0-0-1, node01
2. kubectl top node ${NODE}
1. [root@master0 ~]# kubectl top node
2. NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
3. master0 179m 9% 4632Mi 66%
4. node0 292m 15% 4115Mi 56%
5. [root@master0 ~]# kubectl top node node0
6. NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
7. node0 299m 15% 4079Mi 56%
3. You can also view the node’s resource allocation and upper limit. Note that allocated resources are not equal to used resources.
1. [root@master0 ~]# kubectl describe node node0
2. Name: master0
3. Roles: control-plane,master
4. Labels: beta.kubernetes.io/arch=amd64
5. beta.kubernetes.io/os=linux
6. kubernetes.io/arch=amd64
7. kubernetes.io/hostname=master0
8. kubernetes.io/os=linux
9. node-role.kubernetes.io/control-plane=
10. node-role.kubernetes.io/master=
11. node.kubernetes.io/exclude-from-external-load-balancers=
12. Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
13. node.alpha.kubernetes.io/ttl: 0
14. projectcalico.org/IPv4Address: 10.206.134.8/24
15. projectcalico.org/IPv4VXLANTunnelAddr: 10.234.166.0
16. volumes.kubernetes.io/controller-managed-attach-detach: true
17. CreationTimestamp: Sun, 07 May 2023 12:28:57 +0800
18. Taints: node-role.kubernetes.io/master:NoSchedule
19. Unschedulable: false
20. Lease:
21. HolderIdentity: master0
22. AcquireTime: <unset>
23. RenewTime: Mon, 08 May 2023 10:56:08 +0800
24. Conditions:
25. Type Status LastHeartbeatTime LastTransitionTime Reason Message
26. ---- ------ ----------------- ------------------ ------ -------
27. NetworkUnavailable False Sun, 07 May 2023 12:30:08 +0800 Sun, 07 May 2023 12:30:08 +0800 CalicoIsUp Calico is running on this node
28. MemoryPressure False Mon, 08 May 2023 10:56:07 +0800 Sun, 07 May 2023 12:28:55 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
29. DiskPressure False Mon, 08 May 2023 10:56:07 +0800 Sun, 07 May 2023 12:28:55 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
30. PIDPressure False Mon, 08 May 2023 10:56:07 +0800 Sun, 07 May 2023 12:28:55 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
31. Ready True Mon, 08 May 2023 10:56:07 +0800 Sun, 07 May 2023 20:47:39 +0800 KubeletReady kubelet is posting ready status
32. Addresses:
33. InternalIP: 10.206.134.8
34. Hostname: master0
35. Capacity:
36. cpu: 2
37. ephemeral-storage: 51473868Ki
38. hugepages-1Gi: 0
39. hugepages-2Mi: 0
40. memory: 7782436Ki
41. pods: 110
42. Allocatable:
43. cpu: 1800m
44. ephemeral-storage: 47438316671
45. hugepages-1Gi: 0
46. hugepages-2Mi: 0
47. memory: 7155748Ki
48. pods: 110
49. System Info:
50. Machine ID: fb436be013b5415799d27abf653585d3
51. System UUID: FB436BE0-13B5-4157-99D2-7ABF653585D3
52. Boot ID: 552bd576-56c8-4d22-9549-d950069a5a77
53. Kernel Version: 3.10.0-1160.88.1.el7.x86_64
54. OS Image: CentOS Linux 7 (Core)
55. Operating System: linux
56. Architecture: amd64
57. Container Runtime Version: docker://20.10.23
58. Kubelet Version: v1.23.17
59. Kube-Proxy Version: v1.23.17
60. PodCIDR: 10.234.0.0/23
61. PodCIDRs: 10.234.0.0/23
62. Non-terminated Pods: (12 in total)
63. Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
64. --------- ---- ------------ ---------- --------------- ------------- ---
65. default netchecker-agent-7xnwb 15m (0%) 30m (1%) 64M (0%) 100M (1%) 22h
66. default netchecker-agent-hostnet-bw85f 15m (0%) 30m (1%) 64M (0%) 100M (1%) 22h
67. kruise-system kruise-daemon-xvl8t 0 (0%) 50m (2%) 0 (0%) 128Mi (1%) 20h
68. kube-system calico-node-sbzfc 150m (8%) 300m (16%) 64M (0%) 500M (6%) 22h
69. kube-system dns-autoscaler-7874cf6bcf-l55q4 20m (1%) 0 (0%) 10Mi (0%) 0 (0%) 22h
70. kube-system kube-apiserver-master0 250m (13%) 0 (0%) 0 (0%) 0 (0%) 22h
71. kube-system kube-controller-manager-master0 200m (11%) 0 (0%) 0 (0%) 0 (0%) 22h
72. kube-system kube-proxy-lfkhk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22h
73. kube-system kube-scheduler-master0 100m (5%) 0 (0%) 0 (0%) 0 (0%) 22h
74. kube-system metrics-server-7bd47f88c4-knh9b 100m (5%) 100m (5%) 200Mi (2%) 200Mi (2%) 22h
75. kube-system nodelocaldns-dcffl 100m (5%) 0 (0%) 70Mi (1%) 170Mi (2%) 14h
76. kuboard kuboard-v3-master0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22h
77. Allocated resources:
78. (Total limits may be over 100 percent, i.e., overcommitted.)
79. Resource Requests Limits
80. -------- -------- ------
81. cpu 950m (52%) 510m (28%)
82. memory 485601280 (6%) 1222190848 (16%)
83. ephemeral-storage 0 (0%) 0 (0%)
84. hugepages-1Gi 0 (0%) 0 (0%)
85. hugepages-2Mi 0 (0%) 0 (0%)
86. Events: <none>
89. Name: node0
90. Roles: <none>
91. Labels: beta.kubernetes.io/arch=amd64
92. beta.kubernetes.io/os=linux
93. kubernetes.io/arch=amd64
94. kubernetes.io/hostname=node0
95. kubernetes.io/os=linux
96. Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
97. node.alpha.kubernetes.io/ttl: 0
98. projectcalico.org/IPv4Address: 10.206.134.14/24
99. projectcalico.org/IPv4VXLANTunnelAddr: 10.234.60.0
100. volumes.kubernetes.io/controller-managed-attach-detach: true
101. CreationTimestamp: Sun, 07 May 2023 12:29:46 +0800
102. Taints: <none>
103. Unschedulable: false
104. Lease:
105. HolderIdentity: node0
106. AcquireTime: <unset>
107. RenewTime: Mon, 08 May 2023 10:56:06 +0800
108. Conditions:
109. Type Status LastHeartbeatTime LastTransitionTime Reason Message
110. ---- ------ ----------------- ------------------ ------ -------
111. NetworkUnavailable False Sun, 07 May 2023 12:30:08 +0800 Sun, 07 May 2023 12:30:08 +0800 CalicoIsUp Calico is running on this node
112. MemoryPressure False Mon, 08 May 2023 10:56:12 +0800 Sun, 07 May 2023 12:29:46 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
113. DiskPressure False Mon, 08 May 2023 10:56:12 +0800 Sun, 07 May 2023 12:29:46 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
114. PIDPressure False Mon, 08 May 2023 10:56:12 +0800 Sun, 07 May 2023 12:29:46 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
115. Ready True Mon, 08 May 2023 10:56:12 +0800 Sun, 07 May 2023 20:48:36 +0800 KubeletReady kubelet is posting ready status
116. Addresses:
117. InternalIP: 10.206.134.14
118. Hostname: node0
119. Capacity:
120. cpu: 2
121. ephemeral-storage: 51473868Ki
122. hugepages-1Gi: 0
123. hugepages-2Mi: 0
124. memory: 7782444Ki
125. pods: 110
126. Allocatable:
127. cpu: 1900m
128. ephemeral-storage: 47438316671
129. hugepages-1Gi: 0
130. hugepages-2Mi: 0
131. memory: 7417900Ki
132. pods: 110
133. System Info:
134. Machine ID: a6600151884b44fb9f0bc9af490e44b7
135. System UUID: A6600151-884B-44FB-9F0B-C9AF490E44B7
136. Boot ID: b7f3357f-44e6-425e-8c90-6ada14e92703
137. Kernel Version: 3.10.0-1160.88.1.el7.x86_64
138. OS Image: CentOS Linux 7 (Core)
139. Operating System: linux
140. Architecture: amd64
141. Container Runtime Version: docker://20.10.23
142. Kubelet Version: v1.23.17
143. Kube-Proxy Version: v1.23.17
144. PodCIDR: 10.234.2.0/23
145. PodCIDRs: 10.234.2.0/23
146. Non-terminated Pods: (20 in total)
147. Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
148. --------- ---- ------------ ---------- --------------- ------------- ---
149. default netchecker-agent-6v8rl 15m (0%) 30m (1%) 64M (0%) 100M (1%) 22h
150. default netchecker-agent-hostnet-fb2jn 15m (0%) 30m (1%) 64M (0%) 100M (1%) 22h
151. default netchecker-server-645d759b79-v4bqm 150m (7%) 300m (15%) 192M (2%) 512M (6%) 22h
152. kruise-system kruise-controller-manager-74847d59cf-295rk 100m (5%) 200m (10%) 256Mi (3%) 512Mi (7%) 20h
153. kruise-system kruise-controller-manager-74847d59cf-854sq 100m (5%) 200m (10%) 256Mi (3%) 512Mi (7%) 20h
154. kruise-system kruise-daemon-rz9pj 0 (0%) 50m (2%) 0 (0%) 128Mi (1%) 20h
155. kube-system calico-kube-controllers-74df5cd99c-n9qsn 30m (1%) 1 (52%) 64M (0%) 256M (3%) 22h
156. kube-system calico-node-brqrk 150m (7%) 300m (15%) 64M (0%) 500M (6%) 22h
157. kube-system coredns-76b4fb4578-9cqc7 100m (5%) 0 (0%) 70Mi (0%) 170Mi (2%) 14h
158. kube-system kube-proxy-rpxb5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22h
159. kube-system nginx-proxy-node0 25m (1%) 0 (0%) 32M (0%) 0 (0%) 22h
160. kube-system nodelocaldns-qkxhv 100m (5%) 0 (0%) 70Mi (0%) 170Mi (2%) 14h
161. local-path-storage local-path-storage-local-path-provisioner-d5bb7f8c9-qfp8h 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h
162. mo-hn matrixone-operator-f8496ff5c-fp6zm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h
163. mo-hn mo-dn-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13h
164. mo-hn mo-log-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13h
165. mo-hn mo-log-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13h
166. mo-hn mo-log-2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13h
167. mo-hn mo-tp-cn-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13h
168. mostorage minio-674ccf54f7-tdglh 0 (0%) 0 (0%) 512Mi (7%) 0 (0%) 20h
169. Allocated resources:
170. (Total limits may be over 100 percent, i.e., overcommitted.)
171. Resource Requests Limits
172. -------- -------- ------
173. cpu 785m (41%) 2110m (111%)
174. memory 1700542464 (22%) 3032475392 (39%)
175. ephemeral-storage 0 (0%) 0 (0%)
176. hugepages-1Gi 0 (0%) 0 (0%)
177. hugepages-2Mi 0 (0%) 0 (0%)
178. Events: <none>
Pod Monitoring
- Use the following command to view the Pods of the MatrixOne cluster:
1. NS="mo-hn"
2. kubectl get pod -n ${NS}
2. According to the above-returned results, use the following command to view the resource usage of a specific Pod:
1. POD="[pod name to be monitored]" # According to the above results, for example: dn is mo-dn-0, cn is mo-tp-cn-0, mo-tp-cn-1, ..., logservice is mo -log-0, mo-log-1, ...
2. kubectl top pod ${POD} -n ${NS}
The command will display the CPU and memory usage for the specified Pod, similar to the following output:
1. [root@master0 ~]# kubectl top pod mo-tp-cn-0 -nmo-hn
2. NAME CPU(cores) MEMORY(bytes)
3. mo-tp-cn-0 20m 214Mi
4. [root@master0 ~]# kubectl top pod mo-dn-0 -nmo-hn
5. NAME CPU(cores) MEMORY(bytes)
6. mo-dn-0 36m 161Mi
3. you can also view the resource declaration of a specific Pod to compare it with the actual resource usage.
1. kubectl describe pod ${POD_NAME} -n${NS}
2. kubectl get pod ${POD_NAME} -n${NS} -oyaml
1. [root@master0 ~]# kubectl describe pod mo-tp-cn-0 -nmo-hn
2. Name: mo-tp-cn-0
3. Namespace: mo-hn
4. Priority: 0
5. Node: node0/10.206.134.14
6. Start Time: Sun, 07 May 2023 21:01:50 +0800
7. Labels: controller-revision-hash=mo-tp-cn-8666cdfb56
8. lifecycle.apps.kruise.io/state=Normal
9. matrixorigin.io/cluster=mo
10. matrixorigin.io/component=CNSet
11. matrixorigin.io/instance=mo-tp
12. matrixorigin.io/namespace=mo-hn
13. statefulset.kubernetes.io/pod-name=mo-tp-cn-0
14. Annotations: apps.kruise.io/runtime-containers-meta:
15. {"containers":[{"name":"main","containerID":"docker://679d672a330d7318f97a90835dacefcdd03e8a08062b8844d438f8cdd6bcdc8f","restartCount":0,"...
16. cni.projectcalico.org/containerID: 80b286789a2d6fa9e615c3edee79b57edb452eaeafddb9b7b82ec5fb2e339409
17. cni.projectcalico.org/podIP: 10.234.60.53/32
18. cni.projectcalico.org/podIPs: 10.234.60.53/32
19. kruise.io/related-pub: mo
20. lifecycle.apps.kruise.io/timestamp: 2023-05-07T13:01:50Z
21. matrixone.cloud/cn-label: null
22. matrixone.cloud/dns-based-identity: False
23. Status: Running
24. IP: 10.234.60.53
25. IPs:
26. IP: 10.234.60.53
27. Controlled By: StatefulSet/mo-tp-cn
28. Containers:
29. main:
30. Container ID: docker://679d672a330d7318f97a90835dacefcdd03e8a08062b8844d438f8cdd6bcdc8f
31. Image: matrixorigin/matrixone:nightly-144f3be4
32. Image ID: docker-pullable://matrixorigin/matrixone@sha256:288fe3d626c6aa564684099e4686a9d4b28e16fdd16512bd968a67bb41d5aaa3
33. Port: <none>
34. Host Port: <none>
35. Command:
36. /bin/sh
37. /etc/matrixone/config/start.sh
38. Args:
39. -debug-http=:6060
40. State: Running
41. Started: Sun, 07 May 2023 21:01:54 +0800
42. Ready: True
43. Restart Count: 0
44. Environment:
45. POD_NAME: mo-tp-cn-0 (v1:metadata.name)
46. NAMESPACE: mo-hn (v1:metadata.namespace)
47. HEADLESS_SERVICE_NAME: mo-tp-cn-headless
48. AWS_ACCESS_KEY_ID: <set to the key 'AWS_ACCESS_KEY_ID' in secret 'minio'> Optional: false
49. AWS_SECRET_ACCESS_KEY: <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'minio'> Optional: false
50. AWS_REGION: us-west-2
51. Mounts:
52. /etc/matrixone/config from config (ro)
53. /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ngpcs (ro)
54. Readiness Gates:
55. Type Status
56. InPlaceUpdateReady True
57. KruisePodReady True
58. Conditions:
59. Type Status
60. KruisePodReady True
61. InPlaceUpdateReady True
62. Initialized True
63. Ready True
64. ContainersReady True
65. PodScheduled True
66. Volumes:
67. config:
68. Type: ConfigMap (a volume populated by a ConfigMap)
69. Name: mo-tp-cn-config-5abf454
70. Optional: false
71. kube-api-access-ngpcs:
72. Type: Projected (a volume that contains injected data from multiple sources)
73. TokenExpirationSeconds: 3607
74. ConfigMapName: kube-root-ca.crt
75. ConfigMapOptional: <nil>
76. DownwardAPI: true
77. QoS Class: BestEffort
78. Node-Selectors: <none>
79. Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
80. node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
81. Events: <none>
82. [root@master0 ~]# kubectl get pod mo-tp-cn-0 -nmo-hn -oyaml
83. apiVersion: v1
84. kind: Pod
85. metadata:
86. annotations:
87. apps.kruise.io/runtime-containers-meta: '{"containers":[{"name":"main","containerID":"docker://679d672a330d7318f97a90835dacefcdd03e8a08062b8844d438f8cdd6bcdc8f","restartCount":0,"hashes":{"plainHash":1670287891}}]}'
88. cni.projectcalico.org/containerID: 80b286789a2d6fa9e615c3edee79b57edb452eaeafddb9b7b82ec5fb2e339409
89. cni.projectcalico.org/podIP: 10.234.60.53/32
90. cni.projectcalico.org/podIPs: 10.234.60.53/32
91. kruise.io/related-pub: mo
92. lifecycle.apps.kruise.io/timestamp: "2023-05-07T13:01:50Z"
93. matrixone.cloud/cn-label: "null"
94. matrixone.cloud/dns-based-identity: "False"
95. creationTimestamp: "2023-05-07T13:01:50Z"
96. generateName: mo-tp-cn-
97. labels:
98. controller-revision-hash: mo-tp-cn-8666cdfb56
99. lifecycle.apps.kruise.io/state: Normal
100. matrixorigin.io/cluster: mo
101. matrixorigin.io/component: CNSet
102. matrixorigin.io/instance: mo-tp
103. matrixorigin.io/namespace: mo-hn
104. statefulset.kubernetes.io/pod-name: mo-tp-cn-0
105. name: mo-tp-cn-0
106. namespace: mo-hn
107. ownerReferences:
108. - apiVersion: apps.kruise.io/v1beta1
109. blockOwnerDeletion: true
110. controller: true
111. kind: StatefulSet
112. name: mo-tp-cn
113. uid: 891e0453-89a5-45d5-ad12-16ef048c804f
114. resourceVersion: "72625"
115. uid: 1e3e2df3-f1c2-4444-8694-8d23e7125d35
116. spec:
117. containers:
118. - args:
119. - -debug-http=:6060
120. command:
121. - /bin/sh
122. - /etc/matrixone/config/start.sh
123. env:
124. - name: POD_NAME
125. valueFrom:
126. fieldRef:
127. apiVersion: v1
128. fieldPath: metadata.name
129. - name: NAMESPACE
130. valueFrom:
131. fieldRef:
132. apiVersion: v1
133. fieldPath: metadata.namespace
134. - name: HEADLESS_SERVICE_NAME
135. value: mo-tp-cn-headless
136. - name: AWS_ACCESS_KEY_ID
137. valueFrom:
138. secretKeyRef:
139. key: AWS_ACCESS_KEY_ID
140. name: minio
141. - name: AWS_SECRET_ACCESS_KEY
142. valueFrom:
143. secretKeyRef:
144. key: AWS_SECRET_ACCESS_KEY
145. name: minio
146. - name: AWS_REGION
147. value: us-west-2
148. image: matrixorigin/matrixone:nightly-144f3be4
149. imagePullPolicy: Always
150. name: main
151. resources: {}
152. terminationMessagePath: /dev/termination-log
153. terminationMessagePolicy: File
154. volumeMounts:
155. - mountPath: /etc/matrixone/config
156. name: config
157. readOnly: true
158. - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
159. name: kube-api-access-ngpcs
160. readOnly: true
161. dnsPolicy: ClusterFirst
162. enableServiceLinks: true
163. hostname: mo-tp-cn-0
164. nodeName: node0
165. preemptionPolicy: PreemptLowerPriority
166. priority: 0
167. readinessGates:
168. - conditionType: InPlaceUpdateReady
169. - conditionType: KruisePodReady
170. restartPolicy: Always
171. schedulerName: default-scheduler
172. securityContext: {}
173. serviceAccount: default
174. serviceAccountName: default
175. subdomain: mo-tp-cn-headless
176. terminationGracePeriodSeconds: 30
177. tolerations:
178. - effect: NoExecute
179. key: node.kubernetes.io/not-ready
180. operator: Exists
181. tolerationSeconds: 300
182. - effect: NoExecute
183. key: node.kubernetes.io/unreachable
184. operator: Exists
185. tolerationSeconds: 300
186. volumes:
187. - configMap:
188. defaultMode: 420
189. name: mo-tp-cn-config-5abf454
190. name: config
191. - name: kube-api-access-ngpcs
192. projected:
193. defaultMode: 420
194. sources:
195. - serviceAccountToken:
196. expirationSeconds: 3607
197. path: token
198. - configMap:
199. items:
200. - key: ca.crt
201. path: ca.crt
202. name: kube-root-ca.crt
203. - downwardAPI:
204. items:
205. - fieldRef:
206. apiVersion: v1
207. fieldPath: metadata.namespace
208. path: namespace
209. status:
210. conditions:
211. - lastProbeTime: null
212. lastTransitionTime: "2023-05-07T13:01:50Z"
213. status: "True"
214. type: KruisePodReady
215. - lastProbeTime: null
216. lastTransitionTime: "2023-05-07T13:01:50Z"
217. status: "True"
218. type: InPlaceUpdateReady
219. - lastProbeTime: null
220. lastTransitionTime: "2023-05-07T13:01:50Z"
221. status: "True"
222. type: Initialized
223. - lastProbeTime: null
224. lastTransitionTime: "2023-05-07T13:01:54Z"
225. status: "True"
226. type: Ready
227. - lastProbeTime: null
228. lastTransitionTime: "2023-05-07T13:01:54Z"
229. status: "True"
230. type: ContainersReady
231. - lastProbeTime: null
232. lastTransitionTime: "2023-05-07T13:01:50Z"
233. status: "True"
234. type: PodScheduled
235. containerStatuses:
236. - containerID: docker://679d672a330d7318f97a90835dacefcdd03e8a08062b8844d438f8cdd6bcdc8f
237. image: matrixorigin/matrixone:nightly-144f3be4
238. imageID: docker-pullable://matrixorigin/matrixone@sha256:288fe3d626c6aa564684099e4686a9d4b28e16fdd16512bd968a67bb41d5aaa3
239. lastState: {}
240. name: main
241. ready: true
242. restartCount: 0
243. started: true
244. state:
245. running:
246. startedAt: "2023-05-07T13:01:54Z"
247. hostIP: 10.206.134.14
248. phase: Running
249. podIP: 10.234.60.53
250. podIPs:
251. - ip: 10.234.60.53
252. qosClass: BestEffort
253. startTime: "2023-05-07T13:01:50Z"
MatrixOne Monitoring
View cluster status
During Operator deployment, we defined matrixOnecluster as the custom resource name for the entire cluster. By checking MatrixOneCluster, we can tell if the cluster is functioning correctly. You can check with the following command:
1. MO_NAME="mo"
2. NS="mo-hn"
3. kubectl get matrixonecluster -n ${NS} ${MO_NAME}
If the status is “Ready”, the cluster is healthy. If the status is “NotReady”, further troubleshooting is required.
1. [root@master0 ~]# MO_NAME="mo"
2. [root@master0 ~]# NS="mo-hn"
3. [root@master0 ~]# kubectl get matrixonecluster -n${NS} ${MO_NAME}
4. NAME LOG DN TP AP VERSION PHASE AGE
5. mo 3 1 1 nightly-144f3be4 Ready 13h
To view details of the MatrixOne cluster status, you can run the following command:
1. kubectl describe matrixonecluster -n${NS} ${MO_NAME}
1. [root@master0 ~]# kubectl describe matrixonecluster -n${NS} ${MO_NAME}
2. Name: mo
3. Namespace: mo-hn
4. Labels: <none>
5. Annotations: <none>
6. API Version: core.matrixorigin.io/v1alpha1
7. Kind: MatrixOneCluster
8. Metadata:
9. Creation Timestamp: 2023-05-07T12:54:17Z
10. Finalizers:
11. matrixorigin.io/matrixonecluster
12. Generation: 2
13. Managed Fields:
14. API Version: core.matrixorigin.io/v1alpha1
15. Fields Type: FieldsV1
16. fieldsV1:
17. f:metadata:
18. f:annotations:
19. .:
20. f:kubectl.kubernetes.io/last-applied-configuration:
21. f:spec:
22. .:
23. f:dn:
24. .:
25. f:config:
26. f:replicas:
27. f:imagePullPolicy:
28. f:imageRepository:
29. f:logService:
30. .:
31. f:config:
32. f:pvcRetentionPolicy:
33. f:replicas:
34. f:sharedStorage:
35. .:
36. f:s3:
37. .:
38. f:endpoint:
39. f:secretRef:
40. f:type:
41. f:volume:
42. .:
43. f:size:
44. f:tp:
45. .:
46. f:config:
47. f:nodePort:
48. f:replicas:
49. f:serviceType:
50. f:version:
51. Manager: kubectl-client-side-apply
52. Operation: Update
53. Time: 2023-05-07T12:54:17Z
54. API Version: core.matrixorigin.io/v1alpha1
55. Fields Type: FieldsV1
56. fieldsV1:
57. f:metadata:
58. f:finalizers:
59. .:
60. v:"matrixorigin.io/matrixonecluster":
61. Manager: manager
62. Operation: Update
63. Time: 2023-05-07T12:54:17Z
64. API Version: core.matrixorigin.io/v1alpha1
65. Fields Type: FieldsV1
66. fieldsV1:
67. f:spec:
68. f:logService:
69. f:sharedStorage:
70. f:s3:
71. f:path:
72. Manager: kubectl-edit
73. Operation: Update
74. Time: 2023-05-07T13:00:53Z
75. API Version: core.matrixorigin.io/v1alpha1
76. Fields Type: FieldsV1
77. fieldsV1:
78. f:status:
79. .:
80. f:cnGroups:
81. .:
82. f:desiredGroups:
83. f:readyGroups:
84. f:syncedGroups:
85. f:conditions:
86. f:credentialRef:
87. f:dn:
88. .:
89. f:availableStores:
90. f:conditions:
91. f:logService:
92. .:
93. f:availableStores:
94. f:conditions:
95. f:discovery:
96. .:
97. f:address:
98. f:port:
99. f:phase:
100. Manager: manager
101. Operation: Update
102. Subresource: status
103. Time: 2023-05-07T13:02:12Z
104. Resource Version: 72671
105. UID: be2355c0-0c69-4f0f-95bb-9310224200b6
106. Spec:
107. Dn:
108. Config:
109. [dn]
111. [dn.Ckp]
112. flush-interval = "60s"
113. global-interval = "100000s"
114. incremental-interval = "60s"
115. min-count = 100
116. scan-interval = "5s"
118. [dn.Txn]
120. [dn.Txn.Storage]
121. backend = "TAE"
122. log-backend = "logservice"
124. [log]
125. format = "json"
126. level = "error"
127. max-size = 512
129. Replicas: 1
130. Resources:
131. Service Args:
132. -debug-http=:6060
133. Shared Storage Cache:
134. Memory Cache Size: 0
135. Image Pull Policy: Always
136. Image Repository: matrixorigin/matrixone
137. Log Service:
138. Config:
139. [log]
140. format = "json"
141. level = "error"
142. max-size = 512
144. Initial Config:
145. Dn Shards: 1
146. Log Shard Replicas: 3
147. Log Shards: 1
148. Pvc Retention Policy: Retain
149. Replicas: 3
150. Resources:
151. Service Args:
152. -debug-http=:6060
153. Shared Storage:
154. s3:
155. Endpoint: http://minio.mostorage:9000
156. Path: minio-mo
157. s3RetentionPolicy: Retain
158. Secret Ref:
159. Name: minio
160. Type: minio
161. Store Failure Timeout: 10m0s
162. Volume:
163. Size: 1Gi
164. Tp:
165. Config:
166. [cn]
168. [cn.Engine]
169. type = "distributed-tae"
171. [log]
172. format = "json"
173. level = "debug"
174. max-size = 512
176. Node Port: 31474
177. Replicas: 1
178. Resources:
179. Service Args:
180. -debug-http=:6060
181. Service Type: NodePort
182. Shared Storage Cache:
183. Memory Cache Size: 0
184. Version: nightly-144f3be4
185. Status:
186. Cn Groups:
187. Desired Groups: 1
188. Ready Groups: 1
189. Synced Groups: 1
190. Conditions:
191. Last Transition Time: 2023-05-07T13:02:14Z
192. Message: the object is synced
193. Reason: empty
194. Status: True
195. Type: Synced
196. Last Transition Time: 2023-05-07T13:02:14Z
197. Message:
198. Reason: AllSetsReady
199. Status: True
200. Type: Ready
201. Credential Ref:
202. Name: mo-credential
203. Dn:
204. Available Stores:
205. Last Transition: 2023-05-07T13:01:48Z
206. Phase: Up
207. Pod Name: mo-dn-0
208. Conditions:
209. Last Transition Time: 2023-05-07T13:01:48Z
210. Message: the object is synced
211. Reason: empty
212. Status: True
213. Type: Synced
214. Last Transition Time: 2023-05-07T13:01:48Z
215. Message:
216. Reason: empty
217. Status: True
218. Type: Ready
219. Log Service:
220. Available Stores:
221. Last Transition: 2023-05-07T13:01:25Z
222. Phase: Up
223. Pod Name: mo-log-0
224. Last Transition: 2023-05-07T13:01:25Z
225. Phase: Up
226. Pod Name: mo-log-1
227. Last Transition: 2023-05-07T13:01:25Z
228. Phase: Up
229. Pod Name: mo-log-2
230. Conditions:
231. Last Transition Time: 2023-05-07T13:01:25Z
232. Message: the object is synced
233. Reason: empty
234. Status: True
235. Type: Synced
236. Last Transition Time: 2023-05-07T13:01:25Z
237. Message:
238. Reason: empty
239. Status: True
240. Type: Ready
241. Discovery:
242. Address: mo-log-discovery.mo-hn.svc
243. Port: 32001
244. Phase: Ready
245. Events:
246. Type Reason Age From Message
247. ---- ------ ---- ---- -------
248. Normal ReconcileSuccess 29m (x2 over 13h) matrixonecluster object is synced
View component status
The current MatrixOne cluster includes the following components: DN, CN, and Log Service, which correspond to the custom resource types DNSet, CNSet, and LogSet, respectively, and these objects are generated by the MatrixOneCluster controller.
To check whether each component is standard, take DN as an example; you can run the following command:
1. SET_TYPE="dnset"
2. NS="mo-hn"
3. kubectl get ${SET_TYPE} -n ${NS}
This will display status information for the DN component as follows:
1. [root@master0 ~]# SET_TYPE="dnset"
2. [root@master0 ~]# NS="mo-hn"
3. [root@master0 ~]# kubectl get ${SET_TYPE} -n${NS}
4. NAME IMAGE REPLICAS AGE
5. mo matrixorigin/matrixone:nightly-144f3be4 1 13h
6. [root@master0 ~]# SET_TYPE="cnset"
7. [root@master0 ~]# kubectl get ${SET_TYPE} -n${NS}
8. NAME IMAGE REPLICAS AGE
9. mo-tp matrixorigin/matrixone:nightly-144f3be4 1 13h
10. [root@master0 ~]# SET_TYPE="logset"
11. [root@master0 ~]# kubectl get ${SET_TYPE} -n${NS}
12. NAME IMAGE REPLICAS AGE
13. mo matrixorigin/matrixone:nightly-144f3be4 3 13h
View Pod Status
In addition, you can directly check the native k8s objects generated in the MO cluster to confirm the cluster’s health, usually by querying the pod.
1. NS="mo-hn"
2. kubectl get pod -n ${NS}
This will display status information for the Pod.
1. [root@master0 ~]# NS="mo-hn"
2. [root@master0 ~]# kubectl get pod -n${NS}
3. NAME READY STATUS RESTARTS AGE
4. matrixone-operator-f8496ff5c-fp6zm 1/1 Running 0 19h
5. mo-dn-0 1/1 Running 0 13h
6. mo-log-0 1/1 Running 0 13h
7. mo-log-1 1/1 Running 0 13h
8. mo-log-2 1/1 Running 0 13h
9. mo-tp-cn-0 1/1 Running 0 13h
Typically, the Running state indicates that the Pod is functioning normally. But there are also some special cases where the Pod status may be Running, but the MatrixOne cluster is abnormal. For example, connecting to a MatrixOne cluster through a MySQL client is impossible. In this case, you can look further into the Pod’s logs to check for unusual output.
1. NS="mo-hn"
2. POD_NAME="[the name of the pod returned above]" # For example, mo-tp-cn-0
3. kubectl logs ${POD_NAME} -n ${NS}
If the Pod status is not Running, such as Pending, you can check the Pod events (Events) to confirm the cause of the exception. Taking the previous example, because the cluster resources cannot satisfy the request of mo-tp-cn-3, the Pod cannot be scheduled and is Pending. In this case, you can solve it by expanding node resources.
1. kubectl describe pod ${POD_NAME} -n${NS}
