This document describes how to use updates to upgrade an Apache Doris cluster based on a Doris-Operator deployment.

Similar to conventionally deployed cluster upgrades, Doris clusters deployed by Doris-Operator still require rolling upgrades from BE to FE nodes. Doris-Operator is based on Kubernetes’ Performing a Rolling Update provides rolling upgrade capabilities.

Things to note before upgrading

  • It is recommended that the upgrade operation be performed during off-peak periods.
  • During the rolling upgrade process, the connection to the closed node will fail, causing the request to fail. For this type of business, it is recommended to add retry capabilities to the client.
  • Before upgrading, you can read the General Upgrade Manual to help you understand some principles and precautions during the upgrade. .
  • The compatibility of data and metadata cannot be verified before upgrading. Therefore, cluster upgrade must avoid single copy of data and single FE FOLLOWER node in the cluster.
  • Nodes will be restarted during the upgrade process, so unnecessary cluster balancing and replica repair logic may be triggered. Please shut it down first with the following command.
1. admin set frontend config("disable_balance" = "true");
2. admin set frontend config("disable_colocate_balance" = "true");
3. admin set frontend config("disable_tablet_scheduler" = "true");
  • When upgrading Doris, please follow the principle of not upgrading across two or more key node versions. If you want to upgrade across multiple key node versions, upgrade to the latest key node version first, and then upgrade in sequence. If it is a non-key node version, You can ignore skipping. For details, please refer to Upgrade Version Instructions

Upgrade operation

The order of node types in the upgrade process is as follows. If a certain type of node does not exist, it will be skipped:

1. cn/be -> fe -> broker

It is recommended to modify the image of the corresponding cluster components in sequence and then apply the configuration. After the current type of component is fully upgraded and the status returns to normal, the rolling upgrade of the next type of node can be performed.

Upgrade BE

If you retain the cluster’s crd (Doris-Operator defines the abbreviation of DorisCluster type resource name) file, you can upgrade by modifying the configuration file and running the kubectl apply command.

  1. Modify spec.beSpec.image

Change selectdb/doris.be-ubuntu:2.0.4 to selectdb/doris.be-ubuntu:2.1.0

1. $ vim doriscluster-sample.yaml
  1. Save the changes and apply the changes to be upgraded:
1. $ kubectl apply -f doriscluster-sample.yaml -n doris

It can also be modified directly through kubectl edit dcr.

  1. Check the dcr list under namespace ‘doris’ to obtain the cluster_name that needs to be updated.
1. $ kubectl get dcr -n doris
2. NAME                  FESTATUS    BESTATUS    CNSTATUS
3. doriscluster-sample   available   available
  1. Modify, save and take effect
1. $ kubectl edit dcr doriscluster-sample -n doris

1. After entering the text editor, you will find `spec.beSpec.image` and change `selectdb/doris.be-ubuntu:2.0.4` to `selectdb/doris.be-ubuntu:2.1.0`
  1. View the upgrade process and results:
1. $ kubectl get pod -n doris

When all Pods are rebuilt and enter the Running state, the upgrade is complete.

Upgrade FE

If you retain the cluster’s crd (Doris-Operator defines the abbreviation of the DorisCluster type resource name) file, you can upgrade by modifying the configuration file and running the kubectl apply command.

  1. Modify spec.feSpec.image

Change selectdb/doris.fe-ubuntu:2.0.4 to selectdb/doris.fe-ubuntu:2.1.0

1. $ vim doriscluster-sample.yaml
  1. Save the changes and apply the changes to be upgraded:
1. $ kubectl apply -f doriscluster-sample.yaml -n doris

It can also be modified directly through kubectl edit dcr.

  1. Modify, save and take effect
1. $ kubectl edit dcr doriscluster-sample -n doris

1. After entering the text editor, you will find `spec.feSpec.image` and change `selectdb/doris.fe-ubuntu:2.0.4` to `selectdb/doris.fe-ubuntu:2.1.0`
  1. View the upgrade process and results:
1. $ kubectl get pod -n doris

When all Pods are rebuilt and enter the Running state, the upgrade is complete.

After the upgrade is completed

Verify cluster node status

Access Doris through mysql-client through the method provided in the Access Doris Cluster document. Use SQL such as show frontends and show backends to view the version and status of each component.

1. mysql> show frontends\G;
2. *************************** 1. row ***************************
3. Name: fe_13c132aa_3281_4f4f_97e8_655d01287425
4. Host: doriscluster-sample-fe-0.doriscluster-sample-fe-internal.doris.svc.cluster.local
5. EditLogPort: 9010
6. HttpPort: 8030
7. QueryPort: 9030
8. RpcPort: 9020
9. ArrowFlightSqlPort: -1
10. Role: FOLLOWER
11. IsMaster: false
12. ClusterId: 1779160761
13. Join: true
14. Alive: true
15. ReplayedJournalId: 2422
16. LastStartTime: 2024-02-19 06:38:47
17. LastHeartbeat: 2024-02-19 09:31:33
18. IsHelper: true
19. ErrMsg:
20. Version: doris-2.1.0
21. CurrentConnected: Yes
22. *************************** 2. row ***************************
23. Name: fe_f1a9d008_d110_4780_8e60_13d392faa54e
24. Host: doriscluster-sample-fe-2.doriscluster-sample-fe-internal.doris.svc.cluster.local
25. EditLogPort: 9010
26. HttpPort: 8030
27. QueryPort: 9030
28. RpcPort: 9020
29. ArrowFlightSqlPort: -1
30. Role: FOLLOWER
31. IsMaster: true
32. ClusterId: 1779160761
33. Join: true
34. Alive: true
35. ReplayedJournalId: 2423
36. LastStartTime: 2024-02-19 06:37:35
37. LastHeartbeat: 2024-02-19 09:31:33
38. IsHelper: true
39. ErrMsg:
40. Version: doris-2.1.0
41. CurrentConnected: No
42. *************************** 3. row ***************************
43. Name: fe_e42bf9da_006f_4302_b861_770d2c955a47
44. Host: doriscluster-sample-fe-1.doriscluster-sample-fe-internal.doris.svc.cluster.local
45. EditLogPort: 9010
46. HttpPort: 8030
47. QueryPort: 9030
48. RpcPort: 9020
49. ArrowFlightSqlPort: -1
50. Role: FOLLOWER
51. IsMaster: false
52. ClusterId: 1779160761
53. Join: true
54. Alive: true
55. ReplayedJournalId: 2422
56. LastStartTime: 2024-02-19 06:38:17
57. LastHeartbeat: 2024-02-19 09:31:33
58. IsHelper: true
59. ErrMsg:
60. Version: doris-2.1.0
61. CurrentConnected: No
62. 3 rows in set (0.02 sec)

If the Alive status of the FE node is true and the Version value is the new version, the FE node is upgraded successfully.

1. mysql> show backends\G;
2. *************************** 1. row ***************************
3. BackendId: 10002
4. Host: doriscluster-sample-be-0.doriscluster-sample-be-internal.doris.svc.cluster.local
5. HeartbeatPort: 9050
6. BePort: 9060
7. HttpPort: 8040
8. BrpcPort: 8060
9. ArrowFlightSqlPort: -1
10. LastStartTime: 2024-02-19 06:37:56
11. LastHeartbeat: 2024-02-19 09:32:43
12. Alive: true
13. SystemDecommissioned: false
14. TabletNum: 14
15. DataUsedCapacity: 0.000
16. TrashUsedCapcacity: 0.000
17. AvailCapacity: 12.719 GB
18. TotalCapacity: 295.167 GB
19. UsedPct: 95.69 %
20. MaxDiskUsedPct: 95.69 %
21. RemoteUsedCapacity: 0.000
22. Tag: {"location" : "default"}
23. ErrMsg:
24. Version: doris-2.1.0
25. Status: {"lastSuccessReportTabletsTime":"2024-02-19 09:31:48","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
26. HeartbeatFailureCounter: 0
27. NodeRole: mix
28. *************************** 2. row ***************************
29. BackendId: 10003
30. Host: doriscluster-sample-be-1.doriscluster-sample-be-internal.doris.svc.cluster.local
31. HeartbeatPort: 9050
32. BePort: 9060
33. HttpPort: 8040
34. BrpcPort: 8060
35. ArrowFlightSqlPort: -1
36. LastStartTime: 2024-02-19 06:37:35
37. LastHeartbeat: 2024-02-19 09:32:43
38. Alive: true
39. SystemDecommissioned: false
40. TabletNum: 8
41. DataUsedCapacity: 0.000
42. TrashUsedCapcacity: 0.000
43. AvailCapacity: 12.719 GB
44. TotalCapacity: 295.167 GB
45. UsedPct: 95.69 %
46. MaxDiskUsedPct: 95.69 %
47. RemoteUsedCapacity: 0.000
48. Tag: {"location" : "default"}
49. ErrMsg:
50. Version: doris-2.1.0
51. Status: {"lastSuccessReportTabletsTime":"2024-02-19 09:31:43","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
52. HeartbeatFailureCounter: 0
53. NodeRole: mix
54. *************************** 3. row ***************************
55. BackendId: 11024
56. Host: doriscluster-sample-be-2.doriscluster-sample-be-internal.doris.svc.cluster.local
57. HeartbeatPort: 9050
58. BePort: 9060
59. HttpPort: 8040
60. BrpcPort: 8060
61. ArrowFlightSqlPort: -1
62. LastStartTime: 2024-02-19 08:50:36
63. LastHeartbeat: 2024-02-19 09:32:43
64. Alive: true
65. SystemDecommissioned: false
66. TabletNum: 0
67. DataUsedCapacity: 0.000
68. TrashUsedCapcacity: 0.000
69. AvailCapacity: 12.719 GB
70. TotalCapacity: 295.167 GB
71. UsedPct: 95.69 %
72. MaxDiskUsedPct: 95.69 %
73. RemoteUsedCapacity: 0.000
74. Tag: {"location" : "default"}
75. ErrMsg:
76. Version: doris-2.1.0
77. Status: {"lastSuccessReportTabletsTime":"2024-02-19 09:32:04","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
78. HeartbeatFailureCounter: 0
79. NodeRole: mix
80. 3 rows in set (0.01 sec)

If the Alive status of the BE node is true and the Version value is the new version, the BE node is upgraded successfully.

Restore cluster replica synchronization and balancing

After confirming that the status of each node is correct, execute the following SQL to restore cluster balancing and replica repair:

1. admin set frontend config("disable_balance" = "false");
2. admin set frontend config("disable_colocate_balance" = "false");
3. admin set frontend config("disable_tablet_scheduler" = "false");