Downgrade etcd from v3.6 to v3.5

Downgrade etcd from v3.6 to v3.5

Processes, checklists, and notes on downgrading etcd from v3.6 to v3.5

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
after enabling the downgrade, new features in v3.6 are no longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

Highlighted breaking changes from v3.6 to v3.5:

Difference in flags

If you are using any of the following flags in your v3.6 configurations, make sure to remove, rename, or change the default value when downgrading to v3.5.

NOTE: The diff is based on version v3.6.0 and v.3.5.18. The actual diff would be dependent on your patch version, check with diff <(etcd-3.6/bin/etcd -h | grep \\-\\-) <(etcd-3.5/bin/etcd -h | grep \\-\\-) first.

1. # flags not available in v3.5
2. -etcd --discovery-token ''
3. -etcd --discovery-endpoints ''
4. -etcd --discovery-dial-timeout '2s'
5. -etcd --discovery-request-timeout '5s'
6. -etcd --discovery-keepalive-time '2s'
7. -etcd --discovery-keepalive-timeout '6s'
8. -etcd --discovery-insecure-transport 'true'
9. -etcd --discovery-insecure-skip-tls-verify 'false'
10. -etcd --discovery-cert ''
11. -etcd --discovery-key ''
12. -etcd --discovery-cacert ''
13. -etcd --discovery-user ''
14. -etcd --discovery-password ''
15. -etcd --feature-gates
16. -etcd --log-format

18. # same flag with different names
19. -etcd --bootstrap-defrag-threshold-megabytes
20. +etcd --experimental-bootstrap-defrag-threshold-megabytes
21. -etcd --compaction-batch-limit
22. +etcd --experimental-compaction-batch-limit
23. -etcd --compact-hash-check-time
24. +etcd --experimental-compact-hash-check-time
25. -etcd --compaction-sleep-interval
26. +etcd --experimental-compaction-sleep-interval
27. -etcd --corrupt-check-time
28. +etcd --experimental-corrupt-check-time
29. -etcd --enable-distributed-tracing
30. +etcd --experimental-enable-distributed-tracing
31. -etcd --distributed-tracing-address
32. +etcd --experimental-distributed-tracing-address
33. -etcd --distributed-tracing-instance-id
34. +etcd --experimental-distributed-tracing-instance-id
35. -etcd --distributed-tracing-sampling-rate
36. +etcd --experimental-distributed-tracing-sampling-rate
37. -etcd --distributed-tracing-service-name
38. +etcd --experimental-distributed-tracing-service-name
39. -etcd --downgrade-check-time
40. +etcd --experimental-downgrade-check-time
41. -etcd --max-learners
42. +etcd --experimental-max-learners
43. -etcd --memory-mlock
44. +etcd --experimental-memory-mlock
45. -etcd --peer-skip-client-san-verification
46. +etcd --experimental-peer-skip-client-san-verification
47. -etcd --snapshot-catchup-entries
48. +etcd --experimental-snapshot-catchup-entries
49. -etcd --warning-apply-duration
50. +etcd --experimental-warning-apply-duration
51. -etcd --warning-unary-request-duration
52. +etcd --experimental-warning-unary-request-duration
53. -etcd --watch-progress-notify-interval
54. +etcd --experimental-watch-progress-notify-interval

56. # equivalent flags of v3.6 feature gates
57. -etcd --feature-gates=CompactHashCheck=true
58. +etcd --experimental-compact-hash-check-enabled=true
59. -etcd --feature-gates=InitialCorruptCheck=true
60. +etcd --experimental-enable-initial-corrupt-check=true
61. -etcd --feature-gates=LeaseCheckpoint=true
62. +etcd --experimental-enable-lease-checkpoint=true
63. -etcd --feature-gates=LeaseCheckpointPersist=true
64. +etcd --experimental-enable-lease-checkpoint-persist=true
65. -etcd --feature-gates=StopGRPCServiceOnDefrag=true
66. +etcd --experimental-stop-grpc-service-on-defrag=true
67. -etcd --feature-gates=TxnModeWriteWithSharedBuffer=false
68. +etcd --experimental-txn-mode-write-with-shared-buffer=false

70. # same flag different defaults
71. -etcd --snapshot-count=10000
72. +etcd --snapshot-count=100000
73. -etcd --v2-deprecation='write-only'
74. +etcd --v2-deprecation='not-yet'
75. -etcd --discovery-fallback='exit'
76. +etcd --discovery-fallback='proxy'

Difference in Prometheus metrics

1. # metrics not available in v3.5
2. -etcd_network_known_peers
3. -etcd_server_feature_enabled

Server downgrade checklists

Downgrade requirements

To ensure a smooth rolling downgrade, the running cluster must be healthy. Check the health of the cluster by using the etcdctl endpoint health command before proceeding.

Preparation

Before downgrading etcd, always test the services relying on etcd in a staging environment before deploying the downgrade to the production environment.

Before beginning, download the snapshot backup. Should something go wrong with the downgrade, it is possible to use this backup to rollback back to existing etcd version.

Before beginning, download the latest release of etcd v3.5.

Mixed versions

While downgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is considered downgraded once downgrade is enabled by etcdctl downgrade enable 3.5. Internally, the overall cluster version is set to the downgrade target version, which controls the reported version and the supported features.

Rollback

Before downgrading your etcd cluster, please create and download a snapshot backup of your etcd cluster. This snapshot can be used to restore the cluster to its pre-upgrade state if needed. If users encounter issues during the downgrade, they should first identify and resolve the root cause.

If the downgrade has started after running etcdctl downgrade enabled, and the cluster is still in a mixed-version state—where at least one member remains on v3.6, users can cancel the ongoing downgrade process by running etcdctl downgrade cancel, and restarting all the downgraded members with the original v3.6 binaries.

Once all members have been downgraded to v3.5, the cluster is considered fully downgraded. If users wish to return to the original version after a full downgrade has completed, they should follow the official upgrade guide to ensure consistency and avoid data corruption.

Downgrade procedure

This example shows how to downgrade a 3-member v3.6 etcd cluster running on a local machine.

Step 1: check downgrade requirements

Is the cluster healthy and running v3.6.x?

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint health
2. <<COMMENT
3. localhost:2379 is healthy: successfully committed proposal: took = 2.118638ms
4. localhost:22379 is healthy: successfully committed proposal: took = 3.631388ms
5. localhost:32379 is healthy: successfully committed proposal: took = 2.157051ms
6. COMMENT

8. curl http://localhost:2379/version
9. <<COMMENT
10. {"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
11. COMMENT

13. curl http://localhost:22379/version
14. <<COMMENT
15. {"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
16. COMMENT

18. curl http://localhost:32379/version
19. <<COMMENT
20. {"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
21. COMMENT

23. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
24. <<COMMENT
25. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
26. |    ENDPOINT     |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
27. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
28. |  localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         2 |         10 |                 10 |        |                          |             false |
29. | localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         10 |                 10 |        |                          |             false |
30. | localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.6.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         10 |                 10 |        |                          |             false |
31. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
32. COMMENT

Step 2: download snapshot backup from leader

Download the snapshot backup to provide a downgrade path should any problems occur.

Step 3: validate downgrade target version

Validate the downgrade target version before enabling the downgrade:

We only support downgrading one minor version at a time. e.g downgrading from v3.6 to v3.4 isn’t allowed.
Please do not move on to next step until the validation is successful.

1. etcdctl downgrade validate 3.5
2. <<COMMENT
3. Downgrade validate success, cluster version 3.6
4. COMMENT

Step 4: enable downgrade

1. etcdctl downgrade enable 3.5
2. <<COMMENT
3. Downgrade enable success, cluster version 3.6
4. COMMENT

After enabling downgrade, the cluster will start to operate with v3.5 protocol, which is the downgrade target version. In addition, etcd will automatically migrate the schema to the downgrade target version, which usually happens very fast. Confirm the storage version of all servers has been migrated to v3.5 by checking the endpoint status before moving on to the next step.

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
2. <<COMMENT
3. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
4. |    ENDPOINT     |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
5. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
6. |  localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
7. | localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
8. | localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
9. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
10. COMMENT

NOTE: Once downgrade is enabled, the cluster will remain operating with v3.5 protocol even if all the servers are still running the v3.6 binary, unless the downgrade is canceled with etcdctl downgrade cancel

Step 5: stop one existing etcd server

Before stopping the server, check if it is the leader. We recommend downgrading the leader last.

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
2. <<COMMENT
3. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
4. |    ENDPOINT     |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
5. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
6. |  localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
7. | localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
8. | localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         2 |         12 |                 12 |        |                    3.5.0 |              true |
9. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
10. COMMENT

If the server to be stopped is the leader, you can avoid some downtime by move-leader to another server before stopping this server.

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 move-leader 91bc3c398fb3c146

3. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
4. <<COMMENT
5. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
6. |    ENDPOINT     |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
7. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
8. |  localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         13 |                 13 |        |                    3.5.0 |              true |
9. | localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         3 |         13 |                 13 |        |                    3.5.0 |              true |
10. | localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         13 |                 13 |        |                    3.5.0 |              true |
11. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
12. COMMENT

When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:

1. {"level":"warn","ts":"2025-02-28T17:35:43.795069Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://127.0.0.1:12380/version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
2. {"level":"warn","ts":"2025-02-28T17:35:43.795149Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
3. {"level":"warn","ts":"2025-02-28T17:35:44.368651Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8211f1d0f64f3269","rtt":"483.01µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}
4. {"level":"warn","ts":"2025-02-28T17:35:44.368726Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"8211f1d0f64f3269","rtt":"735.659µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}

Step 6: restart the etcd server with same configuration (minus the flags that are removed or replaced in v3.5)

Restart the etcd server with same configuration but with the new etcd binary.

1. -etcd-3.6/bin --name s1 \
2. +etcd-3.5/bin --name s1 \
3. --data-dir /tmp/etcd/s1 \
4. --listen-client-urls http://localhost:2379 \
5. --advertise-client-urls http://localhost:2379 \
6. --listen-peer-urls http://localhost:2380 \
7. --initial-advertise-peer-urls http://localhost:2380 \
8. --initial-cluster s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380 \
9. --initial-cluster-token tkn \
10. --initial-cluster-state existing

Verify that each member, and then the entire cluster, becomes healthy with the new v3.5 etcd binary:

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
2. <<COMMENT
3. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
4. |    ENDPOINT     |        ID        |    VERSION    | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
5. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
6. |  localhost:2379 | 8211f1d0f64f3269 |        3.5.18 |                 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         14 |                 14 |        |                          |             false |
7. | localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         3 |         14 |                 14 |        |                    3.5.0 |              true |
8. | localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 |           3.5.0 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         14 |                 14 |        |                    3.5.0 |              true |
9. +-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
10. COMMENT

12. etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
13. <<COMMENT
14. localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
15. localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
16. localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
17. COMMENT

NOTE: You will see the DOWNGRADE ENABLED is false for the v3.5 server, because the downgrade info is not implemented in v3.5 status endpoint, downgrade is still enabled for the cluster at this point.

Step 7: repeat step 5 and step 6 for rest of the members

When all members are downgraded, check the health and status of the cluster, and confirm the minor version of all members is v3.5, and storage version is empty:

1. etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
2. <<COMMENT
3. +-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
4. |    ENDPOINT     |        ID        | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
5. +-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
6. |  localhost:2379 | 8211f1d0f64f3269 |  3.5.18 |                 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         26 |                 26 |        |                          |             false |
7. | localhost:22379 | 91bc3c398fb3c146 |  3.5.18 |                 |   20 kB |  16 kB |                   20% |   0 B |      true |      false |         3 |         26 |                 26 |        |                          |             false |
8. | localhost:32379 | fd422379fda50e48 |  3.5.18 |                 |   20 kB |  16 kB |                   20% |   0 B |     false |      false |         3 |         26 |                 26 |        |                          |             false |
9. +-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
10. COMMENT

12. etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
13. <<COMMENT
14. localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
15. localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
16. localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
17. COMMENT

19. curl http://localhost:2379/version
20. <<COMMENT
21. {"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
22. COMMENT

24. curl http://localhost:22379/version
25. <<COMMENT
26. {"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
27. COMMENT

29. curl http://localhost:32379/version
30. <<COMMENT
31. {"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
32. COMMENT

In the log of the leader, you should be able to see message similar to the following:

1. {"level":"info","ts":"2025-02-28T17:59:50.019862Z","caller":"etcdserver/server.go:2749","msg":"the cluster has been downgraded","cluster-version":"3.5.0"}

Last modified March 6, 2025: Add 3.6 downgrade instructions. (530d744)