Distributed Settings

For a distributed setting, the following parameters should be paied attention to.

Configuration Settings

To configurate a distributed setting, following parameters need to be modified in the default Configuration file ‘system_vars_config.toml’.

• nodeID

nodeID is the Node ID of the cube. In a cluster, each node should have a different nodeID. - addr-raft and addr-advertise-raft

addr-raft is the address for raft-group rpc communication.

It is the 10000 port of the node.

1. addr-raft = "localhost:10000"

Only docker deployment need adjust addr-advertise-raft

For docker deployment, the ip is 0.0.0.0

1. addr-raft = "0.0.0.0:10000"

In the case that some clients cannot access the raft-group, addr-advertise-raft must be manually set.

1. addr-advertise-raft = "${HOST}:10000" - addr-client and addr-advertise-client

addr-client is the address for cube service. It is the 20000 port of the node.

1. addr-client = "localhost:20000"

Only docker deployment need adjust addr-advertise-client

For docker deployment, the ip is 0.0.0.0

1. addr-client = "0.0.0.0:20000"

In the case that a client cannot access Cube through the default client URLs listened to by Cube, addr-advertise-client must be manually set.

1. addr-advertise-client = "${HOST}:20000" - dir-data

dir-data is the directory for cube data. In a cluster, each node should have a different dir-data. - store-heartbeat-duration

store-heartbeat-duration is the period for this node to report information to scheduler. It should be less than 10s. - prophet name

name in Prophet Configs is the name of the node. In a cluster, each node should have a different prophet name.

1. name = "node0" - rpc-addr and rpc-advertise-addr

rpc-addr is the address for other clients to access prophet. It is the 30000 port of the node.

1. rpc-addr = "localhost:30000"

Only docker deployment need adjust rpc-advertise-addr

For docker deployment, the ip is 0.0.0.0

1. rpc-addr = "0.0.0.0:30000"

In the case that a client cannot access prophet, rpc-advertise-addr must be manually set.

1. rpc-advertise-addr = "${HOST}:30000" - prophet-node

prophet-node is whether the node is a pure prophet node. For the three prophet nodes, prophet-node is true.

1. prophet-node = true

For pure prophet nodes, prophet-node is false

1. prophet-node = false - external-etcd

For the three prophet nodes, external-etcd is empty.

1. external-etcd = ["", "", ""]

For pure storage nodes, external-etcd is the list of the client-urls of the three prophet nodes, i.e. the 40000 port of the prophet nodes.

1. external-etcd = ["http://ip_of_prophet_node1:40000", "http://ip_of_prophet_node2:40000", "http://ip_of_prophet_node3:40000"] - join

join is for the second and third prophet nodes to join the raft group. For the first prophet node, join is a empty string.

1. join = ""

For the second and third prophet node, join is the peer-urls of the first prophet node.

1. join = "http://ip_of_the_first_prophet_node:40000"

For pure storage nodes, there’s no need to adjust join. - client-urls and advertise-client-urls

client-urls is exposed to other nodes in the cluster. It is 40000 port of the node

1. client-urls = "http://localhost:40000"

Only docker deployment need adjust advertise-client-urls

For docker deployment, the ip is 0.0.0.0

1. client-urls = "http://0.0.0.0:40000"

In the case that some clients ccannot access prophet through the default client URLs listened to by prophet, advertise-client-urls must be manually set.

1. advertise-client-urls = "http://${HOST}:40000 - peer-urls and advertise-peer-urls

peer-urls is the list of peer URLs to be listened to by a prophet node. It is the 40000 port of the node.

1. peer-urls = "http://localhost:50000"

Only docker deployment need adjust advertise-peer-urls

For docker deployment, the ip is 0.0.0.0

1. peer-urls = "http://0.0.0.0:50000"

In the case that a node cannot be listened to by a prophet node, advertise-peer-urls should be manually set.

1. advertise-peer-urls = "http://${HOST}:50000" - max-replicas

max-replicas is the max number of replica in a prophet group. It should be 3.

1. max-replicas = 3