Cluster setup

You can follow this document to start an IoTDB cluster.

Prerequisite

To use IoTDB, you need to have:

1. Java >= 1.8 (Please make sure the environment path has been set)
2. Set the max open files num as 65535 to avoid “too many open files” problem.

Installation

IoTDB provides you three installation methods, you can refer to the following suggestions, choose one of them:

• Installation from source code. If you need to modify the code yourself, you can use this method.
• Installation from binary files. Download the binary files from the official website.
• Using Docker：The path to the dockerfile is <https://github.com/apache/iotdb/blob/master/docker/src/main>

Build from source

You can download the source code from:

1. git clone https://github.com/apache/iotdb.git

The default dev branch is the master branch, If you want to use a released version, for example:

1. git checkout rel/0.12

Under the root path of iotdb:

1. > mvn clean package -pl cluster -am -DskipTests

Then the cluster binary version can be found at cluster/target/{iotdb-project.version}

Download

You can download the binary file from: Download Page (opens new window)

Directory

After installation, the following directories will be generated by default under the root directory of the iotdb cluster:

Configurations

Before starting to use IoTDB, you need to config the configuration files first. For your convenience, we have already set the default config in the files.

In total, we provide users four kinds of configurations module:

• environment configuration file (iotdb-env.bat, iotdb-env.sh). The default configuration file for the environment configuration item. Users can configure the relevant system configuration items of JAVA-JVM in the file.
• system configuration file (iotdb-engine.properties). The default configuration file for the IoTDB engine layer configuration item. Users can configure the IoTDB engine related parameters in the file, such as the precision of timestamp(timestamp_precision), etc. What’s more, Users can configure settings of TsFile (the data files), such as the data size written to the disk per time(group_size_in_byte).
• log configuration file (logback.xml). The default log configuration file, such as the log levels.
• iotdb-cluster.properties. Some configurations required by IoTDB cluster. Such as replication number(default_replica_num), etc.

For detailed description of the two configuration files iotdb-engine.properties, iotdb-env.sh/iotdb-env.bat, please refer to Configuration Manual (opens new window). The configuration items of IoTDB cluster are in the iotdb-cluster.properties file, you can also review the comments in the configuration file (opens new window) directly or you can refer to Cluster Configuration.

Configuration files are located at {cluster_root_dir}/conf

You are necessary to modify the following configuration items of each node to start your IoTDB cluster：

• iotdb-engine.properties:

• rpc_address

• rpc_port
• base_dir
• data_dirs
• wal_dir

• iotdb-cluster.properties

• internal_ip

• internal_meta_port
• internal_data_port
• cluster_info_public_port
• seed_nodes
• default_replica_num

OverWrite the configurations of Stand-alone node

Some configurations in the iotdb-engines.properties will be ignored

• enable_auto_create_schema is always considered as false. Use enable_auto_create_schema in iotdb-cluster.properties to enable it, instead.
• is_sync_enable is always considered as false.

Start Service

Start Cluster

You can deploy a distributed cluster on multiple nodes or on a single machine, the main difference being that the latter needs to handle conflicts between ports and file directories. For detail descriptions, please refer to Configurations.

To start the service of one of the nodes, you need to execute the following commands:

1. # Unix/OS X
2. > nohup sbin/start-node.sh [printgc] [<conf_path>] >/dev/null 2>&1 &

4. # Windows
5. > sbin\start-node.bat [printgc] [<conf_path>]

printgc means whether enable the gc and print gc logs when start the node, <conf_path> use the configuration file in the conf_path folder to override the default configuration file.

If you start all the seed nodes, and all the seed nodes can contact each other without ip/port and file directory conflicts, the cluster has successfully started.

Cluster scalability

In the process of cluster running, users can add new nodes to the cluster or delete existing nodes. At present, it only supports node by node cluster scalability, and multi node cluster scalability can be transformed into a series of single node cluster scalability operations. The cluster will hanlde new cluster extension operations only after the last cluster scalability operation is completed.

Start the following script on the new node to join the cluster to add a new node:

1. # Unix/OS X
2. > nohup sbin/add-node.sh [printgc] [<conf_path>] >/dev/null 2>&1 &
3. ;
4. # Windows
5. > sbin\add-node.bat [printgc] [<conf_path>]

printgc means whether enable the gc and print gc logs when start the node, <conf_path> use the configuration file in the conf_path folder to override the default configuration file. GC log is off by default. For performance tuning, you may want to collect the GC info. GC log is stored at IOTDB_HOME/logs/gc.log.

Start the following script on any node in the cluster to delete a node:

1. # Unix/OS X
2. > sbin/remove-node.sh <internal_ip> <internal_meta_port>

4. # Windows
5. > sbin\remove-node.bat <internal_ip> <internal_meta_port>

internal_ip means the IP address of the node to be deleted internal_meta_port means the meta port of the node to be deleted

Use Cli

please refer to QuickStart (opens new window). You can establish a connection with any node in the cluster according to the rpc_address and rpc_port.

Stop Cluster

To stop the services of all the nodes on a single machine, you need to execute the following commands:

1. # Unix/OS X
2. > sbin/stop-node.sh

4. # Windows
5. > sbin\stop-node.bat

Appendix

Cluster Configuration

• internal_ip

• internal_meta_port

• internal_data_port

• cluster_info_public_port

• open_server_rpc_port

• seed_nodes

• rpc_thrift_compression_enable

• default_replica_num

• multi_raft_factor

• cluster_name

• connection_timeout_ms

• heartbeat_interval_ms

• election_timeout_ms

• read_operation_timeout_ms

• write_operation_timeout_ms

• min_num_of_logs_in_mem

• max_num_of_logs_in_mem

• log_deletion_check_interval_second

• enable_auto_create_schema

• consistency_level

• is_enable_raft_log_persistence