Deployment with hdt

Deployment with hdt

This document provides a way to start an HStreamDB cluster quickly using the deployment tool hdt.

Pre-Require

Start HStreamDB requires an operating system kernel version greater than at least Linux 4.14. Check with command:

1. uname -r - The local host needs to be able to connect to the remote server via SSH - Make sure remote server has docker installed. - Make sure that the log-in user has sudo execute privileges, and configure sudo without password. - For nodes which deploy HStore instances, mount the data disk to /mnt/data*/.

“*“ Matching incremental numbers, start from zero
one disk should mount to one directory. e.g. if we have two data disks /dev/vdb and /dev/vdc, then /dev/vdb should mount to /mnt/data0 and /dev/vdc should mount to /mnt/data1

Deploy `hdt` on the control machine

We’ll use a deployment tool hdt to help us set up the cluster. The binaries are available here: <https://github.com/hstreamdb/deployment-tool/releases>.

Log in to the control machine and download the binaries.
Generate configuration template with command:

1. ./hdt init

The current directory structure will be as follows after running the init command:

1. ├── hdt 2. └── template 3. ├── config.yaml 4. ├── grafana 5. │ ├── dashboards 6. │ └── datasources 7. ├── prometheus 8. └── script

Update `Config.yaml`

template/config.yaml contains the template for the configuration file. Refer to the description of the fields in the file and modify the template according to your actual needs.

Here we will deploy a cluster on 3 nodes, each consisting of a HServer instance, a HStore instance and a Meta-Store instance as a simple example. For hstream monitor stack, refer to monitor components config.

The final configuration file may looks like:

1. global:
2. user: "root"
3. key_path: "~/.ssh/id_rsa"
4. ssh_port: 22

6. hserver:
7. - host: 172.24.47.175
8. - host: 172.24.47.174
9. - host: 172.24.47.173

11. hstore:
12. - host: 172.24.47.175
13. enable_admin: true
14. - host: 172.24.47.174
15. - host: 172.24.47.173

17. meta_store:
18. - host: 172.24.47.175
19. - host: 172.24.47.174
20. - host: 172.24.47.173

Set up cluster

set up cluster with ssh key-value pair

1. ./hdt start -c template/config.yaml -i ~/.ssh/id_rsa -u root

set up cluster with passwd

1. ./hdt start -c template/config.yaml -p -u root

then type your password.

use ./hdt start -h for more information

Remove cluster

remove cluster will stop cluster and remove ALL related data.

remove cluster with ssh key-value pair

1. ./hdt remove -c template/config.yaml -i ~/.ssh/id_rsa -u root

remove cluster with passwd

1. ./hdt remove -c template/config.yaml -p -u root

then type your password.

Detailed configuration items

This section describes in detail the meaning of each field in the configuration file. The configuration file is divided into three large sections: global configuration items, monitoring component configuration items and other component configuration items.

Global

1. global:
2. # # Username to login via SSH
3. user: "root"
4. # # The path of SSH identity file
5. key_path: "~/.ssh/hstream-aliyun.pem"
6. # # SSH service monitor port
7. ssh_port: 22
8. # # Replication factors of store metadata
9. meta_replica: 3
10. # # Local path to MetaStore config file
11. meta_store_config_path: ""
12. # # Local path to HStore config file
13. hstore_config_path: ""
14. # # HStore config file can be loaded from network filesystem, for example, the config file
15. # # can be stored in meta store and loaded via network request. Set this option to true will
16. # # force store load config file from its local filesystem.
17. disable_store_network_config_path: true
18. # # Local path to HServer config file
19. hserver_config_path: ""
20. # # Global container configuration
21. container_config:
22. cpu_limit: 200
23. memory_limit: 8G
24. disable_restart: true
25. remove_when_exit: true

Global section set the default configuration value for all other configuration items. Here are some notes:

meta_replica set the replication factors of HStreamDB metadata. This value should not exceed the number of hstore instances.
meta_store_config_path、hstore_config_path and hserver_config_path are configuration file path for meta_store、hstore and hserver in the control machine. If the paths are set, these configuration files will be synchronized to the specified location on the node where the respective instance is located, and the corresponding configuration items will be updated when the instance is started.
container_config let you set resource limitations for all containers.

monitor

1. monitor:
2. # # Node exporter port
3. node_exporter_port: 9100
4. # # Node exporter image
5. node_exporter_image: "prom/node-exporter"
6. # # Cadvisor port
7. cadvisor_port: 7000
8. # # Cadvisor image
9. cadvisor_image: "gcr.io/cadvisor/cadvisor:v0.39.3"
10. # # List of nodes that won't be monitored.
11. excluded_hosts: []
12. # # root directory for all monitor related config files.
13. remote_config_path: "/home/deploy/monitor"
14. # # root directory for all monitor related data files.
15. data_dir: "/home/deploy/data/monitor"
16. # # Set up grafana without login
17. grafana_disable_login: true
18. # # Global container configuration for monitor stacks.
19. container_config:
20. cpu_limit: 200
21. memory_limit: 8G
22. disable_restart: true
23. remove_when_exit: true

Monitor section sets configuration items related to cadvisor and node-exporter

hserver

1. hserver:
2. # # The ip address of the HServer
3. - host: 10.1.0.10
4. # # HServer docker image
5. image: "hstreamdb/hstream"
6. # # HServer listen port
7. port: 6570
8. # # HServer internal port
9. internal_port: 6571
10. # # HServer configuration
11. server_config:
12. # # HServer log level, valid values: [critical|error|warning|notify|info|debug]
13. server_log_level: info
14. # # HStore log level, valid values: [critical|error|warning|notify|info|debug|spew]
15. store_log_level: info
16. # # Specific server compression algorithm, valid values: [none|lz4|lz4hc]
17. compression: lz4
18. # # Root directory of HServer config files
19. remote_config_path: "/home/deploy/hserver"
20. # # Root directory of HServer data files
21. data_dir: "/home/deploy/data/hserver"
22. # # HServer container configuration
23. container_config:
24. cpu_limit: 200
25. memory_limit: 8G
26. disable_restart: true
27. remove_when_exit: true

HServer section sets configuration items for hserver

hstore

1. hstore:
2. - host: 10.1.0.10
3. # # HStore docker image
4. image: "hstreamdb/hstream"
5. # # HStore admin port
6. admin_port: 6440
7. # # Root directory of HStore config files
8. remote_config_path: "/home/deploy/hstore"
9. # # Root directory of HStore data files
10. data_dir: "/home/deploy/data/store"
11. # # Total used disks
12. disk: 1
13. # # Total shards
14. shards: 2
15. # # The role of the HStore instance.
16. role: "Both" # [Storage|Sequencer|Both]
17. # # When Enable_admin is turned on, the instance can receive and process admin requests
18. enable_admin: true
19. # # HStore container configuration
20. container_config:
21. cpu_limit: 200
22. memory_limit: 8G
23. disable_restart: true
24. remove_when_exit: true

HStore section sets configuration items for hstore.

admin_port: HStore service will listen on this port.
disk and shards: Set total used disks and total shards. For example, disk: 2 and shards: 4 means the hstore will persistant data in two disks, and each disk will contain 2 shards.
role: a HStore instance can act as a Storage, a Sequencer or both, default is both.
enable_admin: set the HStore instance with an admin server embedded.

meta-store

1. meta_store:
2. - host: 10.1.0.10
3. # # Meta-store docker image
4. image: "zookeeper:3.6"
5. # # Meta-store port, currently only works for rqlite. zk will
6. # # monitor on 4001
7. port: 4001
8. # # Raft port used by rqlite
9. raft_port: 4002
10. # # Root directory of Meta-Store config files
11. remote_config_path: "/home/deploy/metastore"
12. # # Root directory of Meta-store data files
13. data_dir: "/home/deploy/data/metastore"
14. # # Meta-store container configuration
15. container_config:
16. cpu_limit: 200
17. memory_limit: 8G
18. disable_restart: true
19. remove_when_exit: true

Meta-store section sets configuration items for meta-store.

port and raft_port: these are used by rqlite

monitor stack components

1. http_server:
2. - host: 10.1.0.15
3. # # Http_server docker image
4. image: "hstreamdb/http-server"
5. # # Http_server service monitor port
6. port: 8081
7. # # Root directory of http_server config files
8. remote_config_path: "/home/deploy/http-server"
9. # # Root directory of http_server data files
10. data_dir: "/home/deploy/data/http-server"
11. container_config:
12. cpu_limit: 200
13. memory_limit: 8G
14. disable_restart: true
15. remove_when_exit: true

17. prometheus:
18. - host: 10.1.0.15
19. # # Prometheus docker image
20. image: "prom/prometheus"
21. # # Prometheus service monitor port
22. port: 9090
23. # # Root directory of Prometheus config files
24. remote_config_path: "/home/deploy/prometheus"
25. # # Root directory of Prometheus data files
26. data_dir: "/home/deploy/data/prometheus"
27. # # Prometheus container configuration
28. container_config:
29. cpu_limit: 200
30. memory_limit: 8G
31. disable_restart: true
32. remove_when_exit: true

34. grafana:
35. - host: 10.1.0.15
36. # # Grafana docker image
37. image: "grafana/grafana-oss:main"
38. # # Grafana service monitor port
39. port: 3000
40. # # Root directory of Grafana config files
41. remote_config_path: "/home/deploy/grafana"
42. # # Root directory of Grafana data files
43. data_dir: "/home/deploy/data/grafana"
44. # # Grafana container configuration
45. container_config:
46. cpu_limit: 200
47. memory_limit: 8G
48. disable_restart: true
49. remove_when_exit: true

51. alertmanager:
52. # # The ip address of the Alertmanager Server.
53. - host: 10.0.1.15
54. # # Alertmanager docker image
55. image: "prom/alertmanager"
56. # # Alertmanager service monitor port
57. port: 9093
58. # # Root directory of Alertmanager config files
59. remote_config_path: "/home/deploy/alertmanager"
60. # # Root directory of Alertmanager data files
61. data_dir: "/home/deploy/data/alertmanager"
62. # # Alertmanager container configuration
63. container_config:
64. cpu_limit: 200
65. memory_limit: 8G
66. disable_restart: true
67. remove_when_exit: true

69. hstream_exporter:
70. - host: 10.1.0.15
71. # # hstream_exporter docker image
72. image: "hstreamdb/hstream-exporter"
73. # # hstream_exporter service monitor port
74. port: 9200
75. # # Root directory of hstream_exporter config files
76. remote_config_path: "/home/deploy/hstream-exporter"
77. # # Root directory of hstream_exporter data files
78. data_dir: "/home/deploy/data/hstream-exporter"
79. container_config:
80. cpu_limit: 200
81. memory_limit: 8G
82. disable_restart: true
83. remove_when_exit: true

Currently, HStreamDB monitor stack contains the following components：node-exporter, cadvisor, http-server, prometheus, grafana, alertmanager and hstream-exporter. The global configuration of the monitor stack is available in monitor field.

Deployment with hdt

Deployment with hdt

Pre-Require

Deploy hdt on the control machine

Update Config.yaml

Set up cluster

set up cluster with ssh key-value pair

set up cluster with passwd

Remove cluster

remove cluster with ssh key-value pair

remove cluster with passwd

Detailed configuration items

Global

monitor

hserver

hstore

meta-store

monitor stack components

Deploy `hdt` on the control machine

Update `Config.yaml`