Quickstart with Docker-Compose

Quickstart with Docker-Compose

Requirement

Start HStream requires an operating system kernel version greater than at least Linux 4.14

Installation

Install docker

TIP

If you have already installed docker, you can skip this step.

See Install Docker Engine (opens new window), and install it for your operating system. Please carefully check that you have met all prerequisites.

Confirm that the Docker daemon is running:

1. docker version

TIP

On Linux, Docker needs root privileges. You can also run Docker as a non-root user, see Post-installation steps for Linux (opens new window).

Install docker compose

TIP

If you have already installed docker compose, you can skip this step.

See Install Docker Compose (opens new window), and install it for your operating system. Please carefully check that you met all prerequisites.

1. docker-compose version

Start HStreamDB Services

WARING

Do NOT use this configuration in your production environment!

Create a docker-compose.yaml file for docker compose, you can download (opens new window) or paste the following contents:

1. # quick-start.yaml

3. version: "3.5"

5. services:
6. hserver0:
7. image: hstreamdb/hstream:v0.11.0
8. depends_on:
9. - zookeeper
10. - hstore
11. ports:
12. - "127.0.0.1:6570:6570"
13. expose:
14. - 6570
15. networks:
16. - hstream-quickstart
17. volumes:
18. - /var/run/docker.sock:/var/run/docker.sock
19. - /tmp:/tmp
20. - data_store:/data/store
21. command:
22. - bash
23. - "-c"
24. - |
25. set -e
26. /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600 \
27. /usr/local/bin/hstream-server \
28. --bind-address 0.0.0.0 --port 6570 \
29. --internal-port 6571 \
30. --server-id 100 \
31. --seed-nodes "$$(hostname -I | awk '{print $$1}'):6571,hserver1:6573" \
32. --advertised-address $$(hostname -I | awk '{print $$1}') \
33. --metastore-uri zk://zookeeper:2181 \
34. --store-config /data/store/logdevice.conf \
35. --store-admin-host hstore --store-admin-port 6440 \
36. --io-tasks-path /tmp/io/tasks \
37. --io-tasks-network hstream-quickstart

39. hserver1:
40. image: hstreamdb/hstream:v0.11.0
41. depends_on:
42. - zookeeper
43. - hstore
44. ports:
45. - "127.0.0.1:6572:6572"
46. expose:
47. - 6572
48. networks:
49. - hstream-quickstart
50. volumes:
51. - /var/run/docker.sock:/var/run/docker.sock
52. - /tmp:/tmp
53. - data_store:/data/store
54. command:
55. - bash
56. - "-c"
57. - |
58. set -e
59. /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600 \
60. /usr/local/bin/hstream-server \
61. --bind-address 0.0.0.0 --port 6572 \
62. --internal-port 6573 \
63. --server-id 101 \
64. --seed-nodes "hserver0:6571,$$(hostname -I | awk '{print $$1}'):6573" \
65. --advertised-address $$(hostname -I | awk '{print $$1}') \
66. --metastore-uri zk://zookeeper:2181 \
67. --store-config /data/store/logdevice.conf \
68. --store-admin-host hstore --store-admin-port 6440 \
69. --io-tasks-path /tmp/io/tasks \
70. --io-tasks-network hstream-quickstart

72. hserver-init:
73. image: hstreamdb/hstream:v0.11.0
74. depends_on:
75. - hserver0
76. - hserver1
77. networks:
78. - hstream-quickstart
79. command:
80. - bash
81. - "-c"
82. - |
83. timeout=120
84. until ( \
85. /usr/local/bin/hadmin server --host hserver0 --port 6570 status && \
86. /usr/local/bin/hadmin server --host hserver1 --port 6572 status \
87. ) >/dev/null 2>&1; do
88. >&2 echo 'Waiting for servers ...'
89. sleep 1
90. timeout=$$((timeout - 1))
91. [ $$timeout -le 0 ] && echo 'Timeout!' && exit 1;
92. done; \
93. /usr/local/bin/hadmin server --host hserver0 init

95. hstore:
96. image: hstreamdb/hstream:v0.11.0
97. networks:
98. - hstream-quickstart
99. volumes:
100. - data_store:/data/store
101. command:
102. - bash
103. - "-c"
104. - |
105. set -ex
106. /usr/local/bin/ld-dev-cluster --root /data/store \
107. --use-tcp --tcp-host $$(hostname -I | awk '{print $$1}') \
108. --user-admin-port 6440 \
109. --no-interactive

111. zookeeper:
112. image: zookeeper
113. expose:
114. - 2181
115. networks:
116. - hstream-quickstart
117. volumes:
118. - data_zk_data:/data
119. - data_zk_datalog:/datalog

121. networks:
122. hstream-quickstart:
123. name: hstream-quickstart

125. volumes:
126. data_store:
127. name: quickstart_data_store
128. data_zk_data:
129. name: quickstart_data_zk_data
130. data_zk_datalog:
131. name: quickstart_data_zk_datalog

then run:

1. docker-compose -f quick-start.yaml up

If you see some thing like this, then you have a running hstream:

1. hserver_1    | [INFO][2021-11-22T09:15:18+0000][app/server.hs:137:3][thread#67]************************
2. hserver_1    | [INFO][2021-11-22T09:15:18+0000][app/server.hs:145:3][thread#67]Server started on port 6570
3. hserver_1    | [INFO][2021-11-22T09:15:18+0000][app/server.hs:146:3][thread#67]*************************

TIP

You can also run in background.

1. docker-compose -f quick-start.yaml up -d

And if you want to show logs of server, run:

1. docker-compose -f quick-start.yaml logs -f hserver

Start HStreamDB’s interactive SQL CLI

1. docker run -it --rm --name some-hstream-cli --network host hstreamdb/hstream:v0.11.0 hstream --port 6570 sql

If everything works fine, you will enter an interactive CLI and see help information like

1. __  _________________  _________    __  ___
2. / / / / ___/_  __/ __ \/ ____/   |  /  |/  /
3. / /_/ /\__ \ / / / /_/ / __/ / /| | / /|_/ /
4. / __  /___/ // / / _, _/ /___/ ___ |/ /  / /
5. /_/ /_//____//_/ /_/ |_/_____/_/  |_/_/  /_/

7. Command
8. :h                           To show these help info
9. :q                           To exit command line interface
10. :help [sql_operation]        To show full usage of sql statement

12. SQL STATEMENTS:
13. To create a simplest stream:
14. CREATE STREAM stream_name;

16. To create a query select all fields from a stream:
17. SELECT * FROM stream_name EMIT CHANGES;

19. To insert values to a stream:
20. INSERT INTO stream_name (field1, field2) VALUES (1, 2);

22. >

Create a stream

What we are going to do first is create a stream by CREATE STREAM statement.

1. CREATE STREAM demo;

Run a continuous query over the stream

Now we can run a continuous query over the stream we just created by SELECT query.

The query will output all records from the demo stream whose humidity is above 70 percent.

1. SELECT * FROM demo WHERE humidity > 70 EMIT CHANGES;

It seems that nothing happened. But do not worry because there is no data in the stream now. Next, we will fill the stream with some data so the query can produce output we want.

Start another CLI session

Start another CLI session, this CLI will be used for inserting data into the stream.

1. docker exec -it some-hstream-cli hstream --port 6570 sql

Insert data into the stream

Run each of the given INSERT statement in the new CLI session and keep an eye on the CLI session created in (2).

1. INSERT INTO demo (temperature, humidity) VALUES (22, 80);
2. INSERT INTO demo (temperature, humidity) VALUES (15, 20);
3. INSERT INTO demo (temperature, humidity) VALUES (31, 76);
4. INSERT INTO demo (temperature, humidity) VALUES ( 5, 45);
5. INSERT INTO demo (temperature, humidity) VALUES (27, 82);
6. INSERT INTO demo (temperature, humidity) VALUES (28, 86);

If everything works fine, the continuous query will output matching records in real time:

1. {"temperature":22,"humidity":80}
2. {"temperature":31,"humidity":76}
3. {"temperature":27,"humidity":82}
4. {"temperature":28,"humidity":86}

Getting Started