Integration with Grafana

NOTE: Some dashboard panels and metrics require Manticore Buddy. If some panels appear empty, ensure that Buddy is installed.

Grafana is an open-source platform for data visualization and monitoring that enables the creation of interactive dashboards and charts. Manticore Search can integrate with Grafana in two primary ways:

  1. Monitoring Manticore Performance: Using Prometheus to collect metrics from Manticore and visualize them in Grafana dashboards. This approach focuses on system health, performance monitoring, and alerting for the search engine itself.
  2. Visualizing Search Data: Connecting via the MySQL connector to query and display data stored in Manticore tables, similar to how Kibana is used for Elasticsearch data visualization. This is ideal for analyzing trends, aggregations, and custom visualizations based on your indexed data.

Currently, Grafana versions 10.0-13.2 are tested and supported.

Monitoring Manticore with Prometheus and Grafana

Manticore provides a prebuilt Grafana dashboard and Prometheus alerting rules in the dedicated repository manticoresoftware/grafana-dashboard. The repository also includes an all-in-one Docker image with Grafana and Prometheus for a quick start. These resources can be used independently with your existing Grafana and Prometheus setup, allowing you to monitor Manticore's operational metrics—such as latency, resource usage, and errors—without needing to run a separate instance.

Manticore Grafana dashboard preview

  • Additional screenshots: <https://github.com/manticoresoftware/grafana-dashboard>
  • Dashboard JSON:
  • Alert rules (Prometheus/Alertmanager):

Quick Start (Docker Image)

For a fully preconfigured setup, run the all-in-one Docker image with Grafana and Prometheus tailored for Manticore monitoring. This is ideal if you don't have an existing setup or want a quick test environment:

docker run -e MANTICORE_TARGETS=localhost:9308 -p 127.0.0.1:3000:3000 manticoresearch/dashboard:latest

MANTICORE_TARGETS is a comma-separated list of Manticore http endpoints in the host:port format (for example: localhost:9308,10.0.0.5:9308).

If you already have your own Grafana and Prometheus running, you can skip the Docker image and directly import the dashboard and alert rules as described below.

Importing the Dashboard

To add the Manticore dashboard to your existing Grafana instance:

  1. In Grafana, navigate to Dashboards → New → Import.
  2. Upload the manticore-dashboard.json file (or paste its JSON content).
  3. Select your Prometheus data source when prompted.
  4. Verify and adjust the job dashboard variable if needed (the default expected label value is manticoresearch).

Using the Alert Rules

To integrate the Manticore alert rules into your existing Prometheus setup:

  1. Download manticore-alerts.yml and add it to your Prometheus rule_files configuration.
  2. If your scrape job name differs from manticoresearch, update the {job=“manticoresearch”} matchers in the rules accordingly.
  3. Reload or restart Prometheus to apply the changes.

Alert Rules (What They Mean)

The manticore-alerts.yml file is designed to focus on key signals related to availability, overload, and resource exhaustion—issues that often impact users directly. Thresholds are set as safe defaults; adjust them based on your specific workload.

  • ManticoreTargetDown (critical): Prometheus fails to scrape Manticore (up == 0 for 2 minutes). This typically indicates the metrics endpoint is down, unreachable, or the scrape configuration is incorrect.
  • ManticoreRecentlyRestarted (warning): Uptime remains below 5 minutes for 5 minutes. This often signals instability, such as crash loops, OOM kills, or orchestrator restarts.
  • ManticoreMaxedOutErrors (warning): manticore_maxed_out_error_count increases over the last 5 minutes. Manticore is rejecting requests due to hitting capacity, concurrency, or resource limits.
  • ManticoreSearchLatencyP95High (warning): P95 search latency exceeds 500 ms for 10 minutes. Many users may perceive searches as slow.
  • ManticoreSearchLatencyP99High (critical): P99 search latency exceeds 1,000 ms for 10 minutes. Tail latency is severely elevated, indicating very slow worst-case requests.
  • ManticoreWorkQueueBacklog (warning): Work queue length exceeds 100 for 5 minutes. Requests are accumulating, which often leads to increased latency.
  • ManticoreWorkersSaturated (warning): More than 90% of workers are active for 10 minutes. Worker threads are nearing full capacity, likely causing higher latency and queueing.
  • ManticoreQueryCacheNearLimit (warning): Query cache usage exceeds 90% of its configured maximum for 10 minutes. This increases the risk of cache churn and evictions, potentially slowing queries; consider expanding the cache or optimizing query patterns.
  • ManticoreAgentRetryCountHigh (warning): manticore_agent_retry_count increases by more than 10 in 5 minutes. This usually points to connectivity issues with remote agents (e.g., network problems, timeouts, or failures in distributed queries).
  • ManticoreCurrentConnectionsHigh (warning): Current connections exceed 500 for 10 minutes. This could indicate a traffic surge, connection leaks, or slow clients.
  • ManticoreSlowestThreadHigh (warning): manticore_slowest_thread_seconds exceeds 30 seconds for 10 minutes. Long-running or stuck queries are present; investigate slow queries and resource contention.
  • ManticoreConnectTimeHigh (warning): manticore_connect_time_seconds exceeds 0.2 seconds for 5 minutes. Connection establishment is delayed (e.g., due to network issues, TLS overhead, or server overload).
  • ManticoreSearchdCrashesRecently (critical): manticore_searchd_crashes_total increases in the last 10 minutes. A crash has occurred; review logs, core dumps, and OOM events.
  • ManticoreBinlogFilesHigh (warning): Binlog file count exceeds 1,000 for 10 minutes. This suggests binlogs are not rotating or cleaning properly; check binlog settings and disk usage.
  • ManticoreSearchdFdCountHigh (warning): searchd file descriptor count exceeds 4,096 for 10 minutes. This risks hitting OS limits; verify ulimit -n, connections, open files, and potential leaks.
  • ManticoreBuddyFdCountHigh (warning): Buddy file descriptor count exceeds 4,096 for 10 minutes. Similar to the above, but for the Buddy process.
  • ManticoreSearchdAnonRssHigh (warning): searchd anonymous RSS exceeds 8 GiB for 10 minutes. High non-file-backed memory usage; examine memory growth, query/index patterns, and container limits.
  • ManticoreBuddyAnonRssHigh (warning): Buddy anonymous RSS exceeds 8 GiB for 10 minutes. Elevated Buddy memory usage; review Buddy logs and workload.
  • ManticoreNonServedTables (warning): One or more tables listed in manticore.json are missing from SHOW TABLES for 10 minutes. This typically means tables failed to load or were dropped/moved; check startup logs and table paths.
  • ManticoreDiskMappedCacheLow (warning): Disk mapped cache ratio falls below 50% for 15 minutes. Increased disk I/O is likely, potentially slowing searches if the working set exceeds RAM.
  • ManticoreDiskMappedCacheVeryLow (critical): Disk mapped cache ratio falls below 20% for 15 minutes. Severe cache misses; anticipate heavy disk I/O and high latency.

Dashboard Panels (What They Show)

The dashboard organizes panels by topic. Key highlights include:

  • Health / Uptime: Time since the last restart; frequent drops indicate restarts or crashes.
  • Performance / Search Latency (P95/P99): Tail latency metrics; persistent increases often signal overload or slow queries.
  • Performance / Work Queue Length and Load / Queue Pressure: Backlog and queueing indicators; growing queues suggest the system is lagging.
  • Workers Utilization % and Load / Workers Active/Total: Worker saturation levels; values near 100% typically correlate with higher latency.
  • Performance / Query Cache Usage and Performance / Query Cache Hit Rate: Cache pressure and effectiveness; high usage with low hit rates may indicate inefficiency.
  • Tables / Disk Mapped Cache Ratio (Worst 10): Low ratios imply more disk reads and potentially slower queries.

Visualizing Manticore Data in Grafana

This approach uses Grafana's MySQL connector to directly query Manticore tables, enabling visualizations of your search data, such as time-series trends or aggregations.

Prerequisites

Before setting up data visualization:

  1. Ensure Manticore Search (version 6.2.0 or later) is installed and configured. See the official Manticore Search installation guide for details.
  2. Install Grafana on your system. Refer to the official Grafana installation guide for instructions.

Connecting Manticore Search to Grafana

To connect Manticore Search to Grafana:

  1. Log in to your Grafana dashboard and click on “Configuration” (the gear icon) in the left sidebar.
  2. Select “Data Sources” and click on “Add data source.”
  3. Choose “MySQL” from the list of available data sources.
  4. In the settings page, provide the following details:
  5. Name: A name for the data source (e.g., “Manticore Search”)
  6. Host: The hostname or IP address of your Manticore Search server (with MySQL port, default: localhost:9306)
  7. Database: Leave empty or specify your database name
  8. User: The username with access to Manticore Search (default: root)
  9. Password: The password for the specified user (default: empty)
  10. Click on “Save & Test” to verify the connection.

Creating Visualizations and Dashboards

After connecting Manticore Search to Grafana, you can create dashboards and visualizations:

  1. In the Grafana dashboard, click on the “+” icon in the left sidebar and select “New dashboard.”
  2. Click on the “Add visualization” button to start configuring your chart.
  3. Choose the Manticore Search data source that you connected via the MySQL connector.
  4. Select the type of chart you want to create (e.g., Time Series, Bar chart, Candlestick, Pie chart).
  5. Use Grafana's query builder or write an SQL query to fetch data from your Manticore Search tables.
  6. Customize the chart's appearance, labels, and other settings as needed.
  7. Click “Apply” to save your visualization to the dashboard.

Example Use Case

Here's a simple example using time-series data. First, create a table and load sample data:

CREATE TABLE btc_usd_trading (
  id bigint,
  time timestamp,
  open float,
  high float,
  low float,
  close float
);

Load the data:

curl -sSL https://gist.githubusercontent.com/donhardman/df109ba6c5e690f73198b95f3768e73f/raw/0fab3aee69d7007fad012f4e97f38901a64831fb/btc_usd_trading.sql | mysql -h0 -P9306

In Grafana, you can create:

  • Time Series chart: Visualize price changes over time
  • Candlestick chart: Display open, high, low, close values for financial data
  • Aggregation charts: Use COUNT, AVG, MAX, MIN functions

Example queries:

-- Time series query
SELECT time, close FROM btc_usd_trading ORDER BY time;
-- Aggregation query
SELECT DATE(time) as date, AVG(close) as avg_price
FROM btc_usd_trading
GROUP BY date
ORDER BY date;

Supported Features

When working with Manticore Search through Grafana, you can:

  • Execute SQL queries against Manticore Search tables
  • Use aggregation functions: COUNT, AVG, MAX, MIN
  • Apply GROUP BY and ORDER BY operations
  • Filter data with WHERE clauses
  • Access table metadata via information_schema.tables
  • Create various visualization types supported by Grafana

Limitations

  • Some advanced MySQL features may not be available when working with Manticore Search through Grafana.
  • Only features supported by Manticore Search are available. Refer to the SQL reference for details.

References

For more information and detailed tutorials:

Last modified: August 19, 2026

Integration of Manticore with Kibana

Kibana is a visual interface that allows you to explore, visualize, and create dashboards for your log data. Integrating Kibana with Manticore Search can speed up the loading of Kibana visualizations by up to 3 times compared to Elasticsearch, as demonstrated in this demo. This integration enables users to seamlessly analyze their data using interactive dashboards, custom visualizations, and real-time search capabilities. It also simplifies handling diverse data sources by supporting tools like Logstash and Filebeat for streamlined data ingestion, making it a great choice for log analysis workflows.

Prerequisites

  1. Download Kibana: Ensure you download a Kibana version compatible with Manticore. Currently, version 7.6.0 is tested and recommended. Other 7.x versions may work but could introduce issues. Version 8.x is not supported.
  2. Verify Manticore: Ensure your Manticore instance is running and its HTTP API is reachable (default: http://localhost:9308).

Configuration

  1. Open the Kibana configuration file (kibana.yml).
  2. Set the URL of your Manticore instance:

elasticsearch.hosts: ["http://localhost:9308"] 3. Start Kibana and open it in your browser at http://localhost:5601. Replace localhost with your server's IP or hostname if necessary.

Note: Manticore does not require authentication setup when working with Kibana. Also note that Manticore must work in real-time mode to be integrated with Kibana.

Manticore configuration example

    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
 }

Supported Features

Discover

  • Use the Discover tab in Kibana to search and filter your data interactively.
  • Refine your searches using the query bar with simple queries in the Kibana query language.

Visualizations

  • Navigate to Visualizations to create custom visualizations:
  • Create a table pattern (it’s called an 'index pattern' in Kibana) if one doesn’t already exist to define your data source.
  • Choose a visualization type (e.g., bar chart, line chart, or pie chart).
  • Configure your visualization, execute it, and explore your data.
  • Save your visualizations for future use.

Dashboards

  • Access Dashboards to create or view interactive dashboards:
  • Add visualizations, filters, or controls for a personalized experience.
  • Interact with your data directly from the dashboard.
  • Save dashboards for future use.

Management

  • Go to Management > Kibana to customize settings like default time zones and visualization preferences.

Limitations

Data Ingestion and Exploration

Integrate Manticore with tools like Logstash, Filebeat, Fluentbit, or Vector.dev to ingest data from sources like web logs. Once the data is loaded into Manticore, you can explore and visualize it in Kibana.

Last modified: June 19, 2026

Integration of Manticore with OpenSearch Dashboards

OpenSearch Dashboards is a visual interface that allows you to explore, visualize, and create dashboards for your log data. It connects to the backend over an HTTP API. Manticore exposes that API on its HTTP listener, so you can point OpenSearch Dashboards at Manticore and use Discover, Visualize, and Dashboards in a similar way to Kibana. With this integration, you can build charts and dashboards, run ad hoc searches in Discover, and keep working with familiar ingestion tools such as Logstash and Filebeat to load log and event data for analysis.

Prerequisites

  1. Download OpenSearch Dashboards: Ensure you download an OpenSearch Dashboards version compatible with Manticore. Currently, version 3.4.0 is tested and recommended. Other versions may work but could introduce issues. Set kibana_version_string in Manticore to match your OpenSearch Dashboards version (see Configuration below).
  2. Verify Manticore: Ensure your Manticore instance is running and its HTTP API is reachable (default: http://localhost:9308).
  3. Manticore Buddy: Ensure Manticore Buddy is installed and running. OpenSearch Dashboards integration is implemented by Buddy's EmulateElastic plugin, which emulates the Elasticsearch-compatible HTTP API that Dashboards expects (the same code path as Kibana). Buddy starts automatically with searchd unless you disable it via buddy_path.

Configuration

  1. Open the OpenSearch Dashboards configuration file (opensearch_dashboards.yml; common paths include config/opensearch_dashboards.yml in the tarball layout or /etc/opensearch-dashboards/opensearch_dashboards.yml on some packages).
  2. Set the URL of your Manticore instance:

opensearch.hosts: ["http://localhost:9308"]

When running OpenSearch Dashboards in Docker, you can set the same value via the OPENSEARCH_HOSTS environment variable.

Since Manticore does not provide the OpenSearch Security plugin, you must disable the security dashboards plugin in OpenSearch Dashboards as well:

  • Docker: set DISABLE_SECURITY_DASHBOARDS_PLUGIN=true in the container environment.
  • Tarball install: stop OpenSearch Dashboards, then run: ./bin/opensearch-dashboards-plugin remove securityDashboards. After that, start OpenSearch Dashboards again.

See Disabling and enabling the Security plugin for details. 3. Start OpenSearch Dashboards and open it in your browser at http://localhost:5601. Replace localhost with your server's IP or hostname if necessary.

Note: Manticore must work in real-time mode to be integrated with OpenSearch Dashboards.

Manticore configuration example

Set kibana_version_string to the same version as your OpenSearch Dashboards install. OpenSearch Dashboards checks the backend version reported by Manticore and may show warnings or fail to start if they do not match.

searchd {
    listen = 127.0.0.1:9308:http
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    kibana_version_string = 3.4.0
}

Supported Features

Discover

  • Use the Discover tab to search and filter documents in Manticore tables interactively.

Visualizations

  • Navigate to Visualizations to create custom visualizations:
  • Create an index pattern that matches a Manticore table name.
  • Choose a visualization type (e.g., bar chart, line chart, or pie chart) backed by supported aggregations: terms, histogram, date_histogram, range, date_range, and metric aggregations max, min, sum, avg.
  • Configure your visualization, execute it, and explore your data.
  • Save visualizations for future use.

Dashboards

  • Access Dashboards to create or view interactive dashboards:
  • Add saved visualizations, filters, or controls for a personalized experience.
  • Interact with your data directly from the dashboard.
  • Save dashboards for future use.

Management

  • Use Management > Dashboards Management for index patterns and saved objects (visualizations, dashboards).
  • Stack-level requests needed for Dashboards startup are emulated (node version, cluster settings, config objects, index listing). Full OpenSearch cluster administration is not available against Manticore.

Limitations

Data Ingestion and Exploration

Integrate Manticore with tools like Logstash, Filebeat, Fluentbit, or Vector.dev to ingest data from sources like web logs. Once the data is loaded into Manticore, you can explore and visualize it in OpenSearch Dashboards.

Last modified: June 19, 2026

Syncing from Kafka

NOTE: this functionality requires Manticore Buddy. If it doesn't work, make sure Buddy is installed.

Manticore supports integration with Apache Kafka real-time data ingestion through Kafka sources and materialized views, allowing for real-time data indexing and search. Currently, apache/kafka versions 3.7.0-4.1.0 are tested and supported.

To get started, you need to:

  1. Define the source: Specify the Kafka topic from which Manticore Search will read messages. This setup includes details like the broker’s host, port, and topic name.
  2. Set up the destination table: Choose a Manticore real-time table to store the incoming Kafka data.
  3. Create a materialized view: Set up a materialized view (mv) to handle data transformation and mapping from Kafka to the destination table in Manticore Search. Here, you’ll define field mappings, data transformations, and any filters or conditions for the incoming data stream.

Source

The source configuration allows you to define the broker, topic list, consumer group, and the message structure.

Schema

Define the schema using Manticore field types like int, float, text, json, etc.

CREATE SOURCE <source name> [(column type, ...)] [source_options]

All schema keys are case-insensitive, meaning Products, products, and PrOdUcTs are treated the same. They are all converted to lowercase.

If your field names don't match the table, field, and attribute name syntax allowed in Manticore Search (for example, if they contain special characters), you must define a schema mapping. For instance, $keyName is a valid key in JSON but not a valid unquoted field name in Manticore Search. If you try to use invalid field names without proper mapping, Manticore will return an error and the source creation will fail.

To handle such cases, use the following schema syntax to map invalid field names to valid ones:

allowed_field_name 'original JSON key name with special symbols' type

For example:

price_field '$price' float    -- maps JSON key '$price' to field 'price_field'
field_123 '123field' text     -- maps JSON key '123field' to field 'field_123'

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

CREATE SOURCE kafka
(id bigint, term text, abbrev '$abbrev' text, GlossDef json)
type='kafka'
broker_list='kafka:9092'
topic_list='my-data'
consumer_group='manticore'
num_consumers='2'
batch=50

POST /sql?mode=raw -d "CREATE SOURCE kafka (id bigint, term text, abbrev '$abbrev' text, GlossDef json) type='kafka' broker_list='kafka:9092' topic_list='my-data' consumer_group='manticore' num_consumers='2' batch=50"

‹›

Response

Query OK, 2 rows affected (0.02 sec)

[
  {
    "total": 2,
    "error": "",
    "warning": ""
  }
]

Options

Option Accepted Values Description
type kafka Sets the source type. Currently, only kafka is supported
broker_list host:port [, ...] Specifies Kafka broker URLs
topic_list string [, ...] Lists Kafka topics to consume from
consumer_group string Defines the Kafka consumer group, defaulting to manticore.
num_consumers int Number of consumers to handle messages.
partition_list int [, ...] List of partitions for reading more.
batch int Number of messages to process before moving on. Default is 100; processes remaining messages on timeout otherwise

Destination table

The destination table is a regular real-time table where the results of Kafka message processing are stored. This table should be defined to match the schema requirements of the incoming data and optimized for the query performance needs of your application. Read more about creating real-time tables here.

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

CREATE TABLE destination_kafka
(id bigint, name text, short_name text, received_at text, size multi);

POST /sql?mode=raw -d "CREATE TABLE destination_kafka (id bigint, name text, short_name text, received_at text, size multi)"

‹›

Response

Query OK, 0 rows affected (0.02 sec)

[
  {
    "total": 0,
    "error": "",
    "warning": ""
  }
]

Materialized view

A materialized view enables data transformation from Kafka messages. You can rename fields, apply Manticore Search functions, and perform sorting, grouping, and other data operations.

A materialized view acts as a query that moves data from the Kafka source to the destination table, letting you use Manticore Search syntax to customize these queries. Make sure that fields in the select match those in the source.

CREATE MATERIALIZED VIEW <materialized view name>
TO <destination table name> AS
SELECT [column|function [as <new name>], ...] FROM <source name>

‹›

  • SQL

SQL

📋

⚙

CREATE MATERIALIZED VIEW view_table
TO destination_kafka AS
SELECT id, term as name, abbrev as short_name,
       UTC_TIMESTAMP() as received_at, GlossDef.size as size FROM kafka

‹›

Response

Query OK, 2 rows affected (0.02 sec)

Data is transferred from Kafka to Manticore Search in batches, which are cleared after each run. For calculations across batches, such as AVG, use caution, as these may not work as expected due to batch-by-batch processing.

Field Mapping

Here's a mapping table based on the examples above:

Kafka Source Buffer MV Destination
id id id id id
term term term term as name name
unnecessary_key which we're not interested in - -
$abbrev abbrev abbrev abbrev as short_name short_name
- - - UTC_TIMESTAMP() as received_at received_at
GlossDef glossdef glossdef glossdef.size as size size

Listing

To view sources and materialized views in Manticore Search, use these commands:

  • SHOW SOURCES: Lists all configured sources.
  • SHOW MVS: Lists all materialized views.
  • SHOW MV view_table: Shows detailed information on a specific materialized view.

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

SHOW SOURCES

POST /sql?mode=raw -d "SHOW SOURCES"

‹›

Response

+-------+
| name  |
+-------+
| kafka |
+-------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "name": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "name": "kafka"
      }
    ]
  }
]

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

SHOW SOURCE kafka;

POST /sql?mode=raw -d "SHOW SOURCE kafka"

‹›

Response

+--------+-------------------------------------------------------------------+
| Source | Create Table                                                      |
+--------+-------------------------------------------------------------------+
| kafka  | CREATE SOURCE kafka                                               |
|        | (id bigint, term text, abbrev '$abbrev' text, GlossDef json)      |
|        | type='kafka'                                                      |
|        | broker_list='kafka:9092'                                          |
|        | topic_list='my-data'                                              |
|        | consumer_group='manticore'                                        |
|        | num_consumers='2'                                                 |
|        | batch=50                                                          |
+--------+-------------------------------------------------------------------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "Source": {
          "type": "string"
        }
      },
      {
        "Create Table": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "Source": "kafka",
        "Create Table": "CREATE SOURCE kafka \n(id bigint, term text, abbrev '' text, GlossDef json)\ntype='kafka'\nbroker_list='kafka:9092'\ntopic_list='my-data'\nconsumer_group='manticore'\nnum_consumers='2'\n batch=50"
      }
    ]
  }
]

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

SHOW MVS

POST /sql?mode=raw -d "SHOW MVS"

‹›

Response

+------------+
| name       |
+------------+
| view_table |
+------------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "name": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "name": "view_table"
      }
    ]
  }
]

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

SHOW MV view_table

POST /sql?mode=raw -d "SHOW MV view_table"

‹›

Response

+------------+--------------------------------------------------------------------------------------------------------+-----------+
| View       | Create Table                                                                                           | suspended |
+------------+--------------------------------------------------------------------------------------------------------+-----------+
| view_table | CREATE MATERIALIZED VIEW view_table TO destination_kafka AS                                            | 0         |
|            | SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size   |           |
|            | FROM kafka                                                                                             |           |
+------------+--------------------------------------------------------------------------------------------------------+-----------+

[
  {
    "total": 1,
    "error": "",
    "warning": "",
    "columns": [
      {
        "View": {
          "type": "string"
        }
      },
      {
        "Create Table": {
          "type": "string"
        }
      },
      {
        "suspended": {
          "type": "string"
        }
      }
    ],
    "data": [
      {
        "View": "view_table",
        "Create Table": "CREATE MATERIALIZED VIEW view_table TO destination_kafka AS SELECT id, term as name, abbrev as short_name, UTC_TIMESTAMP() as received_at, GlossDef.size as size FROM kafka",
        "suspended": 0
      }
    ]
  }
]

Altering materialized views

You can suspend data consumption by altering materialized views.

If you remove the source without deleting the MV, it automatically suspends. After recreating the source, unsuspend the MV manually using the ALTER command.

Currently, only materialized views can be altered. To change source parameters, drop and recreate the source.

‹›

  • SQL
  • JSON

SQL JSON

📋

⚙

ALTER MATERIALIZED VIEW view_table suspended=1

POST /sql?mode=raw -d "ALTER MATERIALIZED VIEW view_table suspended=1"

‹›

Response

Query OK (0.02 sec)

[
  {
    "total": 2,
    "error": "",
    "warning": ""
  }
]

Sharding with Kafka

You can also specify a partition_list for each Kafka topic. One of the main benefits of this approach is the ability to implement sharding for your table via Kafka. To achieve this, you should create a separate chain of source → materialized view → destination table for each shard:

Sources:

CREATE SOURCE kafka_p1 (id bigint, term text)
  type='kafka' broker_list='kafka:9092' topic_list='my-data'
  consumer_group='manticore' num_consumers='1' partition_list='0' batch=50;
CREATE SOURCE kafka_p2 (id bigint, term text)
  type='kafka' broker_list='kafka:9092' topic_list='my-data'
  consumer_group='manticore' num_consumers='1' partition_list='1' batch=50;

Destination Tables:

CREATE TABLE destination_shard_1 (id bigint, name text);
CREATE TABLE destination_shard_2 (id bigint, name text);

Materialized Views:

CREATE MATERIALIZED VIEW mv_1 TO destination_shard_1 AS SELECT id, term AS name FROM kafka_p1;
CREATE MATERIALIZED VIEW mv_2 TO destination_shard_2 AS SELECT id, term AS name FROM kafka_p2;

⚠️ Important Notes:

  • In this setup, rebalancing must be managed manually.
  • Kafka does not distribute messages using a round-robin strategy by default.
  • To achieve round-robin-like distribution when sending data, make sure your Kafka producer is configured with:
  • parse.key=true
  • key.separator={your_delimiter}

Otherwise, Kafka will distribute messages based on its own internal rules, which may lead to uneven partitioning.

Troubleshooting

Duplicate entries

Kafka offsets commit after each batch or when processing times out. If the process stops unexpectedly during a materialized view query, you may see duplicate entries. To avoid this, include an id field in your schema, allowing Manticore Search to prevent duplicates in the table.

How it works internally

  • Worker initialization: After configuring a source and materialized view, Manticore Search sets up a dedicated worker to handle data ingestion from Kafka.
  • Message mapping: Messages are mapped according to the source configuration schema, transforming them into a structured format.
  • Batching: Messages are grouped into batches for efficient processing. Batch size can be adjusted to suit your performance and latency needs.
  • Buffering: Mapped data batches are stored in a buffer table for efficient bulk operations.
  • Materialized view processing: The view logic is applied to data in the buffer table, performing any transformations or filtering.
  • Data transfer: Processed data is then transferred to the destination real-time table.
  • Cleanup: The buffer table is cleared after each batch, ensuring it’s ready for the next set of data.

Last modified: August 27, 2026