数据集与数据源（2/3）

of messages processed in each batch. - Sync deleted files: After this is enabled, messages deleted from the external system are removed from the knowledge base index.

Dingtalk AI Table

The Dingtalk AI Table data source is used to synchronize records in Dingtalk AI Table to a RAGFlow knowledge base. After configuration, online table records, business ledgers, and collaborative data can be queried.

Permission requirements: The Dingtalk account or application authorization must have read permissions for the target AI Table.

Account version requirements: Dingtalk organization accounts and AI Table permissions are required. Whether it can be synchronized depends on the organization permissions and application authorization.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Dingtalk AI Table connection.
Application key: Fill in the Dingtalk application key.
Application secret: Fill in the Dingtalk application secret.
Table ID: Fill in the AI Table ID that needs to be synchronized.
View ID: Fill in the view ID that needs to be synchronized.
Content field: Select the field to be written as document content.
Metadata field: Select the field to be written as metadata.
Sync deleted files: After this is enabled, deleted records are removed from the knowledge base index.

Zendesk

The Zendesk data source is used to synchronize Zendesk tickets, comments, and knowledge base content to a RAGFlow knowledge base. After configuration, customer service records and support knowledge can be queried in a unified way.

Permission requirements: The Zendesk account, API token, or application authorization must have read permissions for the target tickets and knowledge base content.

Account version requirements: Zendesk team or enterprise accounts can be used. The account must have access permissions for the target tickets or help center content.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Zendesk connection.
Subdomain: Fill in the Zendesk subdomain.
Email: Fill in the Zendesk account email.
API token: Fill in the Zendesk API token.
Synchronization scope: Select tickets, help center articles, or both.
Batch size: Set the number of records processed in each batch.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.

Moodle

The Moodle data source is used to synchronize course content, forums, resources, and learning records in Moodle to a RAGFlow knowledge base. After configuration, course materials and teaching records can be queried.

Permission requirements: The Moodle account or token must have read permissions for the target courses, activities, and resources.

Account version requirements: Self-built Moodle and organization Moodle platforms can be used. The account must have access permissions for the target courses.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Moodle connection.
Moodle site URL: Fill in the Moodle service address.
Access token: Fill in the Moodle access token.
Course ID: Fill in the course ID that needs to be synchronized.
Synchronization scope: Select course resources, forums, assignments, or other content.
Batch size: Set the number of records processed in each batch.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.

REST API

The REST API data source is used to synchronize data returned by custom business APIs to a RAGFlow knowledge base. After configuration, records in internal systems or third-party systems can enter the knowledge base according to the API structure.

Permission requirements: The API credentials must have permission to call the target API, and the data returned by the API must be readable.

Account version requirements: Personal or enterprise custom systems can both be used, provided that the target API is accessible and provides a stable data structure and authentication method.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this REST API connection.
Request URL: Fill in the API request address.
Request method: Select GET, POST, or another request method.
Request headers: Fill in the header information required for API authentication or requests.
Request body: Fill in the request body when POST or other methods require it.
Pagination method: Configure the pagination method to ensure that the next page of data can be continuously read.
Content field: Specify the content field returned by the API.
Metadata field: Specify the metadata field returned by the API.
Unique ID field: Specify the unique ID field of each record.
Sync deleted files: After this is enabled, deleted records are removed from the knowledge base index according to the synchronization or cleanup task.

Xquik

The Xquik data source searches public X posts and syncs each matching post into a RAGFlow knowledge base. Use it to retrieve posts by keyword, hashtag, author, language, or another supported X search operator.

Permission requirements: Create an Xquik API key. The key needs access to the Search Tweets API.

Usage requirements: Each returned post uses 1 Xquik credit. The page size and maximum page count bound each sync. Xquik is an independent third-party service.

Configuration parameters:

Name: Choose a name for this connection.
Xquik API key: Enter the key in the password field.
X search query: Enter keywords, hashtags, or operators such as from:username.
Result order: Select Latest for chronological results or Top for ranked results.
Posts per page: Set the maximum posts requested per API page.
Max pages: Stop each sync after this many pages.
Batch size: Set the number of posts sent to RAGFlow per batch.

RAGFlow sends each incremental sync window to Xquik as inclusive sinceTime and exclusive untilTime bounds. Cursor pagination continues until no page remains or the configured maximum is reached.

RSS

The RSS data source is used to subscribe public websites, blogs, announcements, or product updates to a RAGFlow knowledge base. After configuration, the knowledge base can continuously obtain new content from the subscription source, making it convenient for users to view and ask questions in a unified way.

Permission requirements: The subscription source must be accessible. If the subscription source is protected, make sure the access address or credentials have read permissions.

Account version requirements: Public subscription sources can usually be used directly. If the RSS address is protected, provide an account or access address that can access the subscription source.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this RSS connection.
Feed URL: Fill in the RSS or Atom subscription address on the page that needs to be synchronized.
Batch size: Set the number of subscription entries processed in each batch.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.

Sitemap

The Sitemap data source is used to synchronize the web pages listed in a public sitemap.xml to a RAGFlow knowledge base. It is the simplest way to index a documentation site, a corporate website, or a blog without writing a crawler: the site itself declares which pages exist and when they were last modified.

Permission requirements: The sitemap and the pages it lists must be reachable over HTTP or HTTPS from the RAGFlow server. No credentials are used. Every request is checked by the SSRF guard, so private or loopback addresses are rejected.

Account version requirements: None. Sitemaps are a public XML standard (sitemaps.org); both plain urlset files and sitemapindex files (followed recursively, up to 5 levels) are supported.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Sitemap connection.
Sitemap URL: The URL of the sitemap.xml or sitemap index to crawl, for example https://example.com/sitemap.xml.
URL filter (regex): Optional regular expression. Only URLs matching it are indexed, for example ^https://example\.com/docs/ to restrict the sync to one section of the site. The connection test fails when no URL matches. The pattern is limited to 512 characters, nested quantifiers such as (a+)+ are refused, and it is evaluated with a 1-second timeout against the first 4,096 characters of each URL.
Follow PDF links: When enabled, PDF files linked from the crawled HTML pages are indexed as well. Each PDF is indexed once even when several pages link to it.
Restrict PDFs to sitemap domain: When enabled (default), only PDF links hosted on the same domain as the sitemap are followed.
User-Agent: The User-Agent header sent with every request. Leave empty to use RAGFlow-SitemapConnector/1.0. Set it when the target site filters unknown crawlers.
Batch size: The number of pages fetched and sent to RAGFlow per batch.
Sync deleted files: After this is enabled, pages removed from the sitemap are removed from the knowledge base index.

Every request goes through the SSRF guard with the resolved address pinned for the duration of the request, response bodies are capped at 64 MB, and at most 1000 sitemap documents are fetched per sync (each sitemap URL once). HTML pages are converted to Markdown and stored as .md documents; during conversion, script, style, noscript, nav, header, footer, aside, form, iframe, and svg elements are removed. URLs served with Content-Type: application/pdf are stored as .pdf documents and processed by the regular PDF pipeline. Each document keeps the page URL, the sitemap URL, and, for discovered PDFs, the parent page URL in its metadata.

Incremental syncs rely on the <lastmod> element: only pages whose lastmod falls inside the sync window are fetched again, and pages without lastmod are only fetched by a full sync. Every document also carries a content fingerprint, so a page that is fetched again but has not changed is skipped instead of being re-indexed.

Feishu Wiki

The Feishu Wiki data source is used to synchronize downloadable file nodes from a specified Feishu Wiki space to a RAGFlow knowledge base. After configuration, RAGFlow can retrieve files under the specified root node and filter the files to be synchronized according to the configured conditions.

Permission requirements: The Feishu application must have permission to access the target Wiki space and read or download the files to be synchronized.

Account version requirements: A Feishu application with access to the target Wiki space is required. The accessible scope depends on the permissions granted to the application and the target Wiki space.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Feishu Wiki connection.
Feishu App ID: Fill in the App ID of the Feishu application.
Feishu App Secret: Fill in the App Secret of the Feishu application.
Knowledge base Space ID: Fill in the Space ID of the Feishu Wiki space to synchronize.
Wiki root node Token: Fill in the token of the root Wiki node from which synchronization starts.
Allowed file extensions: Specify the file extensions that are allowed to be synchronized.
File name must contain keywords: Specify keywords that must appear in the file name. Only matching files are synchronized.
File name exclusion keywords: Specify keywords used to exclude files from synchronization.
Maximum file size (bytes): Set the maximum size of a single file that can be synchronized. The default value is 52428800 bytes.
Batch size: Set the number of files processed in each batch. The default value is 2.

Azure DevOps

The Azure DevOps data source is used to synchronize source code and pull request content from Azure DevOps organizations, projects, or repositories to a RAGFlow knowledge base. The synchronization scope and content type can be configured as needed.

Permission requirements: The Azure DevOps personal access token must have read access to the target organization, project, or repository. Read access to pull requests is also required when pull request content is synchronized.

Account version requirements: An Azure DevOps account and a valid Personal Access Token (PAT) are required.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Azure DevOps connection.
Azure DevOps personal access token: Fill in the Personal Access Token used to access Azure DevOps.
Base URL: Fill in the Azure DevOps service URL. The default value is https://dev.azure.com.
Azure DevOps organization: Fill in the Azure DevOps organization to access.
Index mode: Select the synchronization scope. The options are Organization, Project, and Repository.
Organization: Index all repositories visible to the token in the specified organization.
Project: Index content in a specified project. When selected, fill in the Project field.
Repository: Index a specified repository. When selected, fill in the Repository field.
Project: Fill this in when the index mode is Project.
Repository: Fill this in when the index mode is Repository.
Content type: Select the content to synchronize. The options are Code, Pull requests, and Both.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.

Xquik

The Xquik data source is used to search posts on X through Xquik and synchronize matching posts to a RAGFlow knowledge base. Search conditions, result ordering, and pagination settings can be configured to control the synchronized content.

Permission requirements: A valid Xquik API key with permission to access the corresponding search API is required.

Account version requirements: An account with access to the Xquik API is required. Available data and API usage limits depend on the Xquik service and account configuration.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Xquik connection.
Xquik API key: Fill in the API key used to access Xquik.
X search query: Fill in the query used to search posts on X, for example, ragflow lang:en.
Result order: Select the order in which search results are returned, such as Latest.
Posts per page: Set the number of posts retrieved per page. The default value is 100.
Maximum pages: Set the maximum number of result pages retrieved during synchronization. The default value is 10.
Batch size: Set the number of posts processed in each batch. The default value is 32.

WebDAV

The WebDAV data source is used to connect to a WebDAV server and synchronize files from a specified remote path to a RAGFlow knowledge base.

Permission requirements: The configured WebDAV account must have read permission for the target remote path.

Account version requirements: A file service that supports the WebDAV protocol is required, together with a valid server URL, username, and password.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this WebDAV connection.
WebDAV server URL: Fill in the WebDAV server URL, for example, https://webdav.example.com.
Username: Fill in the username used to access the WebDAV server.
Password: Fill in the password for the WebDAV account.
Remote path: Specify the remote directory to synchronize. The default value is /.
Custom CA Certificate Path: Optional. Fill in the path to a custom CA certificate when required, for example, /etc/ssl/certs/webdav-ca.pem.
Sync deleted files: After this is enabled, files deleted from the external system are removed from the knowledge base index.

Airtable

The Airtable data source is used to synchronize records from a specified table in an Airtable base to a RAGFlow knowledge base.

Permission requirements: The access token must have read permission for the target Airtable base and table.

Account version requirements: An Airtable account with access to the target base and a valid access token is required.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Airtable connection.
Access token: Fill in the access token used to access the Airtable API.
Base ID: Fill in the ID of the target Airtable base.
Table name or ID: Fill in the name or ID of the table to synchronize.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.

Salesforce

The Salesforce data source is used to connect to Salesforce and synchronize records from specified Salesforce objects to a RAGFlow knowledge base. Supported objects can include accounts, contacts, opportunities, cases, and knowledge content.

Permission requirements: The Salesforce application and account must have API access and read permission for the specified Salesforce objects.

Account version requirements: A Salesforce environment with API access and valid client credentials is required.

Configuration parameters:

Name: Customize the name in RAGFlow to identify this Salesforce connection.
Instance URL: Fill in the Salesforce instance URL, for example, https://your-domain.my.salesforce.com.
Client ID: Fill in the client ID used to connect to Salesforce.
Client secret: Fill in the corresponding client secret.
Objects: Specify the Salesforce objects to synchronize. Multiple objects can be specified, for example, Account,Contact,Opportunity,Case,Knowledge__kav.
API version: Set the Salesforce API version to use. The default value in the current UI is v59.0.
Batch size: Set the number of records processed in each batch. The default value is 2.
Sync deleted files: After this is enabled, content deleted from the external system is removed from the knowledge base index.