3 Methodology

本文为论文《Search-o1：让推理模型学会边思考边检索》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

3.1 Problem Formulation

We consider a complex reasoning task that necessitates multi-step reasoning and the retrieval of external knowledge to derive solutions. The objective is to generate a comprehensive solution for each question $q$, consisting of both a logical reasoning chain $\mathcal{R}$ and the final answer $a$. In this work, we enable the reasoning model to utilize external knowledge sources during the reasoning process. Specifically, we consider three primary inputs in the problem-solving process: the task instruction $I$, the question $q$, and externally retrieved documents $\mathcal{D}$. Here, $I$ provides an overarching description of the reasoning task, $q$ is the specific complex question to be answered, and $\mathcal{D}$ comprises background knowledge dynamically retrieved from a relevant knowledge base.

The goal is to design a reasoning mechanism that effectively integrates $I$, $q$, and $\mathcal{D}$ to produce a coherent reasoning chain $\mathcal{R}$ and a final answer $a$. This can be formalized as the mapping $(I,q,\mathcal{D})\rightarrow(\mathcal{R},a)$. The generation of the reasoning sequence and the final answer can be expressed as:

| | $P(\mathcal{R},a\mid I,q,\mathcal{D})=\underbrace{\prod_{t=1}^{T_{r}}P(\mathcal{R}{t}\mid\mathcal{R}{where $T_{r}$ is the number of tokens in the reasoning sequence $\mathcal{R}$. The token at the position $t$ is $\mathcal{R}{t}$, and $\mathcal{R}{

3.2 Overview of the Search-o1 Framework

The Search-o1 framework addresses knowledge insufficiency in large reasoning models (LRMs) by seamlessly integrating external knowledge retrieval into their reasoning process while maintaining chain-of-thought coherence. As illustrated in Figure 2, we present a comparative analysis of three approaches: vanilla reasoning, agentic retrieval-augmented generation (RAG), and our proposed Search-o1 framework.

• •

Vanilla Reasoning Pattern: Consider the example in Figure 2(a), where the task involves determining the carbon atom count in the final product of a three-step chemical reaction. The vanilla reasoning approach falters when encountering knowledge gaps (e.g., the “structure of trans-Cinnamaldehyde”). Without access to accurate information, the model must rely on assumptions, potentially leading to cascading errors throughout subsequent reasoning steps.

• •

Agentic RAG: To bridge the knowledge gaps during reasoning, we build the agentic RAG mechanism (Figure 2(b)) to enable the model to autonomously retrieve external knowledge when needed. When uncertainty arises—such as regarding the compound’s structure—the model generates targeted search queries (e.g., “structure of trans-Cinnamaldehyde”). However, the direct insertion of retrieved documents, which often contain lengthy and irrelevant information, may disrupt the reasoning flow and hurt coherence.

• •

Search-o1: Our Search-o1 framework (Figure 2(c)) extends the agentic RAG mechanism by incorporating a Reason-in-Documents module. This module condenses retrieved documents into focused reasoning steps that integrate external knowledge while maintaining the logical flow of the reasoning chain. It considers the current search query, retrieved documents, and the existing reasoning chain to generate coherent steps. This iterative process continues until the final answer is reached. The following sections provide detailed explanations of agentic RAG, Reason-in-Documents, and the Search-o1 inference process.

3.3 Agentic Retrieval-Augmented Generation Mechanism

The agentic RAG mechanism is a pivotal component of the Search-o1 framework, empowering the reasoning model to autonomously determine when to retrieve external knowledge during the reasoning process. This mechanism allows the model itself to decide whether to continue generating reasoning steps or to initiate a retrieval step. Detailed model instructions can be found in Appendix A.1.

During the generation of the reasoning chain $\mathcal{R}$, the model may intermittently generate search queries $q_{\text{search}}^{(i)}$ encapsulated between special symbols <|begin_search_query|> and <|end_search_query|>, where $i$ indexes the $i$-th search step. Each search query is generated based on the current state of the reasoning process and the previously retrieved knowledge. The generation of each search query is expressed as:

| | $P(q_{\text{search}}^{(i)}\mid I,q,\mathcal{R}^{(i-1)})=\prod_{t=1}^{T_{q}^{(i)}}P\left(q_{\text{search},t}^{(i)}\mid q_{\text{search},where $T_{q}^{(i)}$ is the length of the $i$-th search query, $q_{\text{search},t}^{(i)}$ denotes the token generated at step $t$ of the $i$-th search query, and $\mathcal{R}^{(i-1)}$ represents all the reasoning steps prior to the $i$-th search step, including both search queries and search results.

Once a new pair of special symbols for the search query is detected in the reasoning sequence, we pause the reasoning process, and the search query $q_{\text{search}}^{(i)}$ is extracted. The retrieval function Search is invoked to obtain relevant documents:

| | $\mathcal{D}^{(i)}=\texttt{Search}(q_{\text{search}}^{(i)}),$ | | (3) |

where $\mathcal{D}^{(i)}={d_{1}^{(i)},d_{2}^{(i)},\ldots,d_{k_{i}}^{(i)}}$ represents the set of top-$k_{i}$ relevant documents retrieved for the $i$-th search query. The retrieved documents $\mathcal{D}^{(i)}$ are subsequently injected into the reasoning chain $\mathcal{R}^{(i-1)}$ between the special symbols <|begin_search_result|> and <|end_search_result|>, allowing the reasoning model to utilize the external knowledge to continue the reasoning process.

This agentic mechanism enables the model to dynamically and efficiently incorporate external knowledge, maintaining the coherence and relevance of the reasoning process while avoiding information overload from excessive or irrelevant retrieval results.

3.4 Knowledge Refinement via Reason-in-Documents

While the agentic RAG mechanism addresses knowledge gaps in reasoning, directly inserting full documents can disrupt coherence due to their length and redundancy. To overcome this, the Search-o1 framework includes the knowledge refinement module, which selectively integrates only relevant and concise information into the reasoning chain through a separate generation process using the original reasoning model. This module processes retrieved documents to align with the model’s specific reasoning needs, transforming raw information into refined, pertinent knowledge while maintaining coherence and logical consistency of the main reasoning chain.

The refinement guidelines for Reason-in-Documents are detailed in Appendix A.1. These guidelines instruct the model to analyze the retrieved web pages based on the previous reasoning steps, current search query, and the content of the searched web pages. The objective is to extract relevant and accurate information that directly contributes to advancing the reasoning process for the original question, ensuring seamless integration into the existing reasoning chain.

For each search step $i$, let $\mathcal{R}^{(where $T_{d}^{(i)}$ is the length of the intermediate reasoning sequence, and ${r}{\text{docs},t}^{(i)}$ denotes the token at step $t$. The refined knowledge ${r}{\text{final}}^{(i)}$ is then generated based on this analysis:

| | $P({r}{\text{final}}^{(i)}\mid{r}{\text{docs}}^{(i)},\mathcal{R}^{(where $T_{r}^{(i)}$ is the length of the refined knowledge sequence, and ${r}{\text{final},t}^{(i)}$ denotes the token at step $t$. The refined knowledge ${r}{\text{final}}^{(i)}$ is then incorporated into the reasoning chain $\mathcal{R}^{(i)}$, enabling the model to continue generating coherent reasoning steps with access to the external knowledge.

| | $P(\mathcal{R},a\mid I,q)=\prod_{t=1}^{T_{r}}P\left(\mathcal{R}{t}\mid\mathcal{R}{where ${r_{\text{final}}^{(j)}}_{j\leq i(t)}$ denotes all previously refined knowledge up to the $i(t)$-th search step. Here, $i(t)$ represents the index of the search step corresponding to the current reasoning step $t$. This refined knowledge integration ensures that each reasoning step can access relevant external information while maintaining the conciseness and focus of the reasoning process.

Algorithm 1 Search-o1 Inference

3.5 Search-o1 Inference Process

Inference Logic for a Single Question.

For each question, the Search-o1 inference begins by initializing the reasoning sequence with the task instruction $I$ concatenated with the specific question $q$. As the reasoning model $\mathcal{M}$ generates the reasoning chain $\mathcal{R}$, it may produce search queries encapsulated within the special symbols <|begin_search_query|> and <|end_search_query|>. Upon detecting the <|end_search_query|> symbol, the corresponding search query $q_{\text{search}}$ is extracted, triggering the retrieval function Search to obtain relevant external documents $\mathcal{D}$. These retrieved documents, along with the reason-in-documents instruction $I_{\text{docs}}$ and the current reasoning sequence $\mathcal{R}$, are then processed by the Reason-in-Documents module. This module refines the raw documents into concise, pertinent information $r_{\text{final}}$, which is seamlessly integrated back into the reasoning chain $\mathcal{R}$ within symbols <|begin_search_result|> and <|end_search_result|>. This iterative process ensures that the reasoning model incorporates necessary external knowledge while maintaining coherence and logical consistency, leading to the generation of a comprehensive reasoning chain $\mathcal{R}$ and the final answer $a$.

Batch Inference Mechanism.

To efficiently handle multiple questions simultaneously, the Search-o1 framework employs a batch inference mechanism that optimizes both token generation and knowledge refinement. Initially, a set of unfinished reasoning sequences $\mathcal{S}$ is created by concatenating the task instruction $I$ with each question $q$ in the batch $\mathcal{Q}$. The reasoning model $\mathcal{M}$ then generates tokens for all sequences in $\mathcal{S}$ in parallel, advancing each reasoning chain until it either completes or requires external knowledge retrieval. When a search query is identified within any sequence, the corresponding queries are extracted and processed in batches through the Search function to retrieve relevant documents $\mathcal{D}$. These documents are then collectively refined by the Reason-in-Documents module, which generates the refined knowledge $r_{\text{final}}$ for each sequence. The refined knowledge is subsequently inserted back into the respective reasoning chains. Completed sequences are moved to the finished set $\mathcal{F}$, while ongoing sequences remain in $\mathcal{S}$ for further processing. By leveraging parallel processing for both generation and refinement steps, the batch inference mechanism enhances system throughput associated with handling multiple inputs concurrently.