LLM

LLM

The LLM pipeline runs prompts through a large language model (LLM). This pipeline autodetects the LLM framework based on the model path.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import LLM

## Create and run LLM pipeline
llm = LLM()
llm(
  """
  Answer the following question using the provided context.

  Question:
  What are the applications of txtai?

  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
)

The LLM pipeline automatically detects the underlying LLM framework. This can also be manually set.

from txtai.pipeline import LLM

## Set method as litellm
llm = LLM("vllm/Open-Orca/Mistral-7B-OpenOrca", method="litellm")

## Set method as llama.cpp
llm = LLM("TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
           method="llama.cpp")

Models can be externally loaded and passed to pipelines. This is useful for models that are not yet supported by Transformers and/or need special initialization.

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from txtai.pipeline import LLM

## Load Mistral-7B-OpenOrca
path = "Open-Orca/Mistral-7B-OpenOrca"
model = AutoModelForCausalLM.from_pretrained(
  path,
  torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(path)

llm = LLM((model, tokenizer))

See the links below for more detailed examples.

Notebook| Description|
---|---|---
Prompt-driven search with LLMs| Embeddings-guided and Prompt-driven search with Large Language Models (LLMs)|
Prompt templates and task chains| Build model prompts and connect tasks together with workflows|
Build RAG pipelines with txtai| Guide on retrieval augmented generation including how to create citations|
Integrate LLM frameworks| Integrate llama.cpp, LiteLLM and custom generation frameworks|
Generate knowledge with Semantic Graphs and RAG| Knowledge exploration and discovery with Semantic Graphs and RAG|

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

## Create pipeline using lower case class name
## Use `generator` or `sequences` to force model type
llm:

## Run pipeline with workflow
workflow:
  llm:
    tasks:
      - action: llm

Similar to the Python example above, the underlying Hugging Face pipeline parameters and model parameters can be set in pipeline configuration.

llm:
  path: Open-Orca/Mistral-7B-OpenOrca
  torch_dtype: torch.bfloat16

Run with Workflows

from txtai.app import Application

## Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("llm", [
  """
  Answer the following question using the provided context.

  Question:
  What are the applications of txtai?

  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"sequences", "elements": ["Answer the following question..."]}'

Methods

Python documentation for the pipeline.

Creates a new LLM.

Parameters:

Source code in txtai/pipeline/llm/llm.py

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
``` |  ```
def init(self, path=None, method=None, kwargs):
    “””
    Creates a new LLM.

    Args:
        path: model path
        method: llm model framework, infers from path if not provided
        kwargs: model keyword arguments
    “””

    # Default LLM if not provided
    path = path if path else “google/flan-t5-base”

    # Generation instance
    self.generator = GenerationFactory.create(path, method, kwargs)

---|---

Generates text using input text

Parameters:

Returns:

Source code in txtai/pipeline/llm/llm.py

36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
``` |  ```
def call(self, text, maxlength=512, kwargs):
    “””
    Generates text using input text

    Args:
        text: text|list
        maxlength: maximum sequence length
        kwargs: additional generation keyword arguments

    Returns:
        generated text
    “””

    # Run LLM generation
    return self.generator(text, maxlength, kwargs)

---|---