Text To Speech

Text To Speech

The Text To Speech pipeline generates speech from text.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import TextToSpeech

## Create and run pipeline with default model
tts = TextToSpeech()
tts("Say something here")

## Stream audio - incrementally generates snippets of audio
yield from tts(
  "Say something here. And say something else.".split(),
  stream=True
)

## Generate audio using a speaker id
tts = TextToSpeech("neuml/vctk-vits-onnx")
tts("Say something here", speaker=15)

## Generate audio using speaker embeddings
tts = TextToSpeech("neuml/txtai-speecht5-onnx")
tts("Say something here", speaker=np.array(...))

See the links below for a more detailed example.

Notebook| Description|
---|---|---
Text to speech generation| Generate speech from text|
Speech to Speech RAG ▶️| Full cycle speech to speech workflow with RAG|
Generative Audio| Storytelling with generative audio workflows|

This pipeline is backed by ONNX models from the Hugging Face Hub. The following models are currently available.

• kokoro-base-onnx | fp16 | int8
• ljspeech-jets-onnx
• ljspeech-vits-onnx
• vctk-vits-onnx
• txtai-speecht5-onnx

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

## Create pipeline using lower case class name
texttospeech:

## Run pipeline with workflow
workflow:
  tts:
    tasks:
      - action: texttospeech

Run with Workflows

from txtai import Application

## Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("tts", ["Say something here"]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"tts", "elements":["Say something here"]}'

Methods

Python documentation for the pipeline.

__init__(path=None, maxtokens=512, rate=22050)

Creates a new TextToSpeech pipeline.

Parameters:

Source code in txtai/pipeline/audio/texttospeech.py

43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
``` |  ```
def init(self, path=None, maxtokens=512, rate=22050):
    “””
    Creates a new TextToSpeech pipeline.

    Args:
        path: optional model path
        maxtokens: maximum number of tokens model can process, defaults to 512
        rate: target sample rate, defaults to 22050
    “””

    if not TTS:
        raise ImportError(‘TextToSpeech pipeline is not available - install “pipeline” extra to enable’)

    # Default path
    path = path if path else “neuml/ljspeech-jets-onnx”

    # Target sample rate
    self.rate = rate

    # Load target tts pipeline
    self.pipeline = None
    if self.hasfile(path, “model.onnx”) and self.hasfile(path, “config.yaml”):
        self.pipeline = ESPnet(path, maxtokens, self.providers())
    elif self.hasfile(path, “model.onnx”) and self.hasfile(path, “voices.json”):
        self.pipeline = Kokoro(path, maxtokens, self.providers())
    else:
        self.pipeline = SpeechT5(path, maxtokens, self.providers())

---|---

__call__(text, stream=False, speaker=1, encoding=None, **kwargs)

Generates speech from text. Text longer than maxtokens will be batched and returned as a single waveform per text input.

This method supports text as a string or a list. If the input is a string, the return type is audio. If text is a list, the return type is a list.

Parameters:

Returns:

Source code in txtai/pipeline/audio/texttospeech.py

 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
``` |  ```
def call(self, text, stream=False, speaker=1, encoding=None, kwargs):
    “””
    Generates speech from text. Text longer than maxtokens will be batched and returned
    as a single waveform per text input.

    This method supports text as a string or a list. If the input is a string,
    the return type is audio. If text is a list, the return type is a list.

    Args:
        text: text|list
        stream: stream response if True, defaults to False
        speaker: speaker id, defaults to 1
        encoding: optional audio encoding format
        kwargs: additional keyword args

    Returns:
        list of (audio, sample rate) or list of audio depending on encoding parameter
    “””

    # Convert results to a list if necessary
    texts = [text] if isinstance(text, str) else text

    # Streaming response
    if stream:
        return self.stream(texts, speaker, encoding)

    # Transform text to speech
    results = [self.execute(x, speaker, encoding, kwargs) for x in texts]

    # Return results
    return results[0] if isinstance(text, str) else results

---|---