语音与多模态（3/4）

This guide covers how to use Azure AI Speech for Speech-to-Text with Open WebUI. Open WebUI sends each recording to Azure's fast transcription API, which returns the text in a single request.

Looking for TTS?

See the companion guide: Using Azure AI Speech for Text-to-Speech

Requirements

• An Azure Speech resource (Microsoft's documentation also calls it a Foundry resource for Speech) in a region that offers fast transcription. Not every region does: check the Fast transcription column of Microsoft's Speech service regions table.
• The resource's key, from Resource Management > Keys and Endpoint in the Azure portal. Either of the two keys works.
• Open WebUI installed and running

Azure Government

Microsoft lists fast transcription as unsupported in Azure Government, so this engine cannot be used there. See Speech service in sovereign clouds.

Quick Setup (UI)

1. Click your profile icon (bottom-left corner)
2. Select Admin Panel, then Settings. Settings opens in a window.
3. In the sidebar, choose Audio under Admin, in the Experience group. The other Audio entry, in the Preferences group, holds your personal audio settings. On a narrow screen these labels are hidden, so open the admin one directly at /?settings=admin:audio instead.
4. Configure the following:

1. Click Save

The key only works with the region the resource was created in. Microsoft notes that keys are region-scoped, so a key used with a different region fails to authenticate.

Language Locales

Azure works out which language is being spoken from the list of locales Open WebUI sends with each recording:

• When Language Locales is set, Azure uses those locales as its candidates. Microsoft notes that if none of them is in the audio, the service tries to identify the language itself.
• When it is left blank, Open WebUI sends this list: en-US, es-ES, es-MX, fr-FR, hi-IN, it-IT, de-DE, en-GB, en-IN, ja-JP, ko-KR, pt-BR and zh-CN.

Azure identifies one main language per recording. Microsoft notes that a smaller, accurate set of candidate locales can improve detection, so list only the languages your users actually speak. For the locales Azure supports, see Language and voice support.

Separate locales with commas only. Open WebUI does not trim spaces, so en-US, de-DE sends " de-DE" with a leading space.

The Language each user can set under Settings → Audio is not used by this engine. Only the admin's Language Locales list is sent to Azure.

Endpoint URL

By default, Open WebUI sends requests to https://<region>.api.cognitive.microsoft.com, built from Azure Region. Endpoint URL replaces that address, and Open WebUI adds /speechtotext/transcriptions:transcribe?api-version=2024-11-15 to it. Enter only the scheme and host, with no path.

You can use your resource's own endpoint from Keys and Endpoint, for example https://<resource-name>.cognitiveservices.azure.com. Microsoft's fast transcription examples call the same path on that address. Leave off the trailing / that Azure shows, because Open WebUI adds the path directly and would otherwise double the slash.

Max Speakers

Open WebUI always asks Azure to separate speakers (diarization), with Max Speakers as the upper limit. The text Open WebUI keeps is the combined transcript, without speaker labels.

Environment Variables Setup

If you prefer to configure via environment variables:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - AUDIO_STT_ENGINE=azure
      - AUDIO_STT_AZURE_API_KEY=your-speech-resource-key
      - AUDIO_STT_AZURE_REGION=westeurope
      - AUDIO_STT_AZURE_LOCALES=en-US,de-DE
# ... other configuration

All Azure STT Environment Variables

Settings saved in the UI take precedence

Saving the admin Audio settings stores every field on that tab, empty ones included, and choosing an engine saves too. With the default ENABLE_PERSISTENT_CONFIG=true, those stored values take precedence over the AUDIO_* environment variables from then on. See ENABLE_PERSISTENT_CONFIG.

Long Recordings

Before transcription, Open WebUI converts recordings in formats outside its supported list to MP3. Audio larger than 20 MB is re-encoded to a smaller MP3 and, if still too large, split into pieces of up to 20 MB. Each piece is sent to Azure as its own request, so the language is detected separately for each piece, and the texts are joined.

The slim image, or BYPASS_PYDUB_PREPROCESSING=true, skips this step and sends files whole. Open WebUI then rejects files over 200 MB, and Microsoft's reference for the API version Open WebUI uses (2024-11-15) limits audio to under 2 hours.

Using STT

1. Click the microphone icon in the chat input
2. Speak your message
3. Click the checkmark (Confirm recording) when you finish
4. Your speech is transcribed and appears in the message box, or is sent right away if Instant Auto-Send After Voice Transcription is on in your Audio settings

Troubleshooting

"Azure API key and region are required for Azure STT"

No API key is saved. Enter your Speech resource key in API Key (or set AUDIO_STT_AZURE_API_KEY) and save.

"External: Access denied due to invalid subscription key or wrong API endpoint"

Azure rejected the key. Check that:

1. The key is copied correctly from Keys and Endpoint
2. Azure Region is the region your Speech resource is in
3. Endpoint URL, if set, is your resource's endpoint with no path

"External: Resource not found"

Azure did not recognize the address. If Endpoint URL is set, make sure it has no path after the host.

Language could not be identified

When Azure cannot identify a language, or finds several with none dominant, Open WebUI shows Azure's own error message. Set Language Locales to the languages actually spoken in your recordings.

Other errors

Open WebUI shows Azure's own message when the audio is empty or too long, and for the two language identification errors. Other errors in Azure's transcription error format show "An error occurred during transcription.", and Azure's error code and message are written to the server log, in a line containing Azure STT error.

A few other messages come from Open WebUI itself:

• A message starting with "Failed to parse Azure response:": Azure answered, but without any transcript text.
• A message starting with "Error transcribing chunk:": the request never reached Azure, for example because the region identifier is mistyped.

If the key and region are correct and transcription still fails, confirm that your region offers fast transcription.

For more troubleshooting, see the Audio Troubleshooting Guide.

Cost Considerations

Azure bills Speech usage under your Azure subscription. Check Azure's Speech pricing page for current rates. Microsoft's quotas and limits list fast transcription limits for the Standard (S0) tier only.

tip

For free STT, use Local Whisper (the default) or the browser's Web API for basic transcription.

This guide covers how to use Azure AI Speech for Text-to-Speech with Open WebUI. Open WebUI sends text to Azure's text to speech REST API and plays back the audio it returns.

Looking for STT?

See the companion guide: Using Azure AI Speech for Speech-to-Text

Requirements

• An Azure Speech resource (Microsoft's documentation also calls it a Foundry resource for Speech). See Microsoft's Speech service regions table for text to speech availability by region.
• The resource's key, from Resource Management > Keys and Endpoint in the Azure portal. Either of the two keys works.
• Open WebUI installed and running

Quick Setup (UI)

1. Click your profile icon (bottom-left corner)
2. Select Admin Panel, then Settings. Settings opens in a window.
3. In the sidebar, choose Audio under Admin, in the Experience group. The other Audio entry, in the Preferences group, holds your personal audio settings. On a narrow screen these labels are hidden, so open the admin one directly at /?settings=admin:audio instead.
4. Configure the following:

1. Click Save. Choosing the engine already saves on its own and clears TTS Voice, which is why the voice comes after this step.
2. Switch to another settings tab and back to Audio. The voice list is fetched when the Audio tab opens, so it only loads with your key after this.
3. Configure the voice and format:

1. Click Save

The key only works with the region the resource was created in. Microsoft notes that keys are region-scoped, so a key used with a different region fails to authenticate.

Always set the region

If Azure Region and Endpoint URL are both blank, speech is still requested from eastus, but the voice list is not loaded at all. Set the region your resource is in.

Choosing Voices

Open WebUI loads the voice list for your region from Azure. Each row in TTS Voice shows the voice's full name, such as en-US-JennyNeural, with its display label next to it in grey. Only a few rows show at a time, so type part of a name or a language, such as en-GB, to narrow the list.

The full name is what Open WebUI sends to Azure, and it takes the speech language from the first two parts of that name (en-US from en-US-JennyNeural). If you type a voice instead of picking one, use the full name.

Which voice is used for a reply:

1. The model's own TTS Voice, if one is set when editing the model in Workspace → Models
2. Otherwise, the voice the user picked under Set Voice in their personal Audio settings, as long as the admin's TTS Voice is still the one that was set when they picked it. If the admin changes it, users hear the new default until they pick again.
3. Otherwise, the admin's TTS Voice

For the full list of voices and languages, see Microsoft's Language and voice support.

Endpoint URL

By default, Open WebUI sends requests to https://<region>.tts.speech.microsoft.com, built from Azure Region. Endpoint URL replaces that address. Open WebUI adds /cognitiveservices/v1 to it to request speech and /cognitiveservices/voices/list to load the voice list, so enter only the scheme and host, with no path.

This is not your resource's endpoint from Keys and Endpoint, the one the STT guide can use. Microsoft documents the voice list at /tts/cognitiveservices/voices/list on that address, while Open WebUI requests /cognitiveservices/voices/list, so the voice list would not load.

For the sovereign clouds, Microsoft documents these addresses, with <region> replaced by your region identifier:

See Speech service in sovereign clouds for the region identifiers.

Output Format

Output format is sent to Azure as the audio format to return. Keep one of the MP3 formats, such as the default audio-24khz-160kbitrate-mono-mp3. Open WebUI stores the audio it receives as an .mp3 file and, except in the slim image, serves it as MP3 whatever format Azure returned. The slim image refuses audio types outside its list of browser-playable formats, such as MP3, WAV and Ogg. Microsoft's audio outputs list has every value.

Environment Variables Setup

If you prefer to configure via environment variables:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - AUDIO_TTS_ENGINE=azure
      - AUDIO_TTS_API_KEY=your-speech-resource-key
      - AUDIO_TTS_AZURE_SPEECH_REGION=westeurope
      - AUDIO_TTS_VOICE=en-US-JennyNeural
# ... other configuration

Set AUDIO_TTS_VOICE to an Azure voice name. Its default, alloy, is an OpenAI voice.

All Azure TTS Environment Variables

Settings saved in the UI take precedence

Saving the admin Audio settings stores every field on that tab, empty ones included, and choosing an engine saves too. With the default ENABLE_PERSISTENT_CONFIG=true, those stored values take precedence over the AUDIO_* environment variables from then on. See ENABLE_PERSISTENT_CONFIG.

Testing TTS

1. Start a new chat
2. Send a message to any model
3. Click the speaker icon on the AI response to hear it read aloud

Troubleshooting

No voices shown in the list

1. Confirm API Key and Azure Region are set and saved
2. Switch to another settings tab and back to Audio to reload the list
3. Check the Open WebUI logs for Error fetching Azure voices

Speech fails with a 401 error

Azure rejected the key. Check that the key is copied correctly from Keys and Endpoint and that Azure Region is the region your Speech resource is in.

Speech fails with another error

1. Confirm TTS Voice is set to a full Azure voice name from the list
2. Confirm Output format is one of the MP3 values from Microsoft's audio outputs list
3. Check the Open WebUI logs for the full error

For broader audio debugging, see the Audio Troubleshooting Guide.

Cost Considerations

Azure bills Speech usage under your Azure subscription. Check Azure's Speech pricing page for current rates.

This guide covers how to use OpenAI's official Text-to-Speech API with Open WebUI. This is the simplest setup if you already have an OpenAI API key.

Looking for STT?

See the companion guide: Using OpenAI for Speech-to-Text

Requirements

• An OpenAI API key with access to the Audio API
• Open WebUI installed and running

Quick Setup (UI)

1. Click your profile icon (bottom-left corner)
2. Select Admin Panel
3. Click Settings → Audio tab
4. Configure the following:

1. Click Save

Available Models

Available Voices

OpenAI provides 6 built-in voices:

tip

Try different voices to find the one that best suits your use case. You can preview voices in OpenAI's documentation.

Per-Model TTS Voice

You can assign a specific TTS voice to individual models, allowing different AI personas to have distinct voices. This is configured in the Model Editor.

Setting a Model-Specific Voice

1. Go to Workspace > Models
2. Click the Edit (pencil) icon on the model you want to configure
3. Scroll down to find the TTS Voice field
4. Enter the voice name (e.g., alloy, echo, shimmer, onyx, nova, fable)
5. Click Save

Voice Priority

When playing TTS audio, Open WebUI uses the following priority:

1. Model-specific TTS voice (if set in Model Editor)
2. User's personal voice setting (if configured in user settings)
3. System default voice (configured by admin)

This allows admins to give each AI persona a consistent voice while still letting users override with their personal preference when no model-specific voice is set.

Use Cases

• Character personas: Give a "British Butler" model the fable voice, while an "Energetic Assistant" uses nova
• Language learning: Assign appropriate voices for different language tutors
• Accessibility: Set clearer voices for models designed for accessibility use cases

Environment Variables Setup

If you prefer to configure via environment variables:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - AUDIO_TTS_ENGINE=openai
      - AUDIO_TTS_OPENAI_API_BASE_URL=https://api.openai.com/v1
      - AUDIO_TTS_OPENAI_API_KEY=sk-...
      - AUDIO_TTS_MODEL=tts-1
      - AUDIO_TTS_VOICE=alloy
# ... other configuration

All TTS Environment Variables

Using OpenRouter as a Text-to-Speech Provider

Open WebUI's OpenAI TTS engine is compatible with any service that implements the OpenAI Audio API, including OpenRouter. This section explains how to configure OpenRouter as your TTS provider and how to avoid a common 400 Bad Request error.

Why the 400 Error Happens

OpenRouter's /audio/speech endpoint defaults response_format to pcm when the field is omitted. Open WebUI does not send response_format in its request payload, so OpenRouter returns raw PCM audio, which Open WebUI cannot play back, resulting in a 400 Bad Request.

The fix is to explicitly request MP3 by adding {"response_format": "mp3"} to the OpenAI Params field (extra parameters) in the TTS settings.

Quick Setup (UI)

1. Click your profile icon (bottom-left corner)
2. Select Admin Panel
3. Click Settings → Audio tab → Text-to-Speech Settings
4. Configure the following:

important

The OpenAI Params field must contain {"response_format": "mp3"}. Without it, OpenRouter defaults to pcm, and speech requests fail with a 400 Bad Request error.

1. Click Save

Environment Variables Setup

If you prefer to configure via environment variables, add the parameters JSON via AUDIO_TTS_OPENAI_PARAMS:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - AUDIO_TTS_ENGINE=openai
      - AUDIO_TTS_OPENAI_API_BASE_URL=https://openrouter.ai/api/v1
      - AUDIO_TTS_OPENAI_API_KEY=sk-or-...
      - AUDIO_TTS_MODEL=openai/tts-1
      - AUDIO_TTS_VOICE=alloy
      - AUDIO_TTS_OPENAI_PARAMS={"response_format":"mp3"}
# ... other configuration

info

OpenRouter supports mp3 and pcm output formats. Always select mp3 in Open WebUI, PCM output is intended for real-time streaming pipelines and cannot be played back by Open WebUI.

Testing TTS

1. Start a new chat
2. Send a message to any model
3. Click the speaker icon on the AI response to hear it read aloud

Response Splitting

When reading long responses, Open WebUI can split text into chunks before sending them to the TTS engine. This is configured in Settings > Admin > Audio under Response Splitting.

tip

Punctuation mode is recommended for most use cases. It provides the best balance of streaming performance (audio starts quickly) and natural speech pacing.

Troubleshooting

No Audio Plays

1. Check your OpenAI API key is valid and has Audio API access
2. Verify the API Base URL is correct (https://api.openai.com/v1)
3. Check browser console (F12) for errors

Audio Quality Issues

• Switch from tts-1 to tts-1-hd for higher quality
• Note: tts-1-hd has slightly higher latency

Rate Limits

OpenAI has rate limits on the Audio API. If you're hitting limits:

• Consider caching common phrases
• Use tts-1 instead of tts-1-hd (uses fewer tokens)

For more troubleshooting, see the Audio Troubleshooting Guide.

Cost Considerations

OpenAI charges per character for TTS. See OpenAI Pricing for current rates. Note that tts-1-hd costs more than tts-1.

info

For a free alternative, consider OpenAI Edge TTS which uses Microsoft's free Edge browser TTS.

Generate images from a prompt, or upload images and describe the changes you want, in an Open WebUI conversation. An administrator must first configure an image provider; keep your normal chat model selected while using it.

Image generation is only available in a chat when all of the following are true:

• Image Generation is enabled in Settings > Admin > Experience > Images.
• Your role has the Image Generation feature permission. Admins always pass this check.
• The model has the Image Generation capability checked in Workspace > Models > Edit. Outside Legacy mode its Builtin Tools capability and the Image Generation builtin tool category must stay enabled too. All of these are on by default.
• The Image toggle is on for that chat, in the Integrations menu of the message input. Tick Image Generation under Default Features in the model editor to have it active in every new chat.

If Image is missing from the Integrations menu entirely, then either image generation is off globally, the Image Generation capability is off on one of the selected models, or your role lacks the permission. The engine's own URL and key are a separate matter. Get those wrong and the toggle still appears, generation just fails once it runs.

Using Image Generation

1. Switch Image on in the Integrations menu of the message input.
2. Enter your image generation prompt.
3. Click Send.

Keep chatting with your chat model

The model selector at the top of the chat stays on your normal chat model. The image model is configured once in the admin image settings and is never selected there.

Picking the image model as the chat model is the most common setup mistake. Everything looks configured, and generation fails or returns nothing useful, because an image model cannot hold a conversation and is not being asked for a picture in the way it expects. If image generation has stopped working and the settings look right, check what the model selector says first.

What happens next depends on the Function Calling mode. It can be set per chat in Chat Controls > Advanced Params, for yourself in Settings > General > Advanced Parameters, per model in the model editor or globally under Model Defaults in the admin model settings. The first of those that is not Default wins, and Native applies when none of them is set.

Native takes the tool path described below. In Legacy mode every message you send while the toggle is on goes straight to the image engine, whatever you asked for. It edits instead of creating when the recent conversation carries images and Image Edit is on. On the create path a task model rewrites your message into the image prompt first, unless you switch Image Prompt Generation off in the admin settings.

Native Tool-Based Generation (Agentic)

With Native Function Calling (see the Central Tool Calling Guide), the model invokes image generation directly as a tool.

How it works:

• Natural Language: You can simply ask the model: "Generate an image of a cybernetic forest."
• Model Dependent: A model that does not call tools reliably will simply never call generate_image, so try a stronger model before suspecting your engine configuration.
• Display: The generated image is displayed directly in the chat interface.
• Editing: The edit_image tool (e.g., "Make the sky in this image red") is handed over as well, but only when Image Edit is enabled in the admin settings.

This approach allows the model to "reason" about the prompt before generating, or even generate multiple images as part of a complex request.

info

Legacy "Generate Image" Button: As of Open WebUI v0.7.0, the native "Generate Image" button (which allowed generating an image directly from a message's content) was removed. If you wish to restore this functionality, you can use the community-built Generate Image Action.

Restoring the "Generate Image" Button

If you prefer the workflow where you can click a button on any message to generate an image from its content, you can easily restore it:

1. Visit the Generate Image Action on the Open WebUI Community site.
2. Click Get to import it into your local instance (or copy the code and paste it into your local instance).
3. Once imported, go to Admin Panel > Functions and ensure the Generate Image action is enabled.

This action adds a "Generate Image" icon to the message action bar, allowing you to generate images directly from LLM responses, which is helpful if you want the assistant to first iterate on the image prompt and generate it once you are satisfied.

info

Requirement: Image editing runs through the same conditions listed at the top of this page. To use Image Editing or Image+Image Generation, also turn on Image Edit in the Edit Image section of Settings > Admin > Experience > Images, with an edit engine (Default (Open AI), ComfyUI or Gemini) and a model that supports inpainting or img2img.

Image Editing (Inpainting)

You can edit an image by providing the image and a text prompt directly in the chat.

1. Upload an image to the chat.
2. Enter a prompt describing the change you want to make (e.g., "Change the background to a sunset" or "Add a hat").
3. The model will generate a new version of the image based on your prompt.

Image Compositing (Multi-Image Fusion)

Seamlessly combine multiple images into a single cohesive scene, a process professionally known as Image Compositing or Multi-Image Fusion. This allows you to merge elements from different sources (e.g., placing a subject from one image into the background of another) while harmonizing lighting, perspective, and style.

1. Upload images to the chat (e.g., upload an image of a subject and an image of a background).
2. Enter a prompt describing the desired composition (e.g., "Combine these images to show the cat sitting on the park bench, ensuring consistent lighting").
3. The model will generate a new composite image that fuses the elements according to your instructions.