Ultravox - OSS Speech LM

Ultravox is a new multimodal LLM that is able to directly understand speech (unlike current voice AI stacks, it does not require a separate speech recognition stage). This approach makes voice AI applications faster, more robust, and allows them to understand the non-textual parts of speech.

It builds on a Llama 3 backbone which means that it can be trained much faster than a typical foundation model. We’ve just open-sourced Ultravox at https://ultravox.ai and are working on growing a community around it.