Automatic speech recognition, well-known as speech-to-text, has been around for quite a long time now, but the recent advances in both hardware and software, made the technology more accessible than ever before.
This technology is now used in many applications, virtual assistants, transcription services and voice controlled systems. For startups and developers looking to add speech to text functionality, open source voice recognition models are a cost effective and flexible solution. In this post we will look at some of the best open source speech to text software and compare their pros and cons to help you choose the one that’s right for you. At VocalJet, we chose to work with few of those open source models and we took the time to benchmark and study their strenght and weaknesses. Here is a synthetis of our discoveries.
1. Kaldi
Overview
Kaldi is a toolkit for speech recognition written in C++ and is one of the most popular open source voice recognition toolkits. Developed at Johns Hopkins University, it’s used in both academic and commercial environments because of its flexibility, scalability and performance. The Kaldi toolkit provides its algorithms in the most generic and modular form possible, to maximize flexibility and reusability
Pros
Highly Customizable: Kaldi has many configuration options, so you can tune the system to your specific needs. This is great for researchers and developers who want to try out different algorithms and models.
State of the Art Performance: Kaldi has state of the art performance in terms of accuracy and speed. It supports various acoustic models, including HMMs and DNNs.
Lots of Features: Kaldi has many features, including support for multiple languages, speaker diarization and robust training algorithms.
Active Community: Kaldi has an active community of developers and researchers so there are continuous improvements and updates. There’s also extensive documentation and tutorials which are super helpful for new users.
Cons
Steep Learning Curve: Kaldi’s flexibility comes at a cost, it has a steep learning curve and requires a lot of effort to master. This can be a barrier for beginners or those with limited technical skills.
Resource Hungry: Kaldi requires a lot of computational resources, especially for training large models. This can be a limitation for smaller organizations or individuals with limited access to high performance computing infrastructure.
2. DeepSpeech
Overview
DeepSpeech is an open source speech to text engine developed by Mozilla in 2017. It’s based on the homonymous Baidu’s Deep Speech research paper and uses deep learning to achieve high accuracy.
Pros
Easy to Use: DeepSpeech is designed to be easy to use, with simple installation and usage instructions.
Pre-built Models: Mozilla provides pre-built models that you can use out of the box, so you can add speech recognition to your application without training.
Community Support: DeepSpeech has an active community and is supported by Mozilla so there are continuous development and support. There’s also extensive documentation and community forums to help you.
Cross Platform: DeepSpeech is cross platform, Windows, macOS and Linux so it’s good for different development environments.
Weaknesses
Accuracy Variability: DeepSpeech works well with clean and well articulated speech but its accuracy degrades in noisy environment or accented speech. So it’s not suitable for some use cases.
Limited Language Support: DeepSpeech is mostly English, limited language support. While DeepSpeech performs well on clean, well-articulated speech, its accuracy can degrade in noisy environments or with accented speech. This makes it less reliable for certain use cases.
Recording limitations: Its recordings are limited to 10 seconds, limiting its use to applications such as command processing but no long transcriptions.
3. Vosk
Overview
Vosk is a lightweight, open-source speech recognition toolkit that focuses on real-time processing and supports many languages. It runs on many devices, including mobile phones and embedded systems such as Raspberry Pi, Android, or iOS. Vosk models are pretty small (50 Mb) but provide continuous large vocabulary transcription, zero-latency response with streaming API, reconfigurable vocabulary and speaker identification.
Vosk Speech recognition bindings implemented for various programming languages like Python, Java, Node.JS, C#, C++, Rust, Go and others.
Pros
Low Resource Requirements: Vosk is designed for low-resource environments and can run on CPUs without GPU.
Real-Time Processing: Vosk is real-time speech recognition with low latency and fast processing. Good for voice assistants and interactive systems.
Multi-Language: Vosk supports English, Spanish, French and Chinese. Good for global applications.
Easy to Integrate: Vosk has easy to use APIs and bindings for Python, Java and C++. Easy to integrate with different applications.
Cons
Accuracy: Vosk works well in many cases, but not as well as more resource hungry models like Kaldi or DeepSpeech. Not good for applications that require high accuracy.
No advanced features: Vosk only has basic speech recognition. No speaker diarization, no complex acoustic model.
4. Wav2Letter++
Overview
Wav2Letter++ is a fast open-source speech recognition system from Facebook’s AI Research (FAIR) lab. Wav2letter is an end-to-end Automatic Speech Recognition (ASR) system for researchers and developers to transcribe speech. It implements the architecture proposed in Wav2Letter: an End-to-End ConvNet-based Speech Recognition System and Letter Based Speech Recognition with Gated ConvNets. It provides pre-trained models for the Librispeech dataset to help developers start transcribing speech right away.
Pros
Fast: Wav2Letter++ is optimized for speed, low latency speech recognition. Good for real-time use cases where speed matters.
End-to-End: End-to-end neural network, no HMMs. Less pipeline, potentially more accurate.
Extensible: Wav2Letter++ is designed to be easy to extend, so you can try different neural network architectures and training methods.
GPU Acceleration: Wav2Letter++ supports GPU acceleration, so you can train and infer faster on compatible hardware.
Cons
Hard to Set up: Setting up Wav2Letter++ is tricky, requires deep learning framework and GPU knowledge. Not for beginners.
Resource Hungry: Despite being optimized for speed, Wav2Letter++ still requires a lot of resources, especially for training large models. Not for those with limited access to high-end hardware.
Limited Community: Compared to Kaldi, Wav2Letter++ has a smaller community and less resources for troubleshooting and support.
5. OpenSeq2Seq
Overview
OpenSeq2Seq is an open-source sequence-to-sequence toolkit from NVIDIA. OpenSeq2Seq is performance optimized for mixed-precision training using Tensor Cores on NVIDIA Volta GPUs. Supports many tasks: speech recognition, machine translation, text-to-speech synthesis.
Pros
Versatile: OpenSeq2Seq is a versatile toolkit for many sequence-to-sequence tasks. Good for developers who work on multiple applications.
Fast: With NVIDIA’s GPU acceleration, OpenSeq2Seq is fast for both training and inference. Good for large scale applications that require speed.
Pre-trained Models: NVIDIA provides pre-trained models for several tasks so you can try out different functionality without training from scratch.
Documentation: OpenSeq2Seq has extensive documentation and examples so you can get started and understand the different features and configurations.
Cons
Complex: OpenSeq2Seq’s versatility comes with complexity. Setting up and configuring the toolkit is tricky, especially for those who are not familiar with deep learning and GPU environments.
Resource Hungry: Like Wav2Letter++, OpenSeq2Seq requires a lot of resources, especially for training large models. Not good for small orgs or individuals with limited access to high-end hardware.
Not Speech Specific: While OpenSeq2Seq supports speech recognition, it may not have the speech-specific features and optimizations of speech-to-text models like Kaldi and DeepSpeech.
6. Whisper
Overview
Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. It has been developed by OpenAI. It’s designed for robust transcription audio to text, with advanced deep learning models. The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Discover what is OpenAI whisper in one of our previous article.
Pros
Accurate: Whisper is accurate in all conditions, noisy or not, and in all accents. Good for many use cases.
Language support: Whisper supports multiple languages. Multilingual transcription is a breeze.
Easy to Use: Whisper is easy to use with simple APIs and docs. Low learning curve, quick to integrate into your app.
Advanced Features: Automatic punctuation, speaker diarization, real-time processing. Good for complex scenarios.
Cons
Resource Hungry: Whisper is resource intensive, especially for training. High end hardware (GPUs) required to use it fully.
Not Customizable: Whisper is good out of the box but has limited customization options compared to Kaldi.
7. Wav2Vec
Summary
Wav2Vec is an advanced speech recognition model by Facebook AI Research (FAIR). It is an algorithm that uses raw, unlabeled audio to train automatic speech recognition (ASR) models. Wav2vec represents a step forward for ASR systems, and it’s a promising direction for recognizing speech in languages that do not have extensive datasets for training AI systems.
Pros
Self-Supervised: Wav2Vec uses self-supervised learning. Less labeled data required. Cost effective and easy to train.
Accurate: Wav2Vec is accurate on many benchmarks. As good as some commercial systems. Robust in many conditions.
Scalable: The model scales well with more data and more compute. Good for projects of all sizes.
Tunable: Wav2Vec can be fine tuned for specific tasks. Customizable performance.
Cons
Hard to Set up: Setting up and training Wav2Vec is complex. Requires deep learning knowledge and a lot of compute.
Resource Hungry: Like Whisper, Wav2Vec is resource intensive. High end hardware required for training.
Not Real-Time: Wav2Vec is accurate but not real-time optimized like Vosk.
Summary
Choose the right open source speech-to-text software for your needs. Kaldi is state of the art and highly customizable but has a high learning curve and resource requirements. DeepSpeech is easy to use and has pre-trained models but struggles with noisy and accented speech. Vosk is good for low resource environments and real-time but not as accurate as resource hungry models. Wav2Letter++ and OpenSeq2Seq are high performance and flexible but require a lot of compute and technical expertise. Whisper and Wav2Vec have advanced features and high accuracy but need a lot of compute and deep learning knowledge.
For startups and dev’s you need to evaluate these models based on your app’s requirements, resources and technical expertise. For example, at VocalJet, we use those models to convert voice memo to text. By using the strengths and minimizing the weaknesses of these open source speech to text software you can build effective and reliable speech recognition solutions that will delight your users and drive innovation.