Of round 7,000 languages on the earth, a tiny fraction are supported by AI language fashions. NVIDIA is tackling the issue with a brand new dataset and fashions that assist the event of high-quality speech recognition and translation AI for 25 European languages — together with languages with restricted obtainable knowledge like Croatian, Estonian and Maltese.
These instruments will allow builders to extra simply scale AI purposes to assist international customers with quick, correct speech know-how for production-scale use instances similar to multilingual chatbots, customer support voice brokers and near-real-time translation companies. They embody:
- Granary, an enormous, open-source corpus of multilingual speech datasets that accommodates round 1,000,000 hours of audio, together with almost 650,000 hours for speech recognition and over 350,000 hours for speech translation.
- NVIDIA Canary-1b-v2, a billion-parameter mannequin skilled on Granary for high-quality transcription of European languages, plus translation between English and two dozen supported languages. It tops Hugging Face’s leaderboard of open fashions for multilingual speech recognition accuracy.
- NVIDIA Parakeet-tdt-0.6b-v3, a streamlined, 600-million-parameter mannequin designed for real-time or large-volume transcription of Granary’s supported languages. It has the best throughput of multilingual fashions on the Hugging Face leaderboard, measured as period of audio transcribed divided by computation time.
The paper behind Granary will probably be introduced at Interspeech, a language processing convention going down within the Netherlands, Aug. 17-21. The dataset, in addition to the brand new Canary and Parakeet fashions, at the moment are obtainable on Hugging Face.
How Granary Addresses Information Shortage
To develop the Granary dataset, the NVIDIA speech AI crew collaborated with researchers from Carnegie Mellon College and Fondazione Bruno Kessler. The crew handed unlabeled audio by an progressive processing pipeline powered by NVIDIA NeMo Speech Information Processor toolkit that turned it into structured, high-quality knowledge.
This pipeline allowed the researchers to reinforce public speech knowledge right into a usable format for AI coaching, with out the necessity for resource-intensive human annotation. It’s obtainable in open supply on GitHub.
With Granary’s clear, ready-to-use knowledge, builders can get a head begin constructing fashions that deal with transcription and translation duties in almost the entire European Union’s 24 official languages, plus Russian and Ukrainian.
For European languages underrepresented in human-annotated datasets, Granary offers a vital useful resource to develop extra inclusive speech applied sciences that higher mirror the linguistic variety of the continent — all whereas utilizing much less coaching knowledge.
The crew demonstrated of their Interspeech paper that, in comparison with different standard datasets, it takes round half as a lot Granary coaching knowledge to realize a goal accuracy degree for automated speech recognition (ASR) and automated speech translation (AST).
Tapping NVIDIA NeMo to Turbocharge Transcription
The brand new Canary and Parakeet fashions supply examples of the sorts of fashions builders can construct with Granary, custom-made to their goal purposes. Canary-1b-v2 is optimized for accuracy on complicated duties, whereas parakeet-tdt-0.6b-v3 is designed for high-speed, low-latency duties.
By sharing the methodology behind the Granary dataset and these two fashions, NVIDIA is enabling the worldwide speech AI developer neighborhood to adapt this knowledge processing workflow to different ASR or AST fashions or extra languages, accelerating speech AI innovation.
Canary-1b-v2, obtainable below a permissive license, expands the Canary household’s supported languages from 4 to 25. It gives transcription and translation high quality akin to fashions 3x bigger whereas operating inference as much as 10x quicker.
NVIDIA NeMo, a modular software program suite for managing the AI agent lifecycle, accelerated speech AI mannequin growth. NeMo Curator, a part of the software program suite, enabled the crew to filter out artificial examples from the supply knowledge in order that solely high-quality samples have been used for mannequin coaching. The crew additionally harnessed the NeMo Speech Information Processor toolkit for duties like aligning transcripts with audio recordsdata and changing knowledge into the required codecs.
Parakeet-tdt-0.6b-v3 prioritizes excessive throughput and is able to transcribing 24-minute audio segments in a single inference move. The mannequin routinely detects the enter audio language and transcribes with out extra prompting steps.
Each Canary and Parakeet fashions present correct punctuation, capitalization and word-level timestamps of their outputs.
Learn extra on GitHub and get began with Granary on Hugging Face.
