SONAR: Multilingual & Multimodal Sentence Embeddings

This research paper introduces a new model called SONAR which can understand and translate between many different languages, including spoken languages. SONAR is special because it can turn sentences into fixed-size representations, kind of like creating a code for each sentence. This code can then be used to compare sentences for similarity or to translate them into different languages, even for languages it hasn't been specifically trained on! The researchers tested SONAR on many tasks, including translation and identifying similar sentences, and found that it performs very well, sometimes even better than existing models, especially when working with less common languages. They also extended SONAR to understand spoken language by training it to match speech recordings with their written transcripts. This allows SONAR to perform speech-to-text translation, even for language combinations it has never seen before! The researchers made the SONAR model freely available for others to use and build upon.

https://arxiv.org/pdf/2308.11466