Post Thumbnail

HuggingFace and IISc create language database of India

HuggingFace and the Indian Institute of Science (IISc) have announced a partnership that will make Vaani, India’s largest multimodal and multilingual dataset, available to developers around the world. The project aims to create inclusive AI technologies that take into account the linguistic and cultural diversity of the country.

The Vaani project, launched in 2022 jointly by IISc/ARTPARK and Google, sets an ambitious goal: to collect more than 150,000 hours of speech and 15,000 hours of transcribed text from 1 million people across all 773 districts of India. The uniqueness of the project lies in its geocentric approach, which allows collecting data about dialects and languages used in remote regions, not just mainstream languages.

Currently, the project is being implemented in phases. The first phase, covering 80 districts, has already been completed, and the data is publicly available. The second phase is now underway, expanding coverage to another 100 districts.

As of February 15, 2025, the open part of the dataset includes a transcribed subset containing 790 hours of audio from approximately 700,000 speakers and covering 70,000 images. This resource is intended for various tasks, including:
– Speech recognition: training models for accurate transcription of spoken language
– Language modeling: creating more advanced language models
– Segmentation: identifying individual speech units to improve transcription accuracy

The partnership between HuggingFace and IISc/ARTPARK aims to increase the accessibility and usability of the Vaani dataset, which should contribute to the development of AI systems that better understand the diversity of Indian languages and meet the digital needs of the country’s population.

This project represents an important step in democratizing AI technologies and creating more inclusive solutions that take into account the linguistic diversity of one of the world’s largest countries.

Autor: AIvengo
For 5 years I have been working with machine learning and artificial intelligence. And this field never ceases to amaze, inspire and interest me.
Latest News
AI chatbots generate content that exacerbates eating disorders

A joint study by Stanford University and the Center for Democracy and Technology showed a disturbing picture. Chatbots with artificial intelligence pose a serious risk to people with eating disorders. Scientists warn that neural networks hand out harmful advice about diets. They suggest ways to hide the disorder and generate "inspiring weight loss content" that worsens the problem.

OpenAGI released the Lux model that overtakes Google and OpenAI

Startup OpenAGI released the Lux model for computer control and claims this is a breakthrough. According to benchmarks, the model overtakes analogues from Google, OpenAI and Anthropic by a whole generation. Moreover, it works faster. About 1 second per step instead of 3 seconds for competitors. And 10 times cheaper in cost per processing 1 token.

Altman declared red alert at OpenAI due to Google's successes

Sam Altman declared "red alert level" at OpenAI, and this is not just corporate drama. This is an admission that the market leader felt competitors breathing down their neck. According to an internal memo, he is mobilizing additional resources to improve ChatGPT amid growing threats from Google.

Users spend more time with Gemini than with ChatGPT

OpenAI still leads in user numbers, but people are starting to spend more time with competitors. And this creates a serious problem.

Companies are bringing back 5% of those fired due to AI implementation failure

Many companies began bringing back employees fired because of artificial intelligence. Analytics company Visier studied employment data of 2.5 million employees from 142 companies worldwide. About 5% of fired employees subsequently returned to their previous employer. This indicator remained stable for several years, but recently began to rise.