BENGALURU: For a few weeks this year, villagers in the southwestern Indian state of Karnataka read out dozens of sentences in their native Kannada language into an app as part of a project to build the country's first AI-based chatbot for Tuberculosis.
There are more than 40 million native Kannada speakers in India, and it is one of the country's 22 official languages and one of over 121 languages spoken by 10,000 people or more in the world's most populous nation.
But few of these languages are covered by natural language processing (NLP), the branch of artificial intelligence that enables computers to understand text and spoken words.
Hundreds of millions of Indians are thus excluded from useful information and many economic opportunities.
"For AI tools to work for everyone, they need to also cater to people who don't speak English or French or Spanish," said Kalika Bali, principal researcher at Microsoft Research India.
"But if we had to collect as much data in Indian languages as went into a large language model like GPT, we'd be waiting another 10 years. So what we can do is create layers on top of generative AI models such as ChatGPT or Llama," Bali told the Thomson Reuters Foundation.
The villagers in Karnataka are among thousands of speakers of different Indian languages generating speech data for tech firm Karya, which is building datasets for firms such as Microsoft and Google to use in AI models for education, healthcare and other services.
The Indian government, which aims to deliver more services digitally, is also building language datasets through Bhashini, an AI-led language translation system that is creating open source datasets in local languages for creating AI tools.
The platform includes a crowdsourcing initiative for people to contribute sentences in various languages, validate audio or text transcribed by others, translate texts and label images.
Tens of thousands of Indians have contributed to Bhashini.
"The government is pushing very strongly to create datasets to train large language models in Indian languages, and these are already in use in translation tools for education, tourism and in the courts," said Pushpak Bhattacharyya, head of the Computation for Indian Language Technology Lab in Mumbai.
"But there are many challenges: Indian languages mainly have an oral tradition, electronic records are not plentiful, and there is a lot of code mixing. Also, to collect data in less common languages is hard, and requires a special effort."
ECONOMIC VALUE
Of the more than 7,000 living languages in the world, fewer than 100 are captured in major NLPs, with English the most advanced.
ChatGPT - whose launch last year triggered a wave of interest in generative AI - is trained primarily on English. Google's Bard is limited to English, and of the nine languages that Amazon's Alexa can respond to, only three are non-European; Arabic, Hindi and Japanese.
Governments and startups are trying to bridge this gap.
Grassroots organisation Masakhane aims to strengthen NLP research in African languages, while in the United Arab Emirates, a new large language model called Jais can power generative AI applications in Arabic.
For a country like India, crowdsourcing is an effective way to collect speech and language data, said Bali, who was named among the 100 most influential people in AI by Time magazine in September.
"Crowdsourcing also helps to capture linguistic, cultural and socio-economic nuances," said Bali.
"But there has to be awareness of gender, ethnic and socio-economic bias, and it has to be done ethically, by educating the workers, paying them, and making a specific effort to collect smaller languages," she said. "Otherwise it doesn't scale."
With the rapid growth of AI, there is demand for languages "we haven't even heard of", including from academics looking to preserve them, said Karya co-founder Safiya Husain.
Karya works with non-profit organisations to identify workers who are below the poverty line, or with an annual income of less than $325, and pays them about $5 an hour to generate data - well above the minimum wage in India.
Workers own a part of the data they generate so they can earn royalties, and there is potential to build AI products for the community with that data, in areas such as healthcare and farming, Husain said.
"We see huge potential for adding economic value with speech data - an hour of Odia speech data used to cost about $3-$4, now it's $40," she said, referring to the language of eastern Odisha state.
VILLAGE VOICE
Fewer than 11% of India's 1.4 billion people speak English. Much of the population is not comfortable reading and writing, so several AI models focus on speech and speech recognition.
Google-funded Project Vaani, or voice, is collecting speech data of about 1 million Indians and open-sourcing it for use in automatic speech recognition and speech-to-speech translation.
Bengaluru-based EkStep Foundation's AI-based translation tools are used at the Supreme Court in India and Bangladesh, while the government-backed AI4Bharat centre has launched Jugalbandi, an AI-based chatbot that can answer questions on welfare schemes in several Indian languages.
The bot, named after a duet where two musicians riff off each other, uses language models from AI4Bharat and reasoning models from Microsoft, and can be accessed on WhatsApp, which is used by about 500 million people in India.
Gram Vaani, or voice of the village, a social enterprise that works with farmers, also uses AI-based chatbots to respond to questions on welfare benefits.
"Automatic speech recognition technologies are helping to mitigate language barriers and provide outreach at the grassroots level," said Shubhmoy Kumar Garg, a product lead at Gram Vaani.
"They will help empower communities which need them the most."
For Swarnalata Nayak in Raghurajpur district in Odisha, the growing demand for speech data in her native Odia has also meant a much-needed additional income from her work for Karya.
"I do the work at night, when I am free. I can provide for my family through talking on the phone," she said.
Reuters
Fri Dec 08 2023
![India turns to AI to capture its 121 languages India turns to AI to capture its 121 languages](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2023-10/71697780064_ArtificialIntellige.jpg)
AI tools are largely in English, European languages. - REUTERS/Filepic
Jualan penyaman udara naik mendadak di negara Balkan
Menetap di negara sejuk, mereka tidak pernah terfikir memasang penyaman udara di dalam rumah memandangkan musim panas bukanlah setiap hari. Namun, cuaca panas melampau membuatkan mereka bertukar fikiran.
![Malaysia kecam sekeras-kerasnya pencerobohan ekstremis Zionis Israel ke atas Masjid Al-Aqsa Malaysia kecam sekeras-kerasnya pencerobohan ekstremis Zionis Israel ke atas Masjid Al-Aqsa](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-01/61706688348_wismaputra.jpg)
Malaysia kecam sekeras-kerasnya pencerobohan ekstremis Zionis Israel ke atas Masjid Al-Aqsa
Kerajaan Malaysia mengecam sekeras-kerasnya pencerobohan Masjid Al-Aqsa oleh ekstremis Zionis Israel pada 13 Ogos.
![Reda dengan pemergian anak walaupun bukan pengakhiran diharapkan - Ayah Nidza Afham Reda dengan pemergian anak walaupun bukan pengakhiran diharapkan - Ayah Nidza Afham](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-08/61723653359_TBMOKHTAR.jpg)
Reda dengan pemergian anak walaupun bukan pengakhiran diharapkan - Ayah Nidza Afham
Ayah kepada penyanyi dan Youtubers Muhammad Nidza Afham Mokhtar reda dengan pemergian anak lelakinya walaupun itu bukan pengakhiran yang beliau harapkan.
Ahli perniagaan diberkas, terlibat dalam aktiviti kumpulan mafia
Seorang ahli perniagaan Itali ditangkap di bandar Natal, Brazil pada selasa.
Giuseppe Bruno kini menghadapi tuduhan termasuk melakukan pemerasan dan pengubahan wang haram dan juga penglibatan bersama kumpulan mafia.
Giuseppe Bruno kini menghadapi tuduhan termasuk melakukan pemerasan dan pengubahan wang haram dan juga penglibatan bersama kumpulan mafia.
Srettha hormati keputusan mahkamah
Srettha Thavisin menghormati keputusan Mahkamah Perlembagaan Thailand yang memecatnya daripada jawatan Perdana Menteri.
Lahir empat hari lepas, bayi kembar maut angkara serangan rejim Zionis
Kembar lelaki diberikan nama Asser manakala Ayssel pula nama diberi buat kembar perempuan, kedua-duanya lahir pada 10 Ogos lepas.
Lelaki tikam ibu saudara direman tujuh hari
Seorang lelaki yang disyaki menikam ibu saudaranya di kepala dengan gunting, semalam direman selama tujuh hari hingga 21 Ogos ini.
PN senarai pendek tiga calon, akan diumum selepas PRK Nenggiri
Perikatan Nasional telah menyenarai pendek tiga calon untuk bertanding pada Pilihan Raya Kecil (PRK) Dewan Undangan Negeri (DUN) Mahkota yang dijadualkan berlangsung pada 28 September depan.
![10 cedera dalam kejadian reaktor kilang terbakar di Bintulu 10 cedera dalam kejadian reaktor kilang terbakar di Bintulu](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/41722145972_KebakaranGeneric.jpg)
10 cedera dalam kejadian reaktor kilang terbakar di Bintulu
Menurut PGOB kesemua mangsa yang cedera telah dikejarkan ke Hospital Bintulu untuk rawatan lanjut.
![Mayat Muhammad Nidza Afham dibawa ke Hospital Shah Alam Mayat Muhammad Nidza Afham dibawa ke Hospital Shah Alam](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-04/61713736324_HusseinOmarKhan.jpg)
Mayat Muhammad Nidza Afham dibawa ke Hospital Shah Alam
Mayat Muhammad Nidza Afham Mokhtar yang ditemukan di Sungai Klang petang tadi dibawa ke Unit Forensik Hospital Shah Alam (HSA) untuk urusan bedah siasat.
![India, Maldives ucap tahniah kepada Muhammad Yunus India, Maldives ucap tahniah kepada Muhammad Yunus](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-08/51723168824_MohammedShahabuddi.jpg)
India, Maldives ucap tahniah kepada Muhammad Yunus
Presiden Mohammed Shahabuddin memimpin upacara angkat sumpah Yunus, yang kini mengetuai Kabinet sementara terdiri daripada 16 anggota.
![Pengerusi anak syarikat IHH Healthcare di India letak jawatan Pengerusi anak syarikat IHH Healthcare di India letak jawatan](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-08/71722994210_RaviRajagopal.jpg)
Pengerusi anak syarikat IHH Healthcare di India letak jawatan
Rajagopal memaklumkan kepada lembaga pengarah mengenai kesediaannya meletak jawatan sebagai ahli lembaga pengarah berkuat kuasa 30 September.
![India digesa hentikan eksport senjata ke Israel India digesa hentikan eksport senjata ke Israel](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-08/41722554261_ArundhatiRoy.jpg)
India digesa hentikan eksport senjata ke Israel
Kumpulan masyarakat sivil mendakwa beberapa syarikat tempatan diberi permit untuk mengeksport senjata, dron dan bahan letupan ke Israel.
![Penduduk terselamat gempa bumi Kerala berlindung di sekolah Penduduk terselamat gempa bumi Kerala berlindung di sekolah](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/71722426286_TBINDIALANDSLIDE.jpg)
Penduduk terselamat gempa bumi Kerala berlindung di sekolah
Penduduk yang terselamat daripada kejadian gempa bumi di Kerala India mencari perlindungan di sebuah sekolah di daerah Wayanad.
![Tanah runtuh Wayanad: Pasukan penyelamat teruskan operasi cari mangsa Tanah runtuh Wayanad: Pasukan penyelamat teruskan operasi cari mangsa](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/51722404851_Wayanadlandslides.jpg)
Tanah runtuh Wayanad: Pasukan penyelamat teruskan operasi cari mangsa
Pasukan penyelamat memulakan operasi mencari mangsa terperangkap di bawah runtuhan sehari selepas gempa bumi di Wayanad, Kerala, India pada Rabu.
![Tanah runtuh Wayanad: Angka korban meningkat lebih 70 orang Tanah runtuh Wayanad: Angka korban meningkat lebih 70 orang](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/61722334389_TanahruntuhKerala.jpg)
Tanah runtuh Wayanad: Angka korban meningkat lebih 70 orang
Kerajaan Kerala dalam kenyataan terbaharunya mengisytiharkan tempoh berkabung dua hari pada 30 dan 31 Julai bagi menghormati mangsa.
![India raikan pingat pertama di Sukan Olimpik Paris India raikan pingat pertama di Sukan Olimpik Paris](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/61722212635_ManuBhaker.jpg)
India raikan pingat pertama di Sukan Olimpik Paris
Berasal dari negeri Haryana di utara India, Bhaker menjadi penembak wanita pertama negara yang memenangi pingat Olimpik.
![India dan Britain lancar inisiatif keselamatan teknologi India dan Britain lancar inisiatif keselamatan teknologi](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/71721878547_DAVIDLAMMY.jpg)
India dan Britain lancar inisiatif keselamatan teknologi
Ia merangkumi bidang seperti telekomunikasi, kecerdasan buatan, kuantum, kesihatan dan bioteknologi, bahan termaju dan semikonduktor.
![India kekal negara paling ramai penduduk abad ini - PBB India kekal negara paling ramai penduduk abad ini - PBB](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/81720771313_TBPonggal.jpg)
India kekal negara paling ramai penduduk abad ini - PBB
India dijangka kekal sebagai negara paling ramai penduduk di dunia dan berdepan tekanan ekonomi dan alam sekitar.
![Wanita paling ramai terkorban dalam tragedi rempuhan perhimpunan keagamaan di India Wanita paling ramai terkorban dalam tragedi rempuhan perhimpunan keagamaan di India](https://resizer-awani.eco.astro.com.my/tr:w-177,h-100,q-100,f-auto/https://img.astroawani.com/2024-07/61720003970_stampede.jpg)
Wanita paling ramai terkorban dalam tragedi rempuhan perhimpunan keagamaan di India
Kebanyakan mangsa rempuhan di sebuah perhimpunan keagamaan Hindu di negeri Uttar Pradesh, utara India pada Selasa adalah wanita, menurut senarai kematian.