Hindi is the third most widely spoken language in the world, but it barely registers in the web data most AI models learn from.

The Falcon team noted in their paper that nearly 60% of the Common Crawl documents they processed are in English, and that Hindi does not even make the top 20 languages. Common Crawl's own statistics for its latest crawl, which count raw pages before any filtering, show the same imbalance: English is still about 42% of pages, and Hindi comes 32nd at about 0.2%.

Hindi is only the most striking example of a wider pattern. The International AI Safety Report 2026 found that the less a language appears in AI training data, the worse the results get. In a test across 83 languages, the results were much weaker for languages not written in the Latin alphabet. The problem goes further than language. It also affects what a model knows about the world. A model that learned mostly from Western material tends to know Western life far better than anywhere else. In one test the report describes, models scored 79% on questions about everyday US culture and only 12% on questions about Ethiopian culture.

Where Multilingual AI Training Data Runs Thin

India shows this pattern on the biggest scale. The country has 22 scheduled languages from four language families and well over a billion speakers. Yet when researchers looked at Common Crawl, they found that Hindi, Bengali, Tamil, Telugu, Marathi and Gujarati together make up less than half of one per cent of it. So a model trained on that archive learns the English-speaking internet in great detail, and Indian life only in small pieces.

Arabic is a different kind of problem. Hundreds of millions of people speak it, but they speak it in many different ways, and the spoken language changes from country to country. Researchers who built speech models for dialects from across the Arabic-speaking world found that existing systems mostly handle Modern Standard Arabic, the formal version, plus a few well-supported dialects, and fall short across the many ways people actually speak.

A recent benchmark of speech recognition for African languages shows how much it comes down to data. The researchers tested four widely used models, giving them as little as an hour of transcribed audio and, for the languages that had enough, a few hundred hours. They describe the field as constrained by limited labelled data. No single model came out best everywhere. The winner changed with the language and the amount of audio available, so what each model had to learn from shaped the result as much as the model itself.

Southeast Asia has its own version of the problem. The team behind SEA-LION, a family of regional models built in Singapore, says existing regional models are less than ideal for languages such as Thai and Tamil, that Lao, Khmer and Filipino still don't have proper benchmarks to test against, and that many English-centric open-source models don't support them.

Why Speech Makes the Gap Wider

Speech needs spoken data, and a web archive has none of that. Someone has to record the audio, transcribe it and check it, ideally a person who actually speaks the language, and that is slow and expensive compared with crawling web pages. OpenAI's Whisper paper shows what this looks like: most languages in its training data had less than a thousand hours of audio, and the authors found that accuracy followed the amount of data closely. More hours meant fewer errors.

The kind of speech matters as much as the amount. One voice reading clean sentences in a quiet studio gives a model tidy but narrow material. Real use is messier, because people interrupt each other, change accent, talk over background noise and slip into another language halfway through a sentence, like an Arabic speaker moving into French or a Hindi speaker into English. Models built for one language alone struggle with that last habit in particular.

No single source covers all of it. Spontaneous recordings capture the hesitations, crosstalk and noise of real conversation. Professionally produced drama and entertainment, written and performed to sound like everyday speech, brings range of a different kind: many speakers, regional accents, shifts in register and mixed-language dialogue, at volume and with clear rights. The IndicVoices project went after variety directly, recording thousands of speakers across India on a wide range of topics.

You can see the cost once models meet real speech. On the IndicVoices test, Whisper made more than twice as many errors on Hindi as a model trained on natural Indian speech, and the gap on Tamil was just as big. That comparison favours the team's own model, since it was trained on the same kind of speech it was tested on, but it still shows how much the training data decides the result. For a team building voice products for these markets, that is the difference between a model that works in a demo and one that works in a bank branch or a call centre.

Why Text in a Language Is Not Enough

Having text in a language does not guarantee that a model learns the culture behind it. Georgia Tech researchers tested 16 language models on Arab and Western cultural contexts and found that they favoured Western entities. The models still referred to alcoholic drinks even when an Arabic prompt mentioned Islamic prayer.

When the team checked six collections of Arabic text, Arabic Wikipedia turned out to be the most Western-centric of them, because so many of its articles are about Western topics. The question is where the text in a language comes from, as well as how much of it there is.

What to Ask a Supplier of Multilingual AI Training Data

Closing the gap takes source material in the language itself, with the accents, settings and mixed-language speech people actually use. Which questions matter most depends on what you're building.

●       Voice assistants and speech recognition: Does the speech sound like real people talking in real situations, whether it was scripted or spontaneous? Which dialects and accents does it cover, and how was that checked? If it matters for your use case, does it include the local words and the mix of languages and dialects people use every day in a specific region, even within the same sentence?

●       Chat models and agents: Which languages and regions does the material cover, and was it made in those languages to begin with? Is it labelled by context and reviewed by native speakers?

●       Anything trained on voices or faces: What do the consent and licence terms cover for each person on screen or on the recording? A voice or a face is personally identifiable in a way a paragraph is not.

If you want to see how a supplier's material answers those questions, you can request a sample set from ContentX Labs and check it against your own pipeline.