Do LLMs Work in Yoruba, Hausa and Igbo? What the Published Numbers Actually Say

Anthropic publishes a table showing how Claude performs in fourteen languages relative to English. Yoruba sits at 79.7% for Claude Sonnet 4.5 with extended thinking. Hausa is absent. Igbo is absent. So when Nigerian developers ask do LLMs work in Yoruba, Hausa and Igbo, the first honest answer is that one of the three is measured in public and two are not.

That absence is not neglect on Anthropic's part specifically. The table is built on OpenAI's MMMLU, the MMLU test set translated by professional human translators into fourteen languages. Yoruba made that list. Hausa and Igbo did not, so no vendor that reuses the benchmark can report them. Roughly 94 million Hausa speakers and over 30 million Igbo speakers sit outside the evaluation that every major lab quotes.

What follows is what is actually published: the vendor model cards, the African-built benchmarks, the orthographic features that break tokenisers, and a method you can run yourself before you commit to a model.

A model card saying it supports 140 languages is not a claim about your language

Read the cards closely and the word support turns out to mean two different things in the same document.

Meta's Llama 4 card lists twelve supported languages: Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai and Vietnamese. No Nigerian language. The same card states the models were pretrained on 200 languages. Meta's launch blog, not the card, adds that over 100 of those had more than a billion tokens each. Then the operative clause: developers may fine-tune for languages beyond the twelve "provided they comply with the Llama 4 Community License and the Acceptable Use Policy", and "developers are responsible for ensuring that their use of Llama 4 in additional languages is done in a safe and responsible manner". Yoruba is in the training data and outside the warranty. The risk transfers to you.

Google's Gemma 3 card advertises support for over 140 languages. Google's launch blog, not the card, is where the narrower figure of over 35 languages supported out of the box appears. Its limitations section contains one sentence that matters more than the headline: "A limitation of our evaluations was they included only English language prompts." The safety testing that justifies the release was run in English. If you deploy Gemma 3 for Hausa customer support, nobody has checked how it refuses, how it handles adversarial input or how it behaves on sensitive topics in Hausa. You have.

So there are three tiers hiding under one word. Languages the model was measured on. Languages it was trained on but not measured. Languages neither trained on deliberately nor measured, which reach the model only through incidental web crawl. Yoruba is tier two at Meta and tier one at Anthropic and OpenAI. Hausa and Igbo are tier two almost everywhere.

At 52.7% of English performance the small model is not a degraded version of the big one. It is a different product.

The cheap model tier collapses in Yoruba and barely moves in Spanish

This is the finding that should change how you architect, and it is sitting in a public table that almost nobody reads down to the bottom row.

Anthropic reports each language twice, once for Sonnet 4.5 and once for Haiku 4.5. For Spanish the drop from flagship to small model is 98.2% to 96.4%, a loss of 1.8 points. French loses 1.8. Portuguese loses 1.7. Bengali loses 5.0. Swahili loses 12.8, from 91.1% down to 78.3%. Yoruba goes from 79.7% to 52.7%. A loss of 27 points.

Read that as an engineering constraint. The standard cost optimisation, routing high-volume or low-stakes traffic to the small model, costs a Spanish-language product under two points of relative accuracy. The identical architecture costs a Yoruba-language product twenty-seven. At 52.7% of English performance the small model is not a degraded version of the big one. It is a different product.

There is a structural reason. Capability in a low-resource language appears to be one of the first things that falls out when a model is distilled or shrunk, because it was thinly represented to begin with. High-resource languages survive compression because they are redundantly encoded. Yoruba is not.

The practical consequence is uncomfortable. Price sensitivity is highest exactly where in-language support matters most, in consumer products serving users who are not comfortable in English. Those products are the ones most likely to run on the cheap tier. The cost curve and the capability curve point in opposite directions.

African-built benchmarks measure things the vendor tables cannot

Two benchmarks from the African NLP community fill in what MMMLU leaves out, and both are peer reviewed.

IrokoBench, published at NAACL 2025 by Masakhane and collaborators, is a human-translated benchmark for 17 typologically diverse African languages including Yoruba, Hausa and Igbo. It has three tasks: AfriXNLI for natural language inference, AfriMGSM for mathematical reasoning and AfriMMLU for multiple-choice knowledge. The headline result is about the open-weight gap. The best open model at the time, Gemma 2 27B, reached only 63% of GPT-4o's performance. If your deployment constraints require self-hosting, that is the number to plan around.

AfroBench, in ACL Findings 2025, is larger: 64 African languages, 15 tasks, 22 datasets, split across nine natural language understanding sets, six generation sets, six knowledge and question-answering tasks and one mathematical reasoning task. On its lite configuration GPT-4o scored 85.1 on English and 66.0 on African languages. Gemma 2 27B scored 80.6 on English and 43.5 on African languages. The proprietary gap is 19 points. The open-weight gap is 37. The paper also finds that performance varies with the availability of monolingual data for a language, which is a statement about the pipeline rather than the model.

One caveat applies to every score above, including Anthropic's Yoruba figure. These are translated English tests. MMMLU is MMLU rendered into Yoruba. AfriMMLU is knowledge questions translated into seventeen languages. A high score means the model can answer American-curriculum questions when they arrive in Yoruba. It says nothing about whether the model can hold a conversation about Yoruba land tenure, follow Igbo kinship terminology or produce Hausa that a Kano reader would accept as unforced. Nobody has a public benchmark for that.

AfroBench does contain one task that tests the writing system directly rather than testing knowledge. AfriADR takes sentences from the MAFAND-MT corpus, strips every accent and diacritic to make the input and keeps the original diacritised text as the target, in five languages including Yoruba and Igbo. Given "Sugbon sibesibe, Mama o gbagbo", the target is "Ṣùgbọ́n síbẹ̀síbẹ̀, Màmá ò gbàgbọ́". That is the closest thing to a test of whether a model has internalised the writing system rather than memorised a translation table.

Three writing systems, three separate failure modes

Grouping these languages as Nigerian obscures that they break models in different ways.

Yoruba carries two orthographic layers. Underdots distinguish letters, so ẹ, ọ and ṣ are distinct from e, o and s. On top of that sit three lexical tones marked with grave, macron and acute. Both layers are semantic and both are routinely dropped in typed text. Strip them and words collapse into each other. This is why AfriADR exists as a task at all, and why a decade of Yoruba NLP work has treated diacritic restoration as a prerequisite rather than a nicety.

Hausa in Boko script uses four characters outside the plain Latin set: ɓ, ɗ, ƙ and ƴ. In real digital text these get substituted with unhooked equivalents, along with spacing errors. Wali and Nisioi built a parallel corpus of over 400,000 sentence pairs by injecting synthetic noise calibrated on roughly 10,000 genuinely noisy Hausa tweets, and found that correcting it improves downstream translation, classification and question answering. Hausa also has a living Ajami tradition in Arabic script that no major vendor lists among its supported languages or scripts, and Ajami input will most likely be detected as Arabic. Ajami input will most likely be detected as Arabic.

Igbo has around thirty dialects and a standard, Central Igbo, that differs from what many people actually write. The IgboAPI dataset, presented at LREC-COLING 2024, was built as a multi-dialectal Igbo-English lexicon precisely because dialectal coverage was the blocking problem, and fine-tuning on it improved dialect handling in machine translation. Igbo also marks tone with diacritics and uses ị, ọ, ụ and ṅ. A model tuned on Central Igbo will read Owerri or Onitsha usage as errors.

Underneath all three sits the tokeniser. Petrov and colleagues showed at NeurIPS 2023 that the same text translated across languages can differ in token count by up to fifteen times, and that this persists in tokenisers built for multilingual use. A 2026 preprint that measures African languages specifically, Somide's The African Language Tax (arXiv:2606.24460), reports Yoruba costing roughly 1.85 times English on OpenAI's o200k vocabulary and about 2.15 times on Llama 3.1's, with Igbo at 1.42 and Hausa at 1.35 on o200k. That preprint is not peer reviewed, so treat the exact multipliers as indicative. The direction is not in doubt. You pay more per sentence, you wait longer, and your context window holds less of the conversation.

Your users will not send you benchmark-quality text

Every score above was computed on clean, professionally translated, correctly diacritised text. Nigerian production traffic is none of those things.

Real input arrives without tone marks, because phone keyboards make them slow. It code-switches inside a single sentence, Yoruba into English and back, or drifts into Nigerian Pidgin mid-message. It uses regional vocabulary. It contains WhatsApp-native abbreviations that no corpus has seen.

The training data is no cleaner. The Quality at a Glance audit in TACL 2022 hand-checked 205 language-specific corpora across CCAligned, ParaCrawl, WikiMatrix, OSCAR and mC4, and found at least fifteen with no usable text and a significant fraction where fewer than half the sentences were of acceptable quality, with mislabelling concentrated in lower-resource languages. Some of what a model learned as Yoruba was never Yoruba.

This is the gap that will hurt you in production. A model that scores 79.7% on translated MMLU can still fail on the first message a Lagos user sends it, because that message contains none of the properties the benchmark held constant.

How to evaluate a model for your own Nigerian-language use case

Published scores narrow the shortlist. They do not settle it. Build your own evaluation, and keep it small enough that you will actually run it on every model change.

Collect 150 to 300 real items in the register you serve. Not translated English. Actual user messages, actual support tickets, actual queries, in the dialect your users speak. Label the correct outcome by hand with a native speaker who is not the person building the system.

Run each item twice, once diacritised and once with marks stripped, and compare. The delta is your diacritic sensitivity, and it predicts how the system will behave on real keyboard input better than any benchmark does.

Test native prompting against translate-test, where you machine-translate the input to English, process it, then translate the response back. IrokoBench found translate-test closes much of the gap for English-centric open models, though it costs you two translation errors and any culturally specific content that does not survive the round trip. Whether it wins for you is an empirical question with a cheap answer.

Measure your own token cost rather than trusting a published multiplier. Load the vendor's tokeniser, encode a few thousand words of your real corpus, divide tokens by whitespace-delimited words, and do the same for the English equivalent. Ten lines. The ratio determines your per-request price, your latency and how much conversation history fits in context.

Test safety and refusal behaviour in-language. Gemma's own card tells you this was not done. Prompt injection, requests for harmful content and adversarial framing all behave differently when the model is operating at reduced competence.

Then evaluate every model tier you might route to, not just the flagship. Given a 27-point flagship-to-small drop in Yoruba against 1.8 in Spanish, a routing rule validated on your English traffic is not validated on your Yoruba traffic.

The strongest argument that this is a solved-in-two-years problem

The case against worrying is real and worth stating properly.

Frontier scores are rising fast. On OpenAI's own multilingual MMLU, o3-high reaches 0.780 on Yoruba against 0.912 on Italian. A 13-point gap on a translated knowledge test is not catastrophic, and it is far better than the same models managed two generations earlier. Reasoning models appear to transfer capability across languages more efficiently than their predecessors, which is exactly what you would hope.

The data pipeline is also finally moving, and moving from inside Nigeria. NaijaVoices released 1,838 hours of speech from 5,455 speakers across Igbo, Hausa and Yoruba, roughly 600 hours each, the largest multi-speaker African speech dataset at release. Masakhane has shipped MasakhaNER 2.0 across 20 languages, MasakhaNEWS across 16, AfriQA with over 12,000 cross-lingual question-answering examples and AfriSenti with more than 110,000 annotated tweets in 14 languages, extending the earlier NaijaSenti work on Hausa, Yoruba, Igbo and Nigerian Pidgin. Lelapa AI's InkubaLM, a 0.4-billion-parameter model trained on 2.4 billion tokens covering five African languages plus English and French, performs comparably to much larger models on translation, question answering, AfriMMLU and AfriXNLI, and beats many larger models on sentiment analysis. Common Crawl has committed since late 2024 to expanding language coverage and ran a language-identification hackathon with Masakhane in 2025. AfroBench's own finding, that performance tracks available monolingual data, means this work converts directly into capability.

Where the optimism breaks down is in what the flywheel does not reach. More web text does not fix Igbo dialect coverage, because the dialects are spoken rather than written. It does not fix Hausa Ajami, because Ajami text is not being crawled as Hausa. It does not fix undiacritised Yoruba, because the crawl is the source of the undiacritised text. And it does nothing about the tier economics, which push the users who most need in-language service onto the models that serve them worst.

The number worth tracking is the distance between the tiers

Most coverage of African languages in AI benchmarks the flagship model and reports the gap to English. That number is improving and will keep improving, because the labs have an incentive to show it improving.

The number nobody optimises is the drop from the flagship to the model people can afford to run at scale. In Spanish it is under two points. In Yoruba it is twenty-seven. That distance is the actual price of being served in your own language, it is currently paid by the users with the least ability to pay it, and no vendor has ever announced a target for reducing it.

Sources

Anthropic, Multilingual support (relative performance table, 14 languages): https://platform.claude.com/docs/en/build-with-claude/multilingual-support

OpenAI simple-evals, Multilingual MMLU benchmark results: https://github.com/openai/simple-evals/blob/main/multilingual_mmlu_benchmark_results.md

Meta, Llama 4 model card (supported languages and developer responsibility clause): https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md

Google, Gemma 3 model card (140+ languages; English-only evaluations): https://ai.google.dev/gemma/docs/core/model_card_3

AfroBench: How Good are Large Language Models on African Languages? (ACL Findings 2025): https://aclanthology.org/2025.findings-acl.976/

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models (NAACL 2025): https://aclanthology.org/2025.naacl-long.139/

Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023): https://arxiv.org/abs/2305.15425

Kreutzer, Caswell et al., Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (TACL 2022): https://aclanthology.org/2022.tacl-1.4/: https://aclanthology.org/2022.tacl-1.4/

Somide, The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs (preprint, 2026): https://arxiv.org/abs/2606.24460