Mengsay Loem/Notes
EN/日本語
Home
Paper note

Does Romanization Really Help Multilingual Transfer?

2026-09-01 · paper-reading / multilingual / llm / cross-lingual-transfer / romanization / tokenization
Reading
One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography ↗Zhang et al. · EMNLP 2026

The question

Multilingual language models often rely on shared token and lexical structure to transfer across languages. But native orthography can hide that structure when closely related languages use different scripts.

This paper asks a simple but surprisingly consequential question: what if multilingual models are pretrained from the start on a shared surface representation instead of native orthography?

The authors compare native text, IPA transcription, and romanization across eight languages and several model scales, with a particular focus on cross-script and unseen-language transfer.

What they find

The headline result is quite strong: romanized pretraining performs best in most of the evaluated settings, with IPA generally between romanization and native text.

A few findings stood out to me:

  • The gains are especially visible for cross-script and unseen-language transfer.
  • Post-hoc romanization is not equivalent: taking a model pretrained on native text and only romanizing downstream data can substantially hurt performance when that language/script was already seen during pretraining.
  • The benefit cannot be explained by subword overlap alone. In some language pairs, IPA creates more overlap without producing better downstream performance than romanization.
  • Representation also changes tokenization efficiency: the same source corpus becomes substantially shorter in token space after romanization/IPA, so a fixed token budget may expose the model to more underlying linguistic content.

Our take

What I found most interesting is that “romanization” is not really a single intervention.

It simultaneously changes several things:

  • the character inventory,
  • subword overlap,
  • sequence length and tokenizer fertility,
  • how much source content fits into a fixed compute budget,
  • the visibility of cognates and phonological similarity,
  • and the amount of native orthographic information that is collapsed.

So I think the strongest conclusion is not simply “romanization improves transfer because related words share more tokens.” The paper shows convincingly that the whole romanized representation bundle works, but the reason it works is still mixed together.

This also made me hesitate to call romanization a truly “shared linguistic representation.” Romanization mainly gives different languages a shared coding space. IPA goes one step further toward phonological sharing, but neither directly aligns meanings that have unrelated surface forms or pronunciations.

A useful way to think about the hierarchy may be:

script normalization → phonological normalization → learned semantic abstraction

Romanization mostly acts at the first level. A genuinely language-independent internal representation would need to emerge at a much deeper level.

There is also an interesting trade-off. Romanization removes some distinctions, which means different native forms can become ambiguous and must be resolved from context. For cross-lingual classification or transfer, those distinctions may behave like nuisance information. For native spelling, named entities, or generation back into the original script, the same information may be essential.

What I want to know next

The question I would most like to separate experimentally is:

Which part of script equalization actually causes the transfer gain?

For example, compare native text and romanization against an arbitrary but reversible remapping that has similar token efficiency, or against a token-length-controlled representation. If an arbitrary mapping works almost as well as romanization, then shared coding and compute efficiency may be doing much of the work. If romanization remains clearly better, linguistically meaningful lexical or phonological alignment becomes a stronger explanation.

That feels like the most interesting direction left open by this paper: not just whether romanization works, but what information we should preserve, collapse, or share to make multilingual transfer easier under finite compute.