Case study — Apple
Multilingual Audio Data Collection for Live Translation
Live translation only works as well as the data behind it. When a model has only heard a handful of languages, usually from people who've lived abroad for years, it fails exactly where real conversations happen: regional topics, local references, the natural back-and-forth of how people actually talk.
6 min read
Overview
I worked as a moderator on a research program built to close that gap, collecting natural conversational audio across 31 new languages, including regional dialects and accents, to evaluate and improve the ML models powering live translation and ASR. The job sat between research operations, participant-facing moderation, and data quality: running human-subjects sessions at scale while staying close enough to the engineering team to shape what we collected and why.
The Problem
The existing models supported only 4 major languages for live translation, and the data behind them had blind spots that surfaced as real failures:
01
Thin regional coverage
Little data on region-specific topics. The model handled general speech but broke down on local subject matter.
02
Unrepresentative speakers
Many available speakers had lived in the US for 5+ years, so their speech no longer reflected how the language is spoken natively today.
03
Unnatural audio
Clean, studio-style clips didn't match real speech, with its pauses, overlap, and background sound.
04
Field-distance gaps
The models needed to handle both near- and far-field audio, which existing data underrepresented.
Why a Moderated Approach
The goal was natural conversation on specific topics, not scripted reads, with real environmental sound instead of a silent booth. That's why I ran this as moderated sessions instead of an unmoderated collection: moderation kept sessions on-topic and held data-quality and consent standards steady, while still capturing speech that sounded like life.
The core use case we designed around: two people having a conversation in their shared language while wearing AirPods. That's the exact real-world setup live translation has to serve.
Approach & Protocol
Scale
31
New languages
including regional dialects and accents
~1,000
Target participants
native speakers, accented and non-accented
24 hrs
Audio per locale
from ~2-hour sessions, ≥30 min usable conversation each
Recruitment
I recruited native speakers, both accented and non-accented, prioritizing people born in and connected to the regions where each language is actually spoken, to counter the "lived-abroad-too-long" skew. Recruitment ran through surveys, direct outreach, and voluntary participation.
Eliciting natural speech
Drawing genuine conversation out of a research session is a craft. I structured sessions across three conversation modes to pull different registers of speech:
Collaborative / task-oriented
Working toward something together
Opinion / debate
Natural disagreement and emphasis
Open-ended / personal
Relaxed, unguarded speech
Prompts centered on culturally grounded, memory-rich topics: family, childhood, festivals, gatherings, local food. These reliably pulled people into fluent, unselfconscious talk and surfaced the regional vocabulary the models were missing. I deliberately rotated topics to keep the dataset diverse instead of letting it converge on the same phrases, and I kept the natural pauses and overlap in recordings instead of cleaning up the flow. Those disfluencies are exactly what the models needed to learn from.
The setup.
Closing the Loop with Engineering
I'm proudest of changing what we collected based on where the model failed. Running sessions alone wouldn't have gotten us there.
When the model showed weak transcription accuracy on regional topics specifically, I worked directly with engineers to diagnose it as a data problem rather than a model problem: the collection wasn't surfacing enough region-specific language. We adjusted the protocol in response, tagging each recording with a prompt/topic ID so specific topics could be linked back to their audio. That made the dataset diagnosable instead of a black box and let the team see which topics and locales were underperforming, so we could target the next round of collection.
Model failure → data diagnosis → protocol change → better coverage: that loop is the thing I'd most want a research team to know I can do.
What Moderating at Scale Taught Me
Running a high volume of sessions surfaced patterns no protocol document would have predicted. Several pointed directly at how the data and tooling should change.
Lesson 01
Dialect lives below the language level
I kept noticing that two participants from different countries, speaking the "same" language, carried distinct regional dialects and accents. A single language label flattened variation the models needed to learn. This pointed to a concrete improvement: letting collection specify region within a language, capturing accent and dialect at the level speakers actually vary. For a model meant to work in someone's real environment, that granularity is the difference between recognizing a speaker and mistranscribing them.
Lesson 02
Natural conversation between strangers is a design problem
The use case assumed two people talking as if they knew each other, but many sessions paired strangers, and strangers don't speak the way friends do. The rapport that produces relaxed, overlapping, natural speech was often missing, which shaped how representative the audio was. Naming that gap mattered: the conditions of a session shape the data as much as the prompts do, and familiarity and warm-up are variables to design for, not assume away.
Lesson 03
Diverse data sometimes costs participant comfort
The topics that produced the richest, most varied speech (debate, personal subjects) also made participants less comfortable than neutral small talk would have. That tension is real: the most emotionally varied speech and the most comfortable session aren't always the same one. I treated it as a deliberate trade-off to weigh and consent to, not ignore, keeping topics within what participants had agreed to while being honest that emotional range in the data came at some cost to ease. Holding that balance consciously, rather than optimizing blindly for data volume, is part of doing human-subjects work responsibly.
Impact
4 → 31
Supported languages
with regional dialects and accents built in from the start
Multilingual conversational dataset
used to evaluate and improve the ML models behind live translation and ASR
Topic-level traceability
prompt IDs made data quality diagnosable and targetable for engineering
Delivered against the program's target scale: sessions run toward ~1,000 participants across all 31 languages, each contributing toward a 24-hour-per-locale dataset built specifically to close the coverage gaps the model was failing on.
What I Took Away
Two lessons stuck. First: at scale, who and how you collect from is a research design decision, not a logistics detail. The speaker skew and the missing regional topics weren't collection errors. They were the difference between a model that works in a demo and one that works in someone's kitchen.
Second: the session itself is data-generating machinery, and its conditions matter. Dialect below the language level, rapport between strangers, the comfort-versus-diversity trade-off: these only became visible by moderating many sessions and paying attention. Moving from executing a protocol to noticing what it misses, and reshaping it alongside the people building the models, is what turns data collection into research.
