Pronunciation Assessment with Azure AI Speech for Language Apps (.NET)

Language-learning apps live or die on feedback. A learner can read a sentence aloud, but without feedback they do not know whether they were understandable. Azure AI Speech includes a pronunciation assessment feature that scores a spoken sentence against reference text and returns accuracy, fluency, completeness and per-word and per-phoneme detail. This guide shows how to put it behind a backend API in .NET, how to design the response for a mobile client and what to watch for in production.

What pronunciation assessment returns

You provide the reference text the learner was supposed to say, plus audio. The service returns scores on a 0-100 scale:

It also returns each word with its own accuracy score and error type (for example omission, insertion or mispronunciation), and optionally each phoneme. That is the data you need to highlight the exact word that went wrong.

Language and locale support differs by feature and changes over time, so check the current language support table for your target locales before you design the product around a specific one.

Architecture

Keep the Speech key off the device. A practical flow:

1. The mobile app records a short audio clip (for example 16 kHz, 16-bit, mono WAV).
2. The app uploads the clip and the reference text to your API.
3. The API calls Azure Speech with pronunciation assessment enabled.
4. The API returns a compact result: overall score, words with scores and colour hints.
5. The app shows the text with problem words highlighted and lets the learner retry.

For lower latency you can use short-lived authorization tokens and call Speech directly from the client, but a backend gives you rate limiting, quota control and a place to store attempts for progress tracking.

Step 1: Create the Speech resource

az group create --name rg-speech --location southeastasia

az cognitiveservices account create \
  --name speech-lang-app \
  --resource-group rg-speech \
  --kind SpeechServices \
  --sku S0 \
  --location southeastasia

az cognitiveservices account keys list \
  --name speech-lang-app \
  --resource-group rg-speech

Pick a region close to your users to reduce latency. For production, prefer Microsoft Entra ID authentication and managed identities over keys when your hosting supports it; see Azure Managed Identities guide.

Step 2: Call pronunciation assessment from .NET

Install the SDK:

dotnet add package Microsoft.CognitiveServices.Speech

Create a service that assesses a WAV stream against reference text:

using Microsoft.CognitiveServices.Speech;
using Microsoft.CognitiveServices.Speech.Audio;
using Microsoft.CognitiveServices.Speech.PronunciationAssessment;

public sealed class PronunciationService(IConfiguration config)
{
    public async Task<AssessmentResult> AssessAsync(Stream wav, string referenceText, string locale)
    {
        var speechConfig = SpeechConfig.FromSubscription(
            config["Speech:Key"]!, config["Speech:Region"]!);
        speechConfig.SpeechRecognitionLanguage = locale; // e.g. "en-US"

        using var pushStream = AudioInputStream.CreatePushStream();
        using var audioConfig = AudioConfig.FromStreamInput(pushStream);

        var buffer = new byte[4096];
        int read;
        while ((read = await wav.ReadAsync(buffer)) > 0)
            pushStream.Write(buffer[..read]);
        pushStream.Close();

        using var recognizer = new SpeechRecognizer(speechConfig, audioConfig);

        var assessment = new PronunciationAssessmentConfig(
            referenceText,
            GradingSystem.HundredMark,
            Granularity.Phoneme,
            enableMiscue: true);
        assessment.ApplyTo(recognizer);

        var result = await recognizer.RecognizeOnceAsync();
        if (result.Reason != ResultReason.RecognizedSpeech)
            return AssessmentResult.NotRecognized(result.Reason.ToString());

        var pa = PronunciationAssessmentResult.FromResult(result);
        var words = pa.Words.Select(w => new WordScore(
            w.Word, w.AccuracyScore, w.ErrorType)).ToList();

        return new AssessmentResult(
            Recognized: true,
            Accuracy: pa.AccuracyScore,
            Fluency: pa.FluencyScore,
            Completeness: pa.CompletenessScore,
            Overall: pa.PronunciationScore,
            Words: words);
    }
}

public record WordScore(string Word, double Accuracy, string ErrorType);

public record AssessmentResult(
    bool Recognized, double Accuracy, double Fluency, double Completeness,
    double Overall, IReadOnlyList<WordScore> Words)
{
    public static AssessmentResult NotRecognized(string reason) =>
        new(false, 0, 0, 0, 0, []);
}

Notes on the code:

• RecognizeOnceAsync handles a single utterance of up to about 30 seconds, which suits one sentence. For longer passages, use continuous recognition and aggregate results.
• enableMiscue: true lets the service flag omitted and inserted words relative to the reference.
• Granularity.Phoneme returns per-phoneme detail. Use Word or FullText if you only need coarser output.

Step 3: Expose a minimal API

app.MapPost("/pronunciation/assess", async (
    HttpRequest request,
    PronunciationService service) =>
{
    var form = await request.ReadFormAsync();
    var audio = form.Files["audio"];
    var text = form["text"].ToString();
    var locale = form["locale"].ToString();

    if (audio is null || string.IsNullOrWhiteSpace(text) || audio.Length > 2_000_000)
        return Results.BadRequest();

    await using var stream = audio.OpenReadStream();
    var result = await service.AssessAsync(stream, text, locale);
    return Results.Ok(result);
}).RequireAuthorization().RequireRateLimiting("assessments");

Limits matter here. Cap audio size and duration, require authentication and rate limit per user. Speech calls are billed by audio duration, so an unauthenticated endpoint is a direct cost risk.

Step 4: Turn scores into feedback learners understand

Raw scores are not feedback. Turn them into something a learner can act on:

• Thresholds for colour. For example green at 80 and above, amber from 60 to 79, red below 60. Tune them with real users, because a native-like score is unrealistic for beginners.
• Highlight the worst words first. Show the two or three lowest-scoring words instead of every flaw.
• Offer a model. Provide a text-to-speech playback of the correct sentence and let the learner compare. Azure Speech's neural voices can generate it.
• Allow instant retry, and track improvement across attempts rather than showing a single score as a verdict.
• Handle non-recognised audio gracefully. Silence, noise and clipped audio are common; ask the learner to try again in a quieter place.

Production concerns

• Audio format. Convert on the device to a format the service accepts. Mismatched sample rates are a common source of poor scores.
• Latency. Keep clips short and the API in the same region as the Speech resource. Show a progress indicator immediately.
• Cost control. Cap daily assessments per free user and cache nothing that contains personal audio without consent.
• Privacy. Voice is personal data. Do not store audio longer than needed, state this in your privacy policy and consider the rules for minors if your app serves children.
• Accent fairness. Scores reflect a reference model. Present them as guidance and avoid "pass or fail" language that discourages learners.
• Observability. Log request duration, result reason and error rates, and send them to Application Insights; see Azure Application Insights guide.

Combining with other Azure AI services

A complete speaking-practice feature usually mixes several services:

• Azure OpenAI to generate practice sentences or give written coaching; see Azure OpenAI Service guide.
• Azure Translator to show meanings in the learner's native language; see Azure Translator guide.
• Speech text-to-speech for model pronunciation.
• Blob Storage for short-lived audio and a lifecycle rule that deletes it automatically.

Related guides

• Azure OpenAI Service guide
• Azure Translator guide
• Azure Managed Identities guide

Conclusion

Azure AI Speech pronunciation assessment gives a language app the data it needs for real feedback: per-word and per-phoneme scores against reference text. Put it behind an authenticated, rate-limited API, translate scores into simple highlights and retries, and treat audio as sensitive data. The same pattern works for any speaking-practice product. If you are planning an AI language or speech feature, the services page describes how I work on these projects.

References

• Use pronunciation assessment (Microsoft Learn)
• Speech service language support (Microsoft Learn)
• Speech SDK (Microsoft Learn)