Language Identifier

Runs inside your own AWS account: your text is processed on a SageMaker endpoint you control, inside your VPC, and is never sent to a third-party API.

This SageMaker model package provides a REST API to detect the language of an input text.

It identifies 20 languages: Arabic, Bulgarian, German, Greek, English, Spanish, French, Hindi, Italian, Japanese, Dutch, Polish, Portuguese, Russian, Swahili, Thai, Turkish, Urdu, Vietnamese, and Chinese. Each prediction returns the ISO language code and a confidence score.

Use it to route incoming support messages to the right language queue, tag multilingual document sets before indexing, or filter a training corpus down to one language.

The API accepts input as JSON or CSV and supports real-time endpoints and batch transform.

We welcome your feedback at [email protected]

Model and training data The model is a ModernBERT-base (answerdotai/ModernBERT-base) text classifier fine-tuned with PyTorch and Hugging Face Transformers on the public papluca/language-identification dataset. It runs in network isolation on SageMaker, so no data leaves your account.

Known limitations - Detects 20 languages only. Text in any other language is still mapped to one of the 20, so use the confidence score to reject it. - The model was fine-tuned on inputs up to 512 tokens. Longer text is accepted, but accuracy is only characterized to that length. - Very short inputs (a few words) carry less signal and are less reliable than a full sentence. - Code-switched text returns a single dominant language.

Measured performance - Accuracy 99.8 percent, macro-F1 0.998, measured on 18,000 held-out examples from papluca/language-identification. The split is seeded (70/20/10, seed 42) and was not used in training. - Real-time endpoint on ml.m5.large: 251 ms median per request (276 ms p95). - Throughput on the same instance: about 24 texts per second at a batch size of 32.