Most localization teams that adopt machine translation end up in one of two places. Either every segment still goes through full human post-editing, so the savings are smaller than the business case promised, or a large share of content ships raw and someone in Munich or Osaka finds a mistranslated safety warning six months later. Neither is a technology failure. Both come from treating MT output as uniformly good or uniformly suspect.
Quality estimation (QE) is the missing piece. A QE model scores each translated segment without needing a reference translation, and that score lets you decide, per segment, whether it needs a human. This post covers how we set that up for German and Japanese workflows, where it works, and where it does not.
Quality estimation versus evaluation metrics
It helps to separate two families of tools. Reference-based metrics such as BLEU, chrF and COMET compare MT output against a human translation. They are useful for choosing between engines or checking whether a fine-tuned model improved, because you run them on a test set where references exist. They are useless in production, where the whole point is that no reference exists yet.
Reference-free QE models take only the source and the MT output and predict a quality score. The COMET family includes such variants (CometKiwi is the best known), and several translation management systems and MT vendors now expose their own QE scores. Large language models can also be prompted to grade a segment or to annotate errors in MQM style. LLM grading is flexible and handles context better, but it is slower and costs more per segment, so we tend to use a dedicated QE model for the bulk scoring and an LLM for second opinions on borderline cases.
One warning: raw QE scores are not comparable across models, language pairs or domains. A score of 0.82 from one model on English to German marketing copy means something different from 0.82 on English to Japanese technical documentation. Every threshold has to be calibrated on your own content.
Calibrating thresholds against MQM
To calibrate, you need human judgments in a consistent format. MQM (Multidimensional Quality Metrics) is the framework we use. Reviewers mark each error with a category and a severity, and the errors are weighted into a score per segment or per document. The top-level categories cover accuracy (mistranslation, omission, addition, untranslated text), fluency or linguistic conventions (grammar, spelling, punctuation), terminology, style, and locale conventions such as dates, number formats and currency. Severities are usually minor, major and critical, with critical reserved for errors that could cause harm, legal exposure or a product failure.
Take a sample of 500 to 1,000 segments per language pair and content type, have qualified linguists annotate them in MQM, and plot the QE score against the annotated errors. You are looking for a score above which segments with major or critical errors become rare enough for your risk tolerance. That is your "publish without full review" threshold. Below a second, lower threshold, the MT is usually poor enough that a translator is faster starting from scratch or close to it.
Routing segments to the right level of review
With thresholds in place, routing becomes a set of explicit rules rather than a blanket policy. A typical setup has three or four bands:
| Band | Condition | Treatment |
|---|---|---|
| High confidence | QE above upper threshold, no term or tag violations | Light review or spot check by sampling |
| Medium | QE between thresholds | Full post-editing by a linguist |
| Low | QE below lower threshold | Retranslate, using MT only as a reference |
| Always human | Legal, safety, pricing, brand claims, regulated content | Full translation and review, regardless of score |
The "always human" band is not optional. QE models are trained to predict typical quality, and they can miss a single flipped negation or a wrong number in an otherwise fluent sentence. Dosage instructions, contract clauses and safety notices go to people. So does anything with a hard legal consequence, such as German Impressum content or product labeling.
Sample the high-confidence band continuously. We usually start by reviewing 10 to 20 percent of it, annotate what reviewers find in MQM, and lower the sample rate only when the error rate stays inside the agreed limit for several releases. If an engine update or a new content source moves the error rate, the sample catches it before customers do.
Termbase enforcement is deterministic work
Terminology errors are the ones clients notice first, and they are the easiest to catch without any model at all. If your termbase says a product feature is called "Arbeitsbereich" in German, every segment containing the English term should contain the approved German term, and none should contain the forbidden alternatives.
- Inject approved terms into the MT request where the engine supports glossaries, or into the prompt when you translate with an LLM.
- Run a check after translation: source term present, target term absent, means a violation. Handle German compounds and inflection with lemmatization or allowed variant lists, or the check will flood you with false alarms.
- For Japanese, match without relying on spaces, and keep separate entries where the katakana rendering of a product name is fixed by the brand.
- Treat any violation as a routing signal. A segment with a perfect QE score and a wrong term still goes to a linguist.
- Check placeholders, tags and numbers the same way. A missing {count} variable breaks the build, not just the translation.
Register deserves its own rule. German content needs a consistent choice between "Sie" and "du", and Japanese content needs consistent politeness levels. Engines drift between them within a single document. A style check that flags the wrong form is cheap and saves reviewers a lot of tedious edits.
The cost per word trade-off
The business case for MTPE comes down to how many words end up in each band and what each band costs. Rates vary a lot by language pair, domain and vendor, so we will not quote numbers that would not hold for your contracts. The structure of the calculation is what matters.
Light review is typically billed or staffed well below full post-editing, and full post-editing below translation from scratch. If QE routing moves a large share of segments into the light band without raising the error rate, the blended cost per word falls. If your thresholds are too strict, almost everything lands in full post-editing and you pay for the QE pipeline on top of the old process. If they are too loose, you save on review and pay later in support tickets, rework and, for regulated content, real liability.
Include the hidden costs: annotation for calibration, periodic recalibration when engines change, engineering time on the routing pipeline, and the reviewer time spent on sampling. And measure post-edit distance, the share of each segment that linguists actually change. If reviewers edit 40 percent of characters in your "high confidence" band, the threshold is wrong, whatever the QE model says.
Linguists also need to be part of this, not just subject to it. Post-editing productivity drops when people are asked to fix output they consider unusable, and good translators leave. Share the QE bands with them, pay fairly for the band they are working in, and use their MQM annotations to improve the engine and the termbase. The system improves fastest when the people correcting it can see what their corrections change.
Questions we get asked
- What is machine translation quality estimation?
- Quality estimation predicts how good a machine translation is from the source and target alone, without a human reference translation. It is used to decide which segments need human review.
- Is COMET a quality estimation metric?
- Standard COMET needs a reference translation, so it is an evaluation metric. Reference-free variants in the same family, such as CometKiwi, are used for quality estimation in production.
- Can QE replace human review for German or Japanese?
- For some low-risk content with well-calibrated thresholds, QE can reduce review to sampling. Legal, safety and brand-critical content should always get full human review.
- What is MQM in translation?
- MQM (Multidimensional Quality Metrics) is a framework for annotating translation errors by category, such as accuracy, terminology and style, and by severity, then scoring the result.