Training and deploying large-scale deep learning models face two unavoidable thresholds: astronomical computational costs during training, and stringent latency requirements during inference. In real-time interactive scenarios like translation, every additional 10 milliseconds of delay directly impacts user experience. The DeepL engineering team recently shared key experiences in training next-generation LLMs using FP8 (8-bit floating point) precision—potentially one of the most noteworthy infrastructure breakthroughs in translation AI in recent years.
How FP8 Breaks Through: From FP16 to Leaner Numerical Precision
Traditional deep learning training commonly uses FP16 (16-bit floating point) or even BF16 precision. FP8 cuts the bit width of each parameter in half, triggering a chain reaction of improvements: GPU memory consumption is halved, meaning a single H100 can hold a model twice as large or batches twice as big in memory; training throughput sees a measured improvement of roughly 40%, meaning a model that used to take a week to train can now converge in under five days; inference latency drops by over 30%. For real-time translation API calls requiring millisecond-level responses, this represents a leap from one order of magnitude to another.
Mixed-Precision Training: More Than Just “Lowering Precision”
Lowering precision is not without cost. Switching all layers directly from FP16 to FP8 causes numerical stability to deteriorate sharply, leading to training divergence. The DeepL team’s solution is a mixed-precision strategy: attention mechanism layers and LayerNorm layers—the ones most impactful on gradient propagation—retain FP16 precision to ensure no quality loss on critical paths; computationally intensive layers like weight matrix multiplications and feed-forward networks run entirely on FP8. Combined with NVIDIA H100 GPUs’ built-in Transformer Engine—which can perform hardware-level on-the-fly format conversion between FP8 and FP16—this hybrid approach achieves dual optimality in both training speed and model quality.
Inference Transformation: Low Latency Opens New Possibilities
If the training-side gain is mainly about cost optimization, the inference-side improvement directly affects what product forms are possible. FP8 inference compresses end-to-end latency per translation from roughly 50 milliseconds to around 30 milliseconds—with translation quality unchanged. What does 30 milliseconds mean? In real-time voice conversations, this level of latency falls below the “stutter” threshold perceptible to the human ear. This means low-latency scenarios like real-time voice translation, simultaneous interpretation, and live captions for video conferencing are no longer distant visions but capabilities that can now be placed on the product roadmap.
Next Steps: FP4 and Sparsification
FP8 is not the endpoint. The DeepL engineering team is already exploring FP4 precision training—further compressing numerical bit width to 4 bits—along with model sparsification techniques that prune redundant neuron connections while preserving inference accuracy. The combined goal of these techniques: on top of FP8, reduce translation costs by another 50%. If this target is achieved, large-scale multilingual real-time translation will become as cheap, as fast, and as ubiquitous as sending a text message.
The essence of language is connection. And what technology does is continuously lower the physical cost of that connection—from handwritten letters to telegrams, from transoceanic phone calls to instant messaging. FP8 and the low-precision technologies that follow are pulling the cost curve of “breaking language barriers” ever downward, until it approaches zero.