A new standard for AI-powered live captions

What are AI powered closed captions

Traditionally, creating closed captions involved human transcription, which is a costly and difficult task in real-time. AI powered solutions eliminate the need for human captioning by using advanced Automatic Speech Recognition (ASR) algorithms to analyze the live audio and convert it to captions in real time.

Machine learning models are trained on vast amounts of data, allowing them to recognize speech patterns, accents, and context. As a result, their output has improved exponentially over time, capturing the nuances of language and delivering a much higher quality transcription.

How the accuracy of captions is determined

WER (Word Error Rate) and FER (Frame Error Rate) are commonly used metrics to assess the accuracy of closed captions. Both rely on a reference transcript from a human transcribers to detect the number of errors. FER is more suited to evaluate real-time captioning because it counts the number of frames or segments in which the live caption differs from the reference transcript. In our metrics, we exclude cosmetic errors (e.g. text not optimally split into two lines) insofar as they are not hindering the viewer’s experience. By accuracy, then, we mean correctly converting speech to text and rendering it in understandable phrases that are in sync with the live stream.

Thus, an accuracy of 99+% means that, on average, our AI powered captioning gets more than 99 out of every 100 words right and is able to put them in understandable phrases that are in sync with the live stream.¹

This extremely high success rate is only possible thanks to the huge progress in Automatic Speech Recognition (ASR) technology, where different ways of speaking, dialects, accents, vocabulary… no longer lead to inaccuracies in transcription. In addition, our ASR engines recognize most public names and abbreviations, and are able to render them correctly. Nowadays most ‘errors’ result from unclearly pronounced personal names and other not commonly known names.²

¹ This has been tested primarily with English and Spanish content, including many different accents and dialects. However, data from the AI solutions show that this should also apply to other commonly used languages such as French, German, Italian, Portuguese, Japanese and others.

² Misspellings of names, which cannot reasonably be assumed to be known by the AI engine and have not been added as additional context information, are not considered to be an error when calculating the level of accuracy.

How we achieve 99+% accuracy

There is a big difference between Clevercast and most other solutions for live AI captions. How does Clevercast attains 99+% accuracy, while other solutions are stuck between 70% and 90% accuracy? Below, we listed some of the things that lead to better live captions. Of course, it also helps that we specialize in multilingual live streaming, and develop and manage our own streaming infrastructure.

Use our real-time correction interface for 100% accuracy

Our correction interface allows you to edit the AI-generated captions in real-time, just before they are sent to the live stream (and translated into other languages). It lets you change words and move them to different lines for improved readibility.

Making these corrections is a simple task that requires no experience or training. Our intuitive interface allows anyone to edit the captions in a browser with mouse and keyboard. If desired, we can also provide you with professional correctors for your event.