The Essential Guide to Closed Captions

Closed captions can be turned on or off by viewers. Often, video publishers and/or viewers can usually customize them to a certain extent, like adjusting the font size, color and background.

Open captions are burned into the video and cannot be turned off or customized. Since open captions have to be encoded as part of the video, they are less suitable for use in real-time.

Offline captions (or “captions for Video on-Demand”) are generated and added to videos after the content has been recorded or produced. Note that most existing captioning guidelines are designed for offline captions.

Real-time captions (or “live captions”), on the other hand, are generated instantaneously as the audio is being spoken or broadcasted, which makes it more difficult to perfect the accuracy and timing of the captions.

SDH Subtitles or SDH captions must include sound effects, speaker IDs and other non-speech elements. By comparison, standard captions (or speech-only captions) only contain the spoken words.

In most parts of the world, the terms closed captions and subtitles are used interchangeably. In the U.S., however, the term ‘closed captions’ often refers to SDH captions, while ‘subtitles’ refers to speech-only captions.

Automatic Speech Recognition (ASR) systems use machine learning techniques to analyze and interpret audio signals, extract the spoken words and transcribe them into textual form. This way, speech is converted into text.

Next, an intelligent rendering engine ensures that the transcription is divided into captions, which are in sync with the audio and video.

For offline captions, the use of ASR is increasingly becoming the norm. Major streaming content providers such as Netflix, Amazon Prime Video and Hulu all use AI to generate captions.

ASR is also increasingly being used for real-time captioning. Recent major advances in AI have made 99% accuracy the norm for ASR as well. Depending on the type of live stream, budget and specific captioning requirements, ASR – with or without real-time correction by a human editor – may be a better option than human captioning.

There are several protocols for storing and displaying captions in multimedia content. The most popular formats for Web video are WebVTT (Web Video Text Tracks) and SRT (SubRip Subtitle), which are used for both offline and real-time captioning.

In case of a Video on-Demand, WebVTT or SRT files can be uploaded to the VoD platform. Most platforms allow you to review the closed captions before publishing the video, allowing you to check timing and accuracy.

When someone watches the video, the video player accesses a local version of the WebVTT or SRT files on the global Content Delivery Network (CDN) used by the platform. In a live stream, this process happens automatically.

There are also formats and protocols for more specific uses such as TTML, IMSC and CEA-608/CEA-708. For more info, see our closed caption format guide.

The accuracy of captions in VoD content depends on the quality of the captioning service employed and the expected turnaround time. In theory, perfect quality closed captions are possible here.

For streaming video content, the use of ASR is increasingly becoming the norm. Major streaming content providers such as Netflix, Amazon Prime Video and Hulu all use AI to generate captions. To avoid errors or inaccuracies, the resulting captions are still often checked and corrected by trained professionals. Most captioning services also have quality control measures in place.

The real-time nature of a virtual meeting makes it difficult to use ASR technology. Because every word has to be translated immediately, the accuracy of an ASR engine will decrease due to a lack of context. As each sentence progresses, an ASR engine will have more context, allowing for better transcription. Therefore, rolling captions could be an option: this allows already visible text to still be corrected. But this usually does not lead to a good user experience.

At present, human-generated captions by professionals with a shorthand keyboard (stenotype) seem to be the best solution for virtual meetings. They are often better at creating literal captions with sufficient accuracy.

Because of the one-to-many nature of live streaming, there is a short delay between when something happens and when viewers get see it. This is called the ‘latency’ of a live stream. It represents the time it takes for the video and audio data to be captured, encoded, transmitted, and finally decoded and displayed by the video player.

The latency of a live stream varies depending on the streaming protocol and the requirements of the event. Achieving lower latency often involves trade-offs with factors like video quality, reliability and scalability.

By using the live stream’s latency, the accuracy of ASR-generated captions can be increased significantly. For starters, it allows longer audio fragments to be forwarded to the ASR system, so it has more context and will transcribe more accurately. Additionally, it allows for a human to correct the ASR captions before they are displayed. In practice, this makes it possible to achieve a higher quality than human-generated captions at a lower cost.

When using the HLS or MPEG-DASH protocol, new WebVTT or SRT files are continuously created and made available via a CDN. A video player ensures that they are displayed synchronously with the video and audio streams. Since the player needs to be able to buffer part of the stream, to ensure smooth streaming, the live stream is shown with a certain delay. Since this is done through standardised protocols, different players can be used to display a live stream with captions.

Captions can be added to the live stream in two ways, depending on the platform:

• Sent as part of the broadcast, which is usually done using the CEA-608/CEA-708 protocol. But this approach has many drawbacks: it is very limited, outdated and most streaming services do not support it.
• Added by the streaming platform through real-time AI and/or manual transcription. This is already happening in the vast majority of cases.