The challenges facing a new codec

I was asked recently to comment on how easy or difficult it might be to introduce a new video compression codec. Here's a summary of my opinion - you can read the full article by John Moulding here.

At present, the market is dominated by standards-based and open source solutions, including H.264 / AVC, VP8, HEVC and VP9. What challenges might be faced by a new entrant to the market?

1. Interoperability: One of the key motivations for the development of standards has been interoperability. Streaming provider X needs to be compatible with playback client Y. Building a critical mass of support for a new codec requires widespread adoption of encoders and decoders that interoperate with each other.

2. Performance: The development of standards such as H.264 or HEVC involved rigorous and thorough testing using agreed protocols for measurement of quality, bitrate, computational requirements, etc. Performance testing needs to be repeatable, such that multiple organisations can check performance and reach the same conclusions.

3. Intellectual property: An open source or standardised solution provides a degree of transparency about who might own intellectual property that is used in a compression solution. For example, implementors of the H.264/AVC standard can take a license to several hundred patents that may be essential to the standard, via the MPEG-LA patent pool.

4. Hardware support: When you play back or stream a video on a consumer device such as a cellphone or tablet, the computationally intensive process of decoding video is assisted by dedicated hardware, enabling smoother playback and better battery life. Support for existing formats such as H.264 and VP8 is built in to chipsets, operating systems and protocols.

A new codec technology has to overcome many hurdles if it is to be widely adopted. However, I am always interested in genuinely new and disruptive approaches to video compression. Is there a challenger out there to the 25-year-old codec model that has been the basis of video compression standards from MPEG-1 to HEVC?

Video compression - early history

Present day video codecs such as HEVC are based on fundamental concepts that go back a surprisingly long way. Key components such as motion compensated prediction, transform coding and entropy coding were developed in the decades before 1990.

I've written a short overview of the early days of video coding, when the building blocks were developed that would enable the emergence of mainstream digital video products in the 1990s. Click here to download the article, or download it from the Vcodex resources page here.

"En L'An 2000", early videophone concept, 1910

A nice example of 4K video encoded using VP9

Do you want to see 4K video in action? Phil Holland has put together a rather good demo clip, captured using a 6K RED camera and encoded using VP9 at 4K resolution (3840 x 2160 pixels).

The clip is 1 minute 55 seconds, 200MBytes in size, which works out at around 13Mbits per second. To view it for yourself, download the Zip file here, extract the .webm file from the Zip file, and drag it into a Chrome browser window. Of course, you'll need a 4K resolution monitor to get the full benefit and you may need a fast processor for smooth playback.

http://www.reduser.net/forum/showthread.php?111230-Google-VP9-4K-HD-Sample

MPEG-LA releases its HEVC patent pool license

On 29th September 2014, licensing company MPEG LA LLC released details of a patent portfolio license covering technologies related to the new High Efficiency Video Coding (HEVC) standard.

Video coding or video compression is an essential technology for video broadcast, streaming and playback. Released in 2013, HEVC is the latest in a series of industry standards for video compression that offers higher performance than its predecessors.

The license grants certain rights to patents held by 23 organisations, which MPEG LA believes to be essential to the practice of the new standard. Under the terms of the license, licensees pay a royalty to MPEG LA for end user products over a certain threshold (currently 100,000 units per year).

MPEG LA lists a total of 55 patents in the HEVC pool (see Figure), including 22 Korean patents and 12 US patents. We can expect to see the number of patents rise over the coming months, as patent applications filed during the development of HEVC are granted.

How to stream better quality video: Part 3 - Ultra High Definition, 4K and next generation video codecs

4K or Ultra High Definition (UHD) video has four times the detail of 1080p full HD. How do you deliver such high resolution video to a TV or mobile device over a limited bandwidth connection? Next-generation video codecs may provide the answer. This article, part 3 of “How to stream better quality video”, gives an overview of two new video codecs: High Efficiency Video Coding (HEVC or H.265) and VP9.

The High Efficiency Video Coding standard

H.264 is widely used amongst broadcasters and in the online video streaming industry, but this may change with the introduction of HEVC. If you want to get up to speed on the technical side of HEVC, the Vcodex website has material to get you started.

Key features of HEVC:

• Up to twice as efficient as H.264 - which means that you can send the same picture quality using much smaller files.

• Support for resolution up to 8K video (7680 × 4320) and frame rates up to 120 frames per second.
• Improved picture quality in terms of noise levels, color spaces and dynamic range.

This means you can shoot and produce video at the higher resolutions of UHD/4K without dramatically increasing the bandwidth required to broadcast or stream your video.

VP9

VP9 is Google’s answer to the problem of streaming 4K video. Offered as royalty free by Google, VP9 also claims to be significantly more efficient than H.264. Like HEVC, VP9 supports compression of videos at resolutions above HD as part of the WebM media format.

When can I start using 4K / Ultra High Definition?

You can try 4K on the web right now. For example, this Youtube playlist features 4K/UHD videos, which your display may or may not be able to handle. DivX's HEVC showcase includes examples of 4K videos. You'll need to download the DivX HEVC plugin to play back these clips.

Hardware support for HEVC is increasing, with a number of demonstrations at this year's IBC show. A reference hardware implementation of VP9 is available for chip developers.

New codecs - a performance comparison

We ran a comparison test to find out how HEVC and VP9 codecs performed compared with H.264 using a 720p video clip downloaded from the Xiph website. Following standard test guidelines for video quality assessment, we carried out subjective video quality tests with 10 non-expert users.

Figure 1: File size versus perceptual quality for H.264, VP9 and HEVC codecs

Figure 1 shows the variation of perceptual video quality measured as mean opinion score (MOS), with file size for different codecs. At the higher end of the quality scale (i.e. lower compression), both HEVC and VP9 achieve a similar viewing quality to H.264 with a 40-45% reduction in file size. At lower quality ratings, the gains of HEVC and VP9 are smaller.

This implies that you should see the most benefit from HEVC and VP9 codecs at high quality / low compression settings.


We hope the three-part series of the blog "How to stream better quality video" has been useful. To find out more about video compression and streaming, please visit the Vcodex website here.

About the authors:

Iain Richardson is an internationally known expert on video compression and the author of several books, including The H.264 Advanced Video Compression Standard: http://www.vcodex.com/books.html

Abharana Bhat specialises in video compression, media streaming and video quality assessment.

Copyright (c) Vcodex Limited 2014-2015

How to stream better quality video: Part 2 - Adaptive streaming

by Iain Richardson and Abharana Bhat
You want to stream a video clip to several customers. One customer has a high bandwidth leased line connection, while another is trying to view your clip on a mobile device using a poor-quality 3G signal. How do you make sure they each get the best possible video quality? Adaptive streaming may be the answer.

Video streaming sends your video (source) to play back on a computer or mobile device (client), using an internet connection to transport the video. Streaming video is a complex process involving many different components. In this series of articles, we help you understand the basics of streaming video and the various factors that can make a big difference to the quality of your video clip.

In our first article, we looked at the basics of streaming video and discussed good and bad practice. In this article, we look at how to stream video effectively over networks with varying connection speeds.

1. Adaptive streaming: the basics

The general concept of adaptive streaming is shown in Figure 1.

a. Multiple versions:

Starting with a high quality video source such as a High Definition video clip, a video encoder produces versions at multiple qualities and bitrates, for example:

1. An HD version at 2Mbps 
2. A Standard Definition (SD) version at 1Mbps 
3. A very low resolution version at 50kbps. 

Each of these versions is split into small chunks, typically each a few seconds long, and stored on a web server ready for streaming.

b. Index file:

The server maintains an index or playlist file which lists all the media chunks, including the bitrate of each stream and the starting position of each chunk.

c. Streaming to the client:

When the client (the user's software application) requests playback of the video stream, the server returns the index file to the client.

The client requests chunks at the most suitable bitrate, based on considerations such as the available connection bandwidth, playback delay and device capabilities. The server sends each requested chunk in order.

The result: video and audio playback that adapts to suit the network conditions.


2. Examples
Example 1:

The client is initially connected to the internet via a good quality WiFi connection. Streaming starts at 2Mbps and the user sees good quality, HD video with low delay.

The WiFi signal is lost and the client switches to a 3G mobile connection. Streaming switches to 1Mbps, then to 50kbps. The user sees continuous video playback. The video quality steps down to SD then to lower resolution because of the drop in network bandwidth.

Example 2:

The client is configured to start playback as quickly as possible. It automatically requests the lowest quality stream and steps up if the bandwidth is good enough. The user sees basic quality playback very soon after pressing "play". After a few seconds, the video quality improves progressively without any delays or breaks in playback.

Example 3:

Click the link below and play each video to see adaptive streaming in action. Click on the "HD" button on the player to select one of 5 different bitrates, or select "Auto" to allow the stream to adapt according to your network connection.

Click here: http://www.beamshare.com/adaptive-streaming-demo

3. Practical considerations
Adaptive streaming format. There are a number of choices of adaptive streaming methods, including:

• HTTP Live Streaming (HLS): Originally specific to Apple devices, now available on other platforms. 
• Microsoft Smooth Streaming: Developed for Silverlight and Windows Phone. 
• Adobe HTTP Dynamic Streaming (HDS): Suitable for streaming to devices compatible with Adobe Flash or Air. 
• MPEG Dynamic Adaptive Streaming (DASH): An open standard for adaptive streaming developed by the MPEG standards committee. 

Each of these methods has pros and cons. When selecting an adaptive streaming technology, it is important to consider:

• Availability and cost of encoding software tools 
• Compatibility with your target clients. 

Stream count. How many streams should be created for each video file? More streams will give the client more flexibility to choose the best stream based on the available bitrate and display capabilities. However, each stream needs to be converted (transcoded) from the original and stored on a server. This may increase computing and storage costs.

Chunks and key frames. How many seconds per chunk? Each chunk starts with a key frame, i.e. a video frame encoded without any prediction from other frames. Key frames require significantly more bit rate to transmit than predicted ("inter") frames. Smaller chunks make it possible for the client to switch streams more frequently, leading to faster startup and smoother adaptation. However, smaller chunks require more frequent key frames, leading to poorer transmission efficiency. A good compromise may be to use smaller chunks early in the video stream, increasing the size later on.


4. Going further Part 3 in this series will compare the latest video codecs in depth.

Copyright (c) Vcodex Limited / Onecodec Limited, 2014

How to stream better quality video: Part 1 - Basics and good practices

by Iain Richardson and Abharana Bhat

You're trying to stream a video to a work colleague who is complaining that it's very, very slow to load. Or perhaps you are trying to embed a video clip in your web page and are not satisfied with the quality of the result. Is there anything that can be done to improve the situation?

Video streaming sends your video (source) to play back on a computer or mobile device (client), using an internet connection to transport the video. Streaming video is a complex process involving many different components. In this series if articles, we help you understand the basics of streaming videoand the various factors that make a big difference to the quality of your video clip.

In this first article, we look at the basics of streaming video and discuss good and bad practice.

1. Streaming video: the basics

These are the basic components involved in streaming video from source to destination.

Source - A video clip you've recorded or created.

Editing and compression - Once you have edited your media file, it is converted into a form suitable for sending over a network. This involves compressing or encoding the video and audio information.

Server - Stores the media file and sends it to the client on demand.

Network - Transports (streams) the video from the server to the client.

Client - Receives the media file, extracts (decodes) video and audio information and plays back the video clip.  

"Streaming" means that the client can start playing the video clip before it's fully downloaded, as soon as there is enough data to start decoding and playback. In this screenshot, the client has received and stored enough data to decode and play the clip, even though the whole clip hasn't been downloaded yet. The video clip plays smoothly, while the client continues to download and store ("buffer") the remaining data.

Figure 2: Streaming playback

2. Compression and codecs

2.1 The basics of compression

A codec is an encoder and a decoder. An encoder compresses audio or video so it takes up less disk space. A decoder extracts audio or video information from the compressed file. Video and audio compression is a complex technical process, but the basic aim of a codec is quite straightforward:

(a) Reduce the size of the compressed media file as much as possible, but...(b) Keep the quality of the decoded audio and video as good as possible.

Most codecs work by removing information that is not noticeable to the viewer / listener and by exploiting similarities in the audio / video data.

Example 1: Small, subtle variations in texture will tend to be "masked" by strong shadows and edges. A video codec can exploit this by keeping stronger image features and throwing away some of the fine detail.
Example 2: A video clip contains 25 or more frames per second. Instead of sending the entire frame every time, a video codec can save a lot of data by only sending the differences between frames. 

Any device that records or plays back video contains a codec, either as part of a chip (hardware) or in software. When you record video on your mobile handset, watch anything on TV, or play video on the web, you're using a codec.

2.2 What's in a compressed media file?

Video or audio files on your computer or mobile device are usually stored in a compressed form. Common media file formats have extensions such as .mp4, .mov, .avi, etc.

There are three main components of a compressed media file: compressed video, compressed audio and the structure of the file itself.

The compressed video will typically use one of a number of standard video codec formats, such as MPEG-2, MPEG-4 or H.264, VP8, etc. The compressed audio will use one of a number of audio codec formats, such as MP3 or AAC. The compressed video and audio are typically packaged into a container format, i.e. a special type of file that is designed to carry video and audio along with other information known as metadata. Examples of container formats include MP4, MOV, MKV and AVI.

Many combinations of video codec, audio codec and container format are possible. One challenge you may face is that not all combinations are supported by every computer or mobile client.

3. Important concepts

Bitrate and bandwidth - The information-carrying capacity of a network connection is commonly known as bitrate or bandwidth and is measured in giga/mega/kilobits per second. A 3G mobile connection might be capable of carrying a few tens of kilobits per second; a home broadband connection may support hundreds of kilobits up to a few megabits per second; a fibre or leased line may be capable of carrying tens of megabits or more per second.

To stream video smoothly, it is important that the bitrate or bandwidth of the network connection is greater than the bitrate of the streaming media file. Even after compression, video and audio can require a significant bitrate capacity.

Video resolution - Most cameras, even on mobile devices, are capable of recording video in High Definition (HD) resolution. However, significantly more bandwidth or bitrate is required to handle HD video compared with lower resolution video.

A single frame of "full HD" or "1080p" video (1920x1080 pixels) has more than twice the pixels of a single frame of "720p" video (1280x720 pixels), which in turn is more than twice the size of a single frame of "Standard Definition" or "SD" video (720x576 pixels or smaller).

Transcoding - If your video/audio material is not suitable for streaming, you may need to transcode it. This may involve:
Changing resolution: For example, resample from 1080p HD down to 720p or SD.
Changing codec or container: For example, change the video codec from MPEG-2 to H.264, or change the file format (container) from AVI to MP4.
Improving streaming performance: For example, reorganising the audio and video samples in the container file, or converting to an adaptive streaming format.

4. Tips for better streaming performance

• Keep the camera as steady as possible. More movement means more data after compression, which means that you will need more network bandwidth to achieve good quality video playback.

• Use good lighting where possible. Poor lighting can produce increased camera “noise” which leads to more data after compression. Here's an example: the first scene is well lit, the second is dark and "grainy" in appearance. The poorly-lit scene will require more network bandwidth to transmit.

• Choose your video codec carefully. Older codec formats such as MPEG-2 are less efficient, i.e. the coded video file will be considerably larger than a newer codec such as H.264. The latest codecs, such as HEVC and VP9, may give the best compression but there is limited software and playback support for these codecs at present (see Part 3).

• Use a lower resolution (e.g. SD rather than HD) or consider adaptive streaming (see Part 2).

• Be aware that video content has a significant effect on the bandwidth required for streaming and/or on the video playback quality. For example, if two SD video clips are both compressed at the same bitrate (400 kilobits per second). The clip with more movement will result in the video quality looking worst even though the video resolution and the streaming bitrate are the same.

• Does your editing software have options such as “slow” or "multi-pass" encoding? If so, consider selecting these options. Processing the file will take longer because the software will spend more time to choose the best compression options for each video frame. However, the end result may be better video quality and/or a smaller compressed file.

5. Going further

Part 2 of this series of articles will cover more advanced topics such as Adaptive Streaming. Part 3 will compare the latest video codecs in depth.

To find out more about video compression and streaming, visit Iain Richardson's website: vcodex.com

Copyright (c) Vcodex Limited / Onecodec Limited, 2014