Chapter 0 - The infamous hello world

This hello world actually won’t show the message "hello world" in the terminal 👅 Instead we’re going to print out information about the video, things like its format (container), duration, resolution, audio channels and, in the end, we’ll decode some frames and save them as image files.

FFmpeg libav architecture

But before we start to code, let’s learn how FFmpeg libav architecture works and how its components communicate with others.

Here’s a diagram of the process of decoding a video:

ffmpeg libav architecture - decoding process

You’ll first need to load your media file into a component called AVFormatContext (the video container is also known as format). It actually doesn’t fully load the whole file: it often only reads the header.

Once we loaded the minimal header of our container, we can access its streams (think of them as a rudimentary audio and video data). Each stream will be available in a component called AVStream.

Stream is a fancy name for a continuous flow of data.

Suppose our video has two streams: an audio encoded with AAC CODEC and a video encoded with H264 (AVC) CODEC. From each stream we can extract pieces (slices) of data called packets that will be loaded into components named AVPacket.

The data inside the packets are still coded (compressed) and in order to decode the packets, we need to pass them to a specific AVCodec.

The AVCodec will decode them into AVFrame and finally, this component gives us the uncompressed frame. Noticed that the same terminology/process is used either by audio and video stream.

Requirements

Since some people were facing issues while compiling or running the examples we’re going to use Docker as our development/runner environment, we’ll also use the big buck bunny video so if you don’t have it locally just run the command make fetch_small_bunny_video.

Chapter 0 - code walkthrough

TLDR; show me the code and execution.

  1. $ make run_hello

We’ll skip some details, but don’t worry: the source code is available at github.

We’re going to allocate memory to the component AVFormatContext that will hold information about the format (container).

  1. AVFormatContext *pFormatContext = avformat_alloc_context();

Now we’re going to open the file and read its header and fill the AVFormatContext with minimal information about the format (notice that usually the codecs are not opened). The function used to do this is avformat_open_input. It expects an AVFormatContext, a filename and two optional arguments: the AVInputFormat (if you pass NULL, FFmpeg will guess the format) and the AVDictionary (which are the options to the demuxer).

  1. avformat_open_input(&pFormatContext, filename, NULL, NULL);

We can print the format name and the media duration:

  1. printf("Format %s, duration %lld us", pFormatContext->iformat->long_name, pFormatContext->duration);

To access the streams, we need to read data from the media. The function avformat_find_stream_info does that. Now, the pFormatContext->nb_streams will hold the amount of streams and the pFormatContext->streams[i] will give us the i stream (an AVStream).

  1. avformat_find_stream_info(pFormatContext,  NULL);

Now we’ll loop through all the streams.

  1. for (int i = 0; i < pFormatContext->nb_streams; i++)
  2. {
  3.   //
  4. }

For each stream, we’re going to keep the AVCodecParameters, which describes the properties of a codec used by the stream i.

  1. AVCodecParameters *pLocalCodecParameters = pFormatContext->streams[i]->codecpar;

With the codec properties we can look up the proper CODEC querying the function avcodec_find_decoder and find the registered decoder for the codec id and return an AVCodec, the component that knows how to enCOde and DECode the stream.

  1. AVCodec *pLocalCodec = avcodec_find_decoder(pLocalCodecParameters->codec_id);

Now we can print information about the codecs.

  1. // specific for video and audio
  2. if (pLocalCodecParameters->codec_type == AVMEDIA_TYPE_VIDEO) {
  3.   printf("Video Codec: resolution %d x %d", pLocalCodecParameters->width, pLocalCodecParameters->height);
  4. } else if (pLocalCodecParameters->codec_type == AVMEDIA_TYPE_AUDIO) {
  5.   printf("Audio Codec: %d channels, sample rate %d", pLocalCodecParameters->channels, pLocalCodecParameters->sample_rate);
  6. }
  7. // general
  8. printf("\tCodec %s ID %d bit_rate %lld", pLocalCodec->long_name, pLocalCodec->id, pCodecParameters->bit_rate);

With the codec, we can allocate memory for the AVCodecContext, which will hold the context for our decode/encode process, but then we need to fill this codec context with CODEC parameters; we do that with avcodec_parameters_to_context.

Once we filled the codec context, we need to open the codec. We call the function avcodec_open2 and then we can use it.

  1. AVCodecContext *pCodecContext = avcodec_alloc_context3(pCodec);
  2. avcodec_parameters_to_context(pCodecContext, pCodecParameters);
  3. avcodec_open2(pCodecContext, pCodec, NULL);

Now we’re going to read the packets from the stream and decode them into frames but first, we need to allocate memory for both components, the AVPacket and AVFrame.

  1. AVPacket *pPacket = av_packet_alloc();
  2. AVFrame *pFrame = av_frame_alloc();

Let’s feed our packets from the streams with the function av_read_frame while it has packets.

  1. while (av_read_frame(pFormatContext, pPacket) >= 0) {
  2.   //...
  3. }

Let’s send the raw data packet (compressed frame) to the decoder, through the codec context, using the function avcodec_send_packet.

  1. avcodec_send_packet(pCodecContext, pPacket);

And let’s receive the raw data frame (uncompressed frame) from the decoder, through the same codec context, using the function avcodec_receive_frame.

  1. avcodec_receive_frame(pCodecContext, pFrame);

We can print the frame number, the PTS, DTS, frame type and etc.

  1. printf(
  2.     "Frame %c (%d) pts %d dts %d key_frame %d [coded_picture_number %d, display_picture_number %d]",
  3.     av_get_picture_type_char(pFrame->pict_type),
  4.     pCodecContext->frame_number,
  5.     pFrame->pts,
  6.     pFrame->pkt_dts,
  7.     pFrame->key_frame,
  8.     pFrame->coded_picture_number,
  9.     pFrame->display_picture_number
  10. );

Finally we can save our decoded frame into a simple gray image. The process is very simple, we’ll use the pFrame->data where the index is related to the planes Y, Cb and Cr, we just picked 0 (Y) to save our gray image.

  1. save_gray_frame(pFrame->data[0], pFrame->linesize[0], pFrame->width, pFrame->height, frame_filename);
  2.  
  3. static void save_gray_frame(unsigned char *buf, int wrap, int xsize, int ysize, char *filename)
  4. {
  5.     FILE *f;
  6.     int i;
  7.     f = fopen(filename,"w");
  8.     // writing the minimal required header for a pgm file format
  9.     // portable graymap format -> https://en.wikipedia.org/wiki/Netpbm_format#PGM_example
  10.     fprintf(f, "P5\n%d %d\n%d\n", xsize, ysize, 255);
  11.  
  12.     // writing line by line
  13.     for (i = 0; i < ysize; i++)
  14.         fwrite(buf + i * wrap, 1, xsize, f);
  15.     fclose(f);
  16. }

And voilà! Now we have a gray scale image with 2MB:

saved frame