Chapter 2 - remuxing

Remuxing is the act of changing from one format (container) to another, for instance, we can change a MPEG-4 video to a MPEG-TS one without much pain using FFmpeg:

  1. ffmpeg input.mp4 -c copy output.ts

It’ll demux the mp4 but it won’t decode or encode it (-c copy) and in the end, it’ll mux it into a mpegts file. If you don’t provide the format -f the ffmpeg will try to guess it based on the file’s extension.

The general usage of FFmpeg or the libav follows a pattern/architecture or workflow:

  • protocol layer - it accepts an input (a file for instance but it could be a rtmp or HTTP input as well)
  • format layer - it demuxes its content, revealing mostly metadata and its streams
  • codec layer - it decodes its compressed streams data optional
  • pixel layer - it can also apply some filters to the raw frames (like resizing)optional
  • and then it does the reverse path
  • codec layer - it encodes (or re-encodes or even transcodes) the raw framesoptional
  • format layer - it muxes (or remuxes) the raw streams (the compressed data)
  • protocol layer - and finally the muxed data is sent to an output (another file or maybe a network remote server)

ffmpeg libav workflow

This graph is strongly inspired by Leixiaohua’s and Slhck’s works.

Now let’s code an example using libav to provide the same effect as in ffmpeg input.mp4 -c copy output.ts.

We’re going to read from an input (input_format_context) and change it to another output (output_format_context).

  1. AVFormatContext *input_format_context = NULL;
  2. AVFormatContext *output_format_context = NULL;

We start doing the usually allocate memory and open the input format. For this specific case, we’re going to open an input file and allocate memory for an output file.

  1. if ((ret = avformat_open_input(&input_format_context, in_filename, NULL, NULL)) < 0) {
  2.   fprintf(stderr, "Could not open input file '%s'", in_filename);
  3.   goto end;
  4. }
  5. if ((ret = avformat_find_stream_info(input_format_context, NULL)) < 0) {
  6.   fprintf(stderr, "Failed to retrieve input stream information");
  7.   goto end;
  8. }
  9.  
  10. avformat_alloc_output_context2(&output_format_context, NULL, NULL, out_filename);
  11. if (!output_format_context) {
  12.   fprintf(stderr, "Could not create output context\n");
  13.   ret = AVERROR_UNKNOWN;
  14.   goto end;
  15. }

We’re going to remux only the video, audio and subtitle types of streams so we’re holding what streams we’ll be using into an array of indexes.

  1. number_of_streams = input_format_context->nb_streams;
  2. streams_list = av_mallocz_array(number_of_streams, sizeof(*streams_list));

Just after we allocated the required memory, we’re going to loop throughout all the streams and for each one we need to create new out stream into our output format context, using the avformat_new_stream function. Notice that we’re marking all the streams that aren’t video, audio or subtitle so we can skip them after.

  1. for (i = 0; i < input_format_context->nb_streams; i++) {
  2.   AVStream *out_stream;
  3.   AVStream *in_stream = input_format_context->streams[i];
  4.   AVCodecParameters *in_codecpar = in_stream->codecpar;
  5.   if (in_codecpar->codec_type != AVMEDIA_TYPE_AUDIO &&
  6.       in_codecpar->codec_type != AVMEDIA_TYPE_VIDEO &&
  7.       in_codecpar->codec_type != AVMEDIA_TYPE_SUBTITLE) {
  8.     streams_list[i] = -1;
  9.     continue;
  10.   }
  11.   streams_list[i] = stream_index++;
  12.   out_stream = avformat_new_stream(output_format_context, NULL);
  13.   if (!out_stream) {
  14.     fprintf(stderr, "Failed allocating output stream\n");
  15.     ret = AVERROR_UNKNOWN;
  16.     goto end;
  17.   }
  18.   ret = avcodec_parameters_copy(out_stream->codecpar, in_codecpar);
  19.   if (ret < 0) {
  20.     fprintf(stderr, "Failed to copy codec parameters\n");
  21.     goto end;
  22.   }
  23. }

Now we can create the output file.

  1. if (!(output_format_context->oformat->flags & AVFMT_NOFILE)) {
  2.   ret = avio_open(&output_format_context->pb, out_filename, AVIO_FLAG_WRITE);
  3.   if (ret < 0) {
  4.     fprintf(stderr, "Could not open output file '%s'", out_filename);
  5.     goto end;
  6.   }
  7. }
  8.  
  9. ret = avformat_write_header(output_format_context, NULL);
  10. if (ret < 0) {
  11.   fprintf(stderr, "Error occurred when opening output file\n");
  12.   goto end;
  13. }

After that, we can copy the streams, packet by packet, from our input to our output streams. We’ll loop while it has packets (av_read_frame), for each packet we need to re-calculate the PTS and DTS to finally write it (av_interleaved_write_frame) to our output format context.

  1. while (1) {
  2.   AVStream *in_stream, *out_stream;
  3.   ret = av_read_frame(input_format_context, &packet);
  4.   if (ret < 0)
  5.     break;
  6.   in_stream  = input_format_context->streams[packet.stream_index];
  7.   if (packet.stream_index >= number_of_streams || streams_list[packet.stream_index] < 0) {
  8.     av_packet_unref(&packet);
  9.     continue;
  10.   }
  11.   packet.stream_index = streams_list[packet.stream_index];
  12.   out_stream = output_format_context->streams[packet.stream_index];
  13.   /* copy packet */
  14.   packet.pts = av_rescale_q_rnd(packet.pts, in_stream->time_base, out_stream->time_base, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX);
  15.   packet.dts = av_rescale_q_rnd(packet.dts, in_stream->time_base, out_stream->time_base, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX);
  16.   packet.duration = av_rescale_q(packet.duration, in_stream->time_base, out_stream->time_base);
  17.   // https://ffmpeg.org/doxygen/trunk/structAVPacket.html#ab5793d8195cf4789dfb3913b7a693903
  18.   packet.pos = -1;
  19.  
  20.   //https://ffmpeg.org/doxygen/trunk/group__lavf__encoding.html#ga37352ed2c63493c38219d935e71db6c1
  21.   ret = av_interleaved_write_frame(output_format_context, &packet);
  22.   if (ret < 0) {
  23.     fprintf(stderr, "Error muxing packet\n");
  24.     break;
  25.   }
  26.   av_packet_unref(&packet);
  27. }

To finalize we need to write the stream trailer to an output media file with av_write_trailer function.

  1. av_write_trailer(output_format_context);

Now we’re ready to test it and the first test will be a format (video container) conversion from a MP4 to a MPEG-TS video file. We’re basically making the command line ffmpeg input.mp4 -c copy output.ts with libav.

  1. make run_remuxing_ts

It’s working!!! don’t you trust me?! you shouldn’t, we can check it with ffprobe:

  1. ffprobe -i remuxed_small_bunny_1080p_60fps.ts
  2.  
  3. Input #0, mpegts, from 'remuxed_small_bunny_1080p_60fps.ts':
  4.   Duration: 00:00:10.03, start: 0.000000, bitrate: 2751 kb/s
  5.   Program 1
  6.     Metadata:
  7.       service_name    : Service01
  8.       service_provider: FFmpeg
  9.     Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
  10.     Stream #0:1[0x101]: Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, 5.1(side), fltp, 320 kb/s

To sum up what we did here in a graph, we can revisit our initial idea about how libav works but showing that we skipped the codec part.

remuxing libav components

Before we end this chapter I’d like to show an important part of the remuxing process, you can pass options to the muxer. Let’s say we want to delivery MPEG-DASH format for that matter we need to use fragmented mp4 (sometimes referred as fmp4) instead of MPEG-TS or plain MPEG-4.

With the command line we can do that easily.

  1. ffmpeg -i non_fragmented.mp4 -movflags frag_keyframe+empty_moov+default_base_moof fragmented.mp4

Almost equally easy as the command line is the libav version of it, we just need to pass the options when write the output header, just before the packets copy.

  1. AVDictionary* opts = NULL;
  2. av_dict_set(&opts, "movflags", "frag_keyframe+empty_moov+default_base_moof", 0);
  3. ret = avformat_write_header(output_format_context, &opts);

We now can generate this fragmented mp4 file:

  1. make run_remuxing_fragmented_mp4

But to make sure that I’m not lying to you. You can use the amazing site/tool gpac/mp4box.js or the site http://mp4parser.com/ to see the differences, first load up the “common” mp4.

mp4 boxes

As you can see it has a single mdat atom/box, this is place where the video and audio frames are. Now load the fragmented mp4 to see which how it spreads the mdat boxes.

fragmented mp4 boxes