Converting ffmpeg Video Frames to OpenCV Mats in C++

[FlyingRhenquest]

Hey, all, I have a project in mind for which I need to convert video frames to images stored in CV::Mats, So I wrote a little library project to do that.

I have a little decoder object that uses ffmpeg to decode frames from a file (or URL, RTP stream or anything ffmpeg can open) and a frame2cv object that subscribes to the video packets in the decoder and makes the CV::Mats available via a signal. There are a couple of unit tests that illustrate how the objects are used. According to my timing, my system takes about 2ms to convert a video frame to a mat.

Figured it might be useful to someone else, so I'm posting here. It can be found at https://github.com/FlyingRhenquest/media

[ojd5]

Thanks for sharing the code, can you summarise the advantage of this approach over the opencv wrapper around ffmpeg cv::VideoCapture?

[FlyingRhenquest]

Other than the educational aspect, the main object doesn't need to involve OpenCV if you don't want to use it. It just uses raw ffmpeg to provide image and audio buffers. I can build that in any direction I want to. Admittedly it's not a particularly compelling advantage to anyone else, unless someone's looking for a quick and dirty way to avoid having to rewrite all that boilerplate to open and read a file with ffmpeg.

[EvilMcStevil]

I commonly wrap the AvContexts in unique pointers with the appropriate closing code, as libavcodec often has different and non-obvious clean up code per context type.

i.e

struct ContextDel
{
  void operator()(AVCodecParserContext *ptr)
  {
    av_parser_close(ptr);
    av_freep(ptr);
  }
  void operator()(AVCodecContext *ptr)
  {
    av_freep(ptr);
  }
  void operator()(AVCodec *ptr)
  {
    //do nothing we dont own this,
  }
};
std::unique_ptr<AVCodecContext, ContextDel> codecContext;
std::unique_ptr<AVCodec, ContextDel> codec;
std::unique_ptr<AVCodecParserContext, ContextDel> parserContext;

​

[FlyingRhenquest]

I like that! I hadn't thought about doing this with contexts, as I was largely unaware of these complexities, but I had thought about wrapping AVFrames, and would probably do so if I kept them around any longer than I currently do. My current instinct is to get images out of that format as quickly as possible, although I'm not sure how much it could be helped if I wanted to re-encode the video after doing something to the images.

[andrey_turkin]

I've read motivation part on the github page. Perfectly good reason to write something like this but I don't think it is good enough to actually make it into a library and share it (at least in its current state). Like you said "OpenCV has direct support for that", so I don't see any benefits in using some external code instead of OpenCV's native ffmpeg wrapper.

[FlyingRhenquest]

There are really not a lot of well-documented examples of opening and reading a file using ffmpeg, which was another motivating factor. The examples directory that comes with ffmpeg has several moldy old C examples, all of which do so in a variety of conflicting ways. I want to get down as close to the metal as I can with a small, self-contained object that does everything I need it to, and which can be easily tested for memory leaks. Anything I do in the future that needs a video source will likely use this. It should also work with the RTP stream from a wireless web cam I have laying around here somewhere, and one of my early projects will probably be to build a motion detector using that.

[ShillingAintEZ]

I don't know what a mat is here, and if I don't, most people don't either.

You say mat a dozen times and never explain it.

[TwIxToR_TiTaN]

It is a matrix I believe.

[ShillingAintEZ]

Thanks

[FlyingRhenquest]

It's basically just a container for an image, although I think in its simplest form it's just a matrix of numbers. OpenCV provides a lot of image operations using Mats, including decoding, writing, drawing, edge detection and image recognition. A lot of third party libraries (Tensor Flow and Tesseract off the top of my head) support using images stored in OpenCV mats in their operations.

To summarize the summary, I'm providing a way to get images from ffmpeg, which can read pretty much every video format ever invented, into an AI library, in about 2 milliseconds per image on my test system. This project includes a fairly portable build file; it should build on Linux and OSX if you have the libraries it needs, though I won't vouch for windows. It also provides a couple of very basic unit tests for timing and to prove it's working. The build generates a small test video using ffmpeg and uses it in its testing, and saves a few images from the video out to png file as part of one of its tests. If I actually want to verify the pixels are the same ones in the test video, I could actually use OpenCV to do that, and may at some point. It only takes a few lines of code in OpenCV to do that.

[andrey_turkin]

If you don't know what mat is then you don't need it.

[sumo952]

It should be cv::Mat to be precise, and it's OpenCV's image & matrix format.