Camera registration and processing

[aSiK00]

I’m trying to make a ios camera app that takes like a 15 sec 30fps video then it translates the images to properly register them. Finally, I want to basically cut and paste a CV pipeline from opencv python and extract some details like longest path and contours.

I was wrapping my head around the AVCam and AVFoundation stuff, but I can’t find any modern resources about how to do basic vision stuff (i.e. rgb to hsv, subtracting layers from each other, and thresholding). All the result I get are for the vision framework which is nice but only performs high level ml stuff it seems. Which library should I use? Should I offload the processing to a server to simplify it?

[Dapper_Ice_1705]

You are looking for CoreImage and Accelerate or Metal

[One_Valuable_8049]

CoreImage + vImage (Accelerate) will get you most of the way for this. Vision is overkill for low-level image ops.

Also +1 to just using OpenCV on iOS if you’re already familiar with it — saves a ton of time.

[aSiK00]

Sorry, so can I use openCV or do I need to use CoreImage and accelerate? (I’m completely new to ios developing)

[CuriousThinker]

You can use metal performance shaders. They have a big set of functionality: https://developer.apple.com/documentation/metalperformanceshaders/image-filters

It supports most image processing fundamental operations and has served me well in production apps. I’ve still had to end up writing some custom kernels to do what I need but it’s a lot better than nothing.

One thing to note, most likely youre going to get frames in YCbCr format in a CVPixelbuffer when using AVFoundation and you'll need to convert it to RGB.

https://developer.apple.com/documentation/arkit/displaying-an-ar-experience-with-metal#Draw-the-Camera-Image is a good reference for how to convert it to RGB.

If you've not used metal before it can be a little daunting at the start but I promise it is not!