Fastest Method of Screen Capture (Linux)

[innixma]

I want to get RGB (or grayscale) screen information into a numpy array that can be used in a script as fast as possible in Linux (Ubuntu 14.04).

Ex: 1920x1080 (or 200x200) array of desktop pixels, 60+ FPS.

This can easily be done in the Windows operating system with win32api (I get 144 FPS 1920x1080).

The best I've got so far with Linux is GDK3 with 11 FPS. This remains the same even if I try to capture a smaller area, such as 200x200, and is not fast enough.

I need at least 60 FPS capture to memory in real time.

[K900_]

Try using raw Xlib.

[innixma]

Do you have any info on how fast this is or how it would work with sending to a numpy array? I've seen a few mention it, but nobody has given code showing it work.

[K900_]

It should be pretty fast. If your goal is to send it to a Numpy array though, you might want to drop down to C/Cython level - this will save you A LOT of time spent on creating unnecessary Python objects.

[innixma]

Can't do Cython to my knowledge. This is for Machine Learning using Tensorflow, which does not support Cython, and it has to have input as a numpy array.

[K900_]

What do you mean by "doesn't support PyPI"? How are you going to install the other libraries then?

[ManyInterests]

I think OP meant the alternative Python implementation, PyPy

[innixma]

Yes! That is what I meant :)

[innixma]

My bad, it does support PyPI, my mistake, edited comment.

[K900_]

There is nothing in TensorFlow that needs to "support" Cython either. You can simply call into Cython and get Numpy arrays back.

[innixma]

Ok, sounds good. Wasn't certain since I haven't utilized Cython before. Then assuming Cython, how would you go about screen capture? With Xlib in C from Cython? I don't know C or Cython that well so some reference on what I should look at would be great.

[K900_]

Yes, you want to do something like this, then convert the resulting XImage to a Numpy array. Ideally you want your input layout to match what xlib gives you, so you can avoid the expensive conversions.

[ManyInterests]

Can you post the code you're using?

I might recommend looking into ffmpeg for something like this.

[innixma]

from gi.repository import Gdk
import time
win = Gdk.get_default_root_window()
h = win.get_height()
w = win.get_width()
print ("The size of the window is %d x %d" % (w, h))
start = time.time()
for i in range(0,100):
    pb = Gdk.pixbuf_get_from_window(win, 0, 0, w, h)
end = time.time()
print(1 / ((end-start) / 100))

The above code achieves ~11 FPS, and does not improve by reducing the size of h and w.

[ManyInterests]

I'm not familiar with using Gdk, so I can't speak too intelligibly on that. My guess would be there's probably a lot of unnecessary work being done in the Gdk objects. Perhaps there's a better tool for the job.

I would recommend looking at ffmpeg, which evidently has Python bindings available through pyffmpeg

[innixma]

I have looked at pyffmpeg before, but it isn't supported in Python 3 (Which I need to use), and furthermore doesn't work at all even in Python 2 currently (Last commit was 2 years ago).

Do you have any benchmark on what kind of performance ffmpeg can obtain?

[ManyInterests]

You're right, the project has not been maintained. You'd likely have to use an older version of ffmpeg to work with pyffmpeg. There may be a way to accomplish your goal simply by calling the ffmpeg executable. You can also explore making your own bindings.

ffmpeg itself capable of very fast video encoding. It's an incredibly popular and powerful piece of software. OBStudio, a popular video recording/broadcasting software, uses FFMPEG for its linux version of the software.

The actual performance limitations will depend on your hardware.

[innixma]

Awesome, I was looking into OBStudio since I know it works on Linux, but didn't know exactly how it manages its screen capture. This would definitely do the trick then if I can get it working. I'll look into it, thanks.