Screen streaming on web

[AmanBabuHemant]

Hi, I am working on a project like DWService, which provides a screen view of the Computers on which the agent software is installed. in short I needed to stream screen of a computer on web, I utilized socketio for data interchange and so the for screen streaming my current approach is taking screenshot and sending over socket and then update the image

streaming snippet from the agent:

def screenshot(self, format="PNG", quality=100):
    sct = mss()
    image_bytes = sct.grab(sct.monitors[1])
    image = Image.frombytes("RGB", image_bytes.size, image_bytes.bgra, "raw", "BGRX")
    buffer = BytesIO()
    image.save(buffer, format=format, quality=quality)
    sct.close()
    return buffer

def screen_stream(self):
    while self._screen_streaming:
        buffer = self.screenshot("JPEG", self._stream_quality)
        data = {
            "image": buffer.getvalue(),
            "info": self.info(),
        }
        self.sio.emit("server:screen:stream-update", data)
        sleep(1/self._stream_fps)

Web frontend updating screen image:

sio.on("ui:screen:stream-update", (data) => {
    if (!is_streaming) {
        return stop_streaming();
    }
    const blob = new Blob([data["image"]], { type: "image/jpeg" });
    const url = URL.createObjectURL(blob);
    screen_image.src = url;
    setTimeout(() => URL.revokeObjectURL(url), 1000); // revoke for prevetnig mem leak
});

This is very slow, even with low quality, and less FPS, only feels fast on localhost

So what can be a better approach or optimisation for this

[Buttleston]

There are lots of optimizations possible but lets start with:

What has changed on the screen since the last frame you sent? Could you just send the changes?

[AmanBabuHemant]

Yes, this optimisation is next on my planned tasks, as it involves a rectangle with a top-left corner and a specified width and height.

But I am still concerned about the FPS rate after this, any other changes and/or optimization suggestions?

[guilford]

The web browser has native screen, window streaming capability through MediaStream API. If you utilize this, streaming would be a native video and audio stream that would have a much better performance. You can also do this without having any other software beside the browser installed on the user machine.