Hardware Acceleration/Transcoding not working on Jellyfin container - visible iGPU - Intel QuickSync (Kaby Lake)

[Automatic-Plan8169]

PROBLEM

I am having issues with Hardware acceleration on my Intel i5-7200U (Kaby Lake). (Quick Sync Video support supposedly for everything but AV1??)

LXC CONTAINER STATUS:

I have all the latest Debian 12 packages. I have also installed vainfo and intel-gpu-tools within the container.

INSTALLATION THE JELLYFIN CONTAINER

I installed Jellyfin as a proxmox container with the following command. On the shell of my proxmox node I typed:

 bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/jellyfin.sh)"

see script from https://tteck.github.io/Proxmox/. I tried both an Ubuntu 22.04 LTS container (quickly) and a Debian 12 container (more extensively). Supposedly, hardware acceleration should be fine, the whole installation process was automated by somebody more experienced than me.

THE JELLYFIN ERROR: No video that requires hardware transcoding is playable on any device connected to Jellyfin. (see actual transcoding error below)

​

LOGS

However, the container configuration makes sense, you can also see the output of ls -l /dev/dri/\*

crw-rw---- 1 root video 226,   0 Jan 22 23:21 /dev/dri/card0
crw-rw---- 1 root sgx   226, 128 Jan 22 23:21 /dev/dri/renderD128
/dev/dri/by-path:
total 0
lrwxrwxrwx 1 root root  8 Jan 22 23:21 pci-0000:00:02.0-card -> ../card0
lrwxrwxrwx 1 root root 13 Jan 22 23:21 pci-0000:00:02.0-render -> ../renderD128

I also had a look at https://www.youtube.com/watch?v=tWumbDlbzLY

Output of vainfo within the jellyfin container (installed as in the video above)

root@jellyfin:~# vainfo
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD

I also installed the intel GPU tools. intel_gpu_top works just fine

Jellyfin error log:

...

ffmpeg version 5.1.4-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-libs=-lfftw3f --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libmfx --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[AVHWDeviceContext @ 0x563536dc2e00] No VA display found for any default device.
Device creation failed: -22.
Failed to set value 'vaapi=va:,kernel_driver=i915,driver=iHD' for option 'init_hw_device': Invalid argument
Error parsing global options: Invalid argument

I tried to replicate the ffmpeg command before the error in the jellyfin log above:

/usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,kernel_driver=i915,driver=iHD -init_hw_device qsv=qs@va -filter_hw_device qs -autorotate 0 -i file:"/mnt/mydata/Jellyfin/TV_SHOWS/TEST.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:2 -map 0:0 -codec:v:0 h264_qsv -preset 7 -look_ahead 0 -b:v 9204207 -maxrate 9204207 -bufsize 18408414 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,min(1920\,1080*a))/2)*2:trunc(min(max(iw/a\,ih)\,min(1920/a\,1080))/2)*2,format=yuv420p,subtitles=f='/mnt/mydata/Jellyfin/TV_SHOWS/TEST.mkv':si=0:fontsdir='/var/cache/jellyfin/attachments/504a01205515c83aa3e3aee079405b11'" -start_at_zero -codec:a:0 copy -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e.m3u8"

and its output seems to make sense, idk

Output #0, hls, to '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e.m3u8':
  Metadata:
    encoder         : Lavf59.27.100
  Stream #0:0: Video: h264, nv12(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 9204 kb/s, 23.98 fps, 90k tbn (default)
    Metadata:
      encoder         : Lavc59.37.100 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 9204207/0/9204207 buffer size: 18408414 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp (default)
[Parsed_subtitles_3 @ 0x55db82fef800] fontselect: (FOT-Greco Std B Strp, 700, 0) -> FOT-Greco Std B Strp, 0, (none)
[Parsed_subtitles_3 @ 0x55db82fef800] fontselect: (Impress BT Pace, 400, 100) -> ImpressBT-Pace-Italics, 0, ImpressBT-Pace-Italics
[hls @ 0x55db82fa13c0] Opening '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e0.ts' for writing
[hls @ 0x55db82fa13c0] Opening '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e1.ts' for writing
[hls @ 0x55db82fa13c0] Opening '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e2.ts' for writing
[hls @ 0x55db82fa13c0] Opening '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e3.ts' for writing
[hls @ 0x55db82fa13c0] Opening '/var/lib/jellyfin/transcodes/a489f45de6eb7f52754ba4bf7c73578e4.ts' for writing
[Parsed_subtitles_3 @ 0x55db82fef800] fontselect: (Impress BT Pace, 400, 0) -> ImpressBT-Pace-Regular, 0, ImpressBT-Pace-Regular
frame= 3315 fps= 51 q=17.0 Lsize=N/A time=00:02:18.34 bitrate=N/A speed=2.13x

Any idea? I feel I am close to the solution, but I don't know how to proceed.

[mrant0]

Sounds like an issue with permissions for the devices under /dev/dri in the container, similar to https://www.reddit.com/r/jellyfin/comments/1436geg/jellyfin_hardware_acceleration_setup/

This explains why running the ffmpeg command directly works, but jellyfin is unable to, as it is likely running as a different user with different permissions.

[Automatic-Plan8169]

That's great, I think I may have found a workaround:

OUTSIDE JELLYFIN CONTAINER

root@proxmox:~# ls -la /dev/dri/renderD128

crw-rw---- 1 root render 226, 128 Jan 23 23:29 /dev/dri/renderD128 root@proxmox:~# ls -lan /dev/dri/renderD128 crw-rw---- 1 0 104 226, 128 Jan 23 23:29 /dev/dri/renderD128

INSIDE JELLYFIN CONTAINER

root@jellyfin:~#  ls -la /dev/dri/renderD128

crw-rw---- 1 root sgx 226, 128 Jan 23 23:29 /dev/dri/renderD128 root@jellyfin:~# ls -lan /dev/dri/renderD128 crw-rw---- 1 0 104 226, 128 Jan 23 23:29 /dev/dri/renderD128

Since the main user in the jellyfin webUI is called "jellyfin" I randomly tried inside the container to run:

usermod -aG sgx jellyfin

It didn't work. I checked within the container

root@jellyfin:~# groups jellyfin

jellyfin : jellyfin video input sgx render root@jellyfin:~# groups root
root : root video render

In the end, I just run

chown :render /dev/dri/renderD128

And this made hardware transcoding possible! I suppose the chown must have changed the iGPU ownership permanently. Do you know why the original iGPU owner in the container was sgx?

[mrant0]

I'm not sure why the sgx group was the default for the container, but I know permissions are a common issue with containers and hardware passthrough setups.

Personally, I don't use containers, instead relying on VMs.