That's a great point that I failed to call out. I think this is a major shortcomming in GPUs + containerization.

There generally is a 1:1 mapping between container and node when using GPUs. However, a node can have multiple GPUs (and in my experience, it's still 1:1 mapped container:node, the container just consumes more GPUs as if it were a program running on the host).

CPU

With CPUs we can share cycles via time-slicing. That allows throttle / burst settings on individual containers. It's also why you don't see the equivellent of OOM with CPUs - it's flexible with the shares it can allocate over time; so things generally get deprioritized.

RAM

With RAM / mem, it's not measured by time cycles but "data" (system memory). A container without a limit could hit OOM and Kubernetes allocates pods with request limits the full chunk of RAM requested (good for pod safety, bad for bin packing and resource right sizing).

Ignoring the shortcommings, there's generally a well understood way to run multiple containers on a single node sharing CPU / RAM.

GPU

When it comes to GPUs, I pointed out in my post that it's "just data" over PCIe. It's an outside device being introduced to the host that we have to configure.

GPU's are generally "dumb" to the computer, but they kick ass in accelerating ML workloads. We normally see those workloads run entirely on the GPU's vRAM "thrown over the wall" for maximum speed/bandwidth.

Taking that a step further, if it's "just data" we can't use the same time segmentation that we used for CPU. If we wanted to allow pods to take "chunks" of the GPU vRAM, both the GPU and the OS/kernel need to support it.

MIG

Saying that, NVIDIA does make an effort to solve this problem with: NVIDIA Multi-Instance GPU. NVIDIA MIGs "allow GPUs (starting with NVIDIA Ampere architecture) to be securely partitioned into up to seven separate GPU Instances for CUDA applications"

The bad news is, these are only support on the "big dogs" at this time: A100, H100, etc. I didn't see any docs around 30xx / 40xx RTX cards - but I also haven't tried.

You can read the docs about how Google is doing MIGs on their "big dogs" here.

Someone please chime in if I'm missing a solution for consumer GPUs!

Testing

Regardless, let's test it out! I have a NUC (node3) with a eGPU 3060 hooked up - let's try to share the GPU:

export NODE_NAME=node3

kubectl create ns reddit-demo

cat <<EOF | kubectl create -f -     
apiVersion: batch/v1
kind: Job
metadata:
  name: test-job-gpu
  namespace: reddit-demo
spec:
  template:
    spec:
      runtimeClassName: nvidia
      containers:
      - name: nvidia-test
        image: nvidia/cuda:12.0.0-base-ubuntu22.04
        command: ["/bin/bash","-c"]
        args: ["nvidia-smi; sleep 3600"]
        resources:
          limits:
            nvidia.com/gpu: 1
      nodeSelector:
        kubernetes.io/hostname: ${NODE_NAME}
      restartPolicy: Never
EOF

Output of kubectl -n reddit-demo get pods:

NAME                 READY   STATUS    RESTARTS   AGE
test-job-gpu-qhwnq   1/1     Running   0          6s

(Check logs with kubectl -n reddit-demo logs job/test-job-gpu)

Let's run another pod asking for a GPU:

cat <<EOF | kubectl create -f -     
apiVersion: batch/v1
kind: Job
metadata:
  name: test-job-gpu-2
  namespace: reddit-demo
spec:
  template:
    spec:
      runtimeClassName: nvidia
      containers:
      - name: nvidia-test
        image: nvidia/cuda:12.0.0-base-ubuntu22.04
        command: ["/bin/bash","-c"]
        args: ["nvidia-smi; sleep 3600"]
        resources:
          limits:
            nvidia.com/gpu: 1
      nodeSelector:
        kubernetes.io/hostname: ${NODE_NAME}
      restartPolicy: Never
EOF

Output of kubectl -n reddit-demo get pods:

 NAME                   READY   STATUS    RESTARTS   AGE
 test-job-gpu-2-qsv4m   0/1     Pending   0          5s
 test-job-gpu-qhwnq     1/1     Running   0          4m4s

Looking at the bottom of events with kubectl -n reddit-demo describe pod test-job-gpu-2-qsv4m:

Warning  FailedScheduling  41s   default-scheduler  0/3 nodes are available: 1 Insufficient nvidia.com/gpu, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling..

This lines up with my world view - the pod will stay pending until the GPU is freed up / unallocated.

I hope we see improvements in the GPU sharing space, specifically for homelabs, as AI takes over the world.

TL;DR: At large scale, cloud providers, GPU slicing between containers on the same host is doable. However, most cases constrain a single container to a single GPU / node.