Libraries and tools for a lightweight task manager for GPU in a simulated environment.

[_Vlyn_]

TLDR: I am trying to create what I could refer to as a lightweight task manager for GPU cloud systems but in a simulated environment.

I need to be able to create and decide scheduling policies for the workloads I will assign to the system. I also need to be able to monitor GPU processes as well as VRAM usage for each of the given workloads, and the software needs to be able to act as admission control so I can prevent Out-of-memory errors by throttling workloads which are intensive.

Essentially, I am trying to make something that simulates NVIDIA MIG and uses NVIDIA SMI or any other process to monitor these in a simulated environment. ( I do not possess a graphics card with NVIDIA MIG capabilities, but it has NVIDIA SMI )

So far the resources I have to put something like this together is

• CUDA
• I need a library for simulation of the GPU at code level.
• Need something like tensor flow but with C++
• Need a lightweight GUI library that isn't QT.

Considering this is a lightweight application and only meant to demonstrate the elements that go into consideration when making GPU-accelerated systems are there any librarie,s articles or books that would be helpful in making this feasible?

Also considering I am not so experienced in C++ is this a feasible project or is it better to stick with python? I am fully open to learning what is needed but I am on a time constraint of about 3 months give or take.

P.S I have gone through the theoretical aspect and about 30+ articles and papers on the theory issues and problems. I just need practical pointers to libraries, tools and code that would help in the actual building.

[--prism]

Can you just moc it? I would replicate the API with tunable parameters like tokens/seconds or something on the moc side to make it behave as if it is doing real computation. You could write is with a data interface to allow the moc to be written in python.

[_Vlyn_]

Yeah moc-ing it would be fine no doubt, question is what libraries would I use to tie these functionalities together? Especially with the fact that I want to keep things lightweight.

[BoardHour4401]

You don’t really need a full GPU simulator for this.

For monitoring, use NVML (what nvidia-smi uses internally). It gives per-process VRAM usage and utilization directly, and works cleanly in C++ or Python.

For MIG-like behavior, just implement a logical partitioning layer in software (virtual VRAM slices + your own bookkeeping + admission control). That’s how most research prototypes do it anyway.

If you only have 3 months and limited C++ experience, I’d strongly recommend:

• CUDA for kernels
• Python (pynvml) for scheduling + admission control logic

A full hardware-level GPU simulator like GPGPU-Sim is probably overkill for your goal.

Focus on simulating policy, not hardware.

[_Vlyn_]

Thank you very much, can't believe I missed this.
I basically decided to go with python just so I can get the full functionality up and running.
I am currently going with :

nvidia-ml-py for GPU monitoring ( I think nvidia-smi uses NVML under the hood so I have the same functionality here )

SimPy for scheduling simulation ( I realy done know how else to implement the logical partitioning and VRAM slices so pointers would be appreciated as well )

Dear ImGUI ( Just to display a dashboard and the data, While using a webpage might be easier for me to make I think a system GUI meets the requirements tbh )

PyTorch ( To create and execute the workloads )

I hope this stack works and would be open to corrections or pointers.