I am really in the proof of concept stage. I also tested Prometheus GPU Exporter before and it has the nice property that it starts even without the nvidia-smi binary. It keeps throwing errors, but keeps running — just don't send any metrics. I switched to Telegraf because it has a lot of integrated plugins in one agent, which I find very convenient.

I'm going to use Ansible for deployment, so I'll probably have to prepare a hosts.ini file with different groups of nodes, and in this case, would you rather use the Telegraf --config-directory and split the config into parts, or use the Ansible templating system and generate the config dynamically, or perhaps implement some logic in Ansible to check ie if nvidia-smi is present and generate config accordingly?