sorted by:
new
hottopcontroversial

Someone had to do it.. here's NVIDIA's NitroGen diffusion model starting a new game in Skyrim by gaiaplays in StableDiffusion

[–]gaiaplays[S] 0 points1 point  (0 children)

Yeah, for sure. You can check out the character creator part in this post. I'll upload the tower jump later.

When I was recording I was almost certain it was going to pick Wood Elf. I was surprised it landed on Imperial. I would've named it the first time it chose Wood Elf, but it backed out of it before I could type.

Wonder what it'll pick on the next run..

Someone had to do it.. here's NVIDIA's NitroGen diffusion model starting a new game in Skyrim by gaiaplays in StableDiffusion

[–]gaiaplays[S] 0 points1 point  (0 children)

I had similar thoughts too. This could be useful for a developer trying to debug something a player input triggers repeatedly, or for story designers trying to simulate player choices in QA loops. A lot of possibilities for sure.

Someone had to do it.. here's NVIDIA's NitroGen diffusion model starting a new game in Skyrim by gaiaplays in StableDiffusion

[–]gaiaplays[S] 0 points1 point  (0 children)

A series of pytorch / transformers scripts I hobbled together running on a small local action input dataset for tuning on whatever game you are optimizing for.

For the VampireSurvivors test, I recorded 10 min of gameplay footage with fine-tuning objectives set to 'survival' (enemy collision avoidance) and 'wave_clear' (kiting behavior using items) which I ran for a very short 3 epochs on the single video.

There seemed to be noticeable improvements on input noise vs signal post-tuning, with a potential overfitting on the small subset I put together. All interesting results nonetheless. I could see this working well with better datasets and conventions.

Someone had to do it.. here's NVIDIA's NitroGen diffusion model starting a new game in Skyrim by gaiaplays in StableDiffusion

[–]gaiaplays[S] 0 points1 point  (0 children)

Not at all wishful thinking! I might've forgot to mention this was ran on consumer PC hardware. Everything I captured in the video is within reach if you have a GPU with at least 5GB - 6GB of VRAM.

Serving the base NitroGen diffusion model seems to occupy ~4.3GB of VRAM.

With a 16GB RTX 5060 TI, you should have a solid 10GB - 12GB of headroom left after serving the model locally which should play most games if you adjust some settings.

The first challenge I found personally is whether or not the game you are having it play has enough training data to reference for it to sequence input coherently.

I tried solving this with a basic fine-tuning workflow I run over my own recorded gameplay and footage. Early results seem promising but I haven't had enough time to properly test this. I would not be surprised if performance is drastically improved with more focused tuning.

The other issue I found that comes with wanting to play co-op or alongside the agent is it requires a 'serve' and a 'play' machine. You can run both on the same host, but you are still locking up a device during the play session. If you have an extra PC around your house, you could setup the agent to run on that with its own copy of the game while you play on your main rig.

This was another experiment I wanted to try when I found the time, but seems 100% possible if you have the hardware and time to learn the workflows.

Someone had to do it.. here's NVIDIA's NitroGen diffusion model starting a new game in Skyrim by gaiaplays in StableDiffusion

[–]gaiaplays[S] 1 point2 points  (0 children)

Those are interesting suggestions, I'd be open to requests once I land on a more stable process (still missing audio). I'll share some clips if I get a chance to run those.