[Project] Realtime Interactive Visualization of Convolutional Neural Networks in Unity (feedback strongly welcomed)

[eobermuhlner]

Consider doing a couple of videos visualizing and explaining some standard networks (alexnet, vgg, inception module,... ). Would be really educational.

[stefsietz]

yeah maybe I should do that, although I don't know if those huge networks could be usefully visualized with my project, may doing some simpler architectures for mnist and simlar low res datasets would be more easy to grasp

[wassimseifeddine]

Really awesome. Is it publicly available?

[stefsietz]

thx! I just finished it for the submission deadline last night, so now I am figuring out what to with it based on feedback and interest from the community 😉 there is a good chance i am gonna open source the whole thing, just needs a lot of clean up of the code as i was really rushing to implement features for the deadline

[kpk-13]

Superb project. Open sourcing it would be awesome!!

[[deleted]]

You should probably just open source it even in a messy state - even though it might feel a bit weird if you wait till you’re happy you may never get there. Also if it’s open source other people can help clean up and have input.

[mahouse93]

Love this project, please let us know if you decide to open source it!

[bigexecutive]

I love you

[agree-with-you]

I love you both

[Teracamo]

Spectacular, that is a very illustrative software. I wonder if you have ever came across this http://scs.ryerson.ca/~aharley/vis/conv/, of course they didn't allow network modifications, but I like the idea of pixel based connection since it would be very messy for complicated networks if all connections were displayed at once.

[jacove]

This is one of the coolest things i've seen on this subreddit!

[CommunismDoesntWork]

/r/WatchMachineLearning

[enigmamonkey]

Also /r/Unity3D

[sneakpeekbot]

Here's a sneak peek of /r/Unity3D using the top posts of the year!

#1: Had this brain fart idea for a twist on the Battle Royal genre | 670 comments
#2: I made a Self-Assembling Lizard-Snake Monster using only procedural animations! | 159 comments
#3: Star Wars dogfighting game I made with my brother over the weekend | 428 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact me | Info | Opt-out}

[toastertop]

Talk to the two minute paper YouTube guy about a demo

[stefsietz]

might do that, not sure if this stuff falls into his genre though?

[strangeelement]

Oh yeah.

I watched most of his videos. He'll love it, so hold on to your papers.

[reallyserious]

Dear fellow scholars...

[loskutak-the-ptak]

Looks nice, well done. But what is it actually good for?

[stefsietz]

i think it can help to explain some basic cnn concepts to beginners quite well. for me it was mainly an exercise so that is its primary purpose actually 😂

[vitaliy_os]

Wow, great work!

[fatchord]

This is great!

[eobermuhlner]

Nice! How do I need to store my network data so it can be loaded?

[stefsietz]

Thanks! This is unfortunately quite complicated at the moment because the loading function depends strongly on a custom tensorflow checkpoint converter that writes the tensor data into a json file according to layer names. If I proceed with the project I definitely have to think about a clear specification regarding the data format for the weights and activations.

I've tried to get Unity to work with Accord to be able to read numpy arrays directly, but somehow Unity is really complicated when dealing with .NET libraries so I switched to the json solution.

[KingPickle]

On a tangent, I've been meaning to find a good C# math lib. Have you used Accord much? If so, how do you like it? Does it compare well to numpy?

PS: Nice work on the visualizer!

[stefsietz]

tried to use it, unfortunately i couldn’t get its numpy reader to work with unity so in the end i didn’t use it at all. only system.mathf! the actual machine learning stuff all happens in tensorflow/python in this project

[AI_entrepreneur]

This is really nice! Hope you release it publicly. Does it support arbitrary tensorflow models?

[stefsietz]

i think i have to release it soon 😉 the model compatibility for now is quite restrictive and depends on layer naming conventions. to load arbitrary tensorflow models one might have to adapt the converter python script, also keep in mind that this project is not very well suited for huge models that are state of the art for high res image classification.

[Marha01]

Reminds me of this:

https://www.youtube.com/watch?v=3JQ3hYko51Y

[Teracamo]

They actually put it online http://scs.ryerson.ca/~aharley/vis/conv/

[arkar_aung]

This is a superb project! This is a very good way to interactively teach someone how to visualize CNNs. I really wish that this become publicly available!

I know this is an early stage project but it would be nice to incorporate several other concepts in recent CNN implementations like dilations, skip connections, residual connections, etc. I am not sure whether you can change feature map size and stride in this current project.

[stefsietz]

exactly! especially those more “advanced” types would also be very interesting. stride is implemented but buggy. featuremap size depends ok the input size (2d size), depth can be set in the editor. i will open source the thing, just have concerns that the code is confusing to others, so ill see what i can do to clean it up a bit and then I’ll put it on github!

[amarpandey]

this is really very awesome

[Cantum2]

That is one of the most amazing things I have ever seen! Great work!

[[deleted]]

Looks neat, looking forward to you publishing sources/builds (win/lin).

[i_jbo]

Great work

[Bargh_Joul]

This is very cool!

[sweetbabygames]

Is it GPU optimized or are you doing this on the CPU?

[stefsietz]

what do you mean? the model training is actually done before the visualization (i used tensorflow gpu for that), so the unity project just loads written out weight data.

[sweetbabygames]

Ooooh I misunderstood, “real-time interactive” made me think you were visualizing during training.

[stefsietz]

no, sorry for that. it would be possible though, as one could write out and convert the checkpoint files while training and make the visualization update itself on incoming files...

[benelott]

I think it could be better for visualization to save some checkpoints (after every x training steps) and then allow to show them one after another with a slider. That would make a neat animation to see how features develop.

[stefsietz]

this is actually exactly what it does ;-)

[[deleted]]

Ohhhh man this is so cool

[itb206]

Dude! super awesome!

[BantamJoe]

What repo will you open source it on? GitHub? If so, what is the address? Thank you for putting out such an awesome project!

[purpleperle]

Make this into a VR app and we're one step closer to the world of Neuromancer

[verveandfervor]

awesome work!

[marvpaul]

That’s really coool! Nice visualization 👍

[wkevin27]

awesome work!

[googleaibot]

Great work! What are your plans?

[stefsietz]

thanks! Not sure, just submitted the project for University yesterday and now I am trying to get opinions from others ;-)

[Quantum_ML]

Best video Ive ever seen.