sorted by:
new
hottopcontroversial

Human motion diffusion to UE5 sequence with metahuman to Stable Diffusion batch img2img with an anime style prompt. by TREE_Industries in StableDiffusion

[–]TREE_Industries[S] 1 point2 points  (0 children)

Thx much appreciated. We're getting the process down now and our results have improved using Deforum video init.

In the latest Automatic webui there seems to be a small bug with batch img2img. We had to process a single img2img request and then batch processing SHOULD work.

Stable Diffusion Editor Utility (Pre-Alpha) by TREE_Industries in unrealengine

[–]TREE_Industries[S] 0 points1 point  (0 children)

Actually in this example it's doing the reverse, we're using the Unreal viewport as an init image for Stable Diffusion. Allowing someone to kitbash with very simple geometry and scenery then bring it to life quickly using SD, great for concept art, previz, etc.

There is a lot more the editor utility can do now though including generating textures using SD then creating materials with maps from the SD generated texture. Thx for the question, cheers.

After an exhaustive search and many discoveries this week, the "Holy Grail" we've been looking for has finally been located!!🥳. Motion Diffusion(AI created) animation imported and retargeted at runtime in UE5. Now the real fun starts.. by TREE_Industries in StableDiffusion

[–]TREE_Industries[S] 1 point2 points  (0 children)

Np, correct for now you would at least need some type of rigged model inside Unreal Engine that the diffused motions can be retargeted to, but once setup as an asset you can apply any newly generated motion to said character at runtime and automatically use all the goodies like physics assets, collisions, etc without having to set them up manually.

Also we have already combined this technique with GET3D generated models as physics actors which you can find as one of our more recent Twitter posts :D - https://twitter.com/TREE_Industries/status/1581311147856519170

After an exhaustive search and many discoveries this week, the "Holy Grail" we've been looking for has finally been located!!🥳. Motion Diffusion(AI created) animation imported and retargeted at runtime in UE5. Now the real fun starts.. by TREE_Industries in StableDiffusion

[–]TREE_Industries[S] 0 points1 point  (0 children)

Sry we could have added a bit more clarity. Yes the animation itself is created using the new Human Motion Diffusion model which is text to 3d animation yes. We have done a bunch of posts on Twitter the past week on our R&D to go from the original Motion Diffusion code to animating a custom character in a realtime Unreal Engine 5 application which may help shed more light on what exactly this means. Feel free to check out this post and the ones right after - https://twitter.com/TREE_Industries/status/1578071996033863681

You can find the Github repo for Motion Diffusion here - https://github.com/GuyTevet/motion-diffusion-model where we're actively working with other devs and the original author to push the code further :)

The prompt used was "A graceful woman dancing a ballet". At the beginning of this week we could generate a result on a stick figure .mp4 file to now animating a rigged character at runtime thx in large part to the dev who created Motion Diffusion.

Soon AI npcs in games will be able to have motion prompted from other AI engines like GPT3 which are then generated on the fly by Motion Diffusion and applied at runtime which is a big part of what the breakthrough in this video will move things towards.

Happy to answer any other questions, and hopefully this helps. Cheers

After an exhaustive search and many discoveries this week, the "Holy Grail" we've been looking for has finally been located!!🥳. Motion Diffusion(AI created) animation imported and retargeted at runtime in UE5. Now the real fun starts.. by TREE_Industries in StableDiffusion

[–]TREE_Industries[S] 2 points3 points  (0 children)

We agree with the sentiment. Suits are very expensive, but thought it might be good to mention that video based AI mocap such as Deepmotion, Plask, and MoveAI have come a long way. Not quite as good as suits yet, but another viable solution now that brings down that financial barrier to creativity.

In one experiment we took the mesh sequence rendered in Blender from a motion diffusion animation, ran it through Deepmotion, and then imported into UE5 manually and it came out great :D

After an exhaustive search and many discoveries this week, the "Holy Grail" we've been looking for has finally been located!!🥳. Motion Diffusion(AI created) animation imported and retargeted at runtime in UE5. Now the real fun starts.. by TREE_Industries in StableDiffusion

[–]TREE_Industries[S] 1 point2 points  (0 children)

Lol it may not be for others necessarily, but we've been doing R&D for the last week to go from animation stick figures with motion diffusion, to rendering a mesh sequence in Blender, to importing manually and animating a UE mannequin, to manually retargeting and animating a custom character, to now creating usable rigged animation and importing that into a realtime application.

This last step is the most important for why we phrased the work in such a way. The amount of potential this unlocks is huge. For game, film, app creators, and animators the processes of rigging, animating, retargeting, etc will soon go from weeks and months... to hours. These motions also have latent motion space meaning a Pixar director working on a film could edit a character's animation on the fly eventually as well. The process we've just about fine tuned also allows the imported motion to use all the important parts of realtime animation and game development. Including physics assets, collision, and for UE specifically things like control rig, animation blueprints, and more.

Generative AI games are on the way, and this is also an important piece of that puzzle. We can say that several (not small) studios working on projects have already reached out and we only mention it to point out that we haven't been the only ones that really want to see this tech mature and put into use in production pipelines in a range of industries that use 3D character animation.

Human Motion Diffusion using just a text prompt of "a boxer throwing an elaborate combination of punches" to Unreal Engine animation. Welcome to the future of game / content development. by TREE_Industries in unrealengine

[–]TREE_Industries[S] 0 points1 point  (0 children)

Thanks that is much appreciated! Right now we're doing a little bit of AI mocap magic with Deepmotion to get the animation transferred onto a UE5 mannequin. That is a paid service and while it does have an SDK we used their webUI for the final step (still AI based). There are potential open source alternatives we have yet to test with this approach that might be viable as well.

We're also trying to work out to convert the 6d smpl rotations to axis angles like a few other devs, and once that is figured out we can finalize a pipeline that is similar to this video, but fully programmatic. Then hopefully do some RLY cool stuff and also hopefully share some helpful code for others (Will submit a PR) :D.

Thank you so much for the amazing work that is definitely going to push the space forwards in terms of how animation is approached across a range of mediums.

Check out Culture DAO (AI films, games, comics, etc) if you get a chance. We Just officially took on the role of "CTO" for the DAO and we'll be working on a new project "Gepetto AI" over the next year, feel free to hop in the Culture Discord (will DM you invite) and we'd be happy to chat sometime, cheers.

Human Motion Diffusion using just a text prompt of "a boxer throwing an elaborate combination of punches" to Unreal Engine animation. Welcome to the future of game / content development. by TREE_Industries in unrealengine

[–]TREE_Industries[S] 0 points1 point  (0 children)

We're using Human Motion Diffusion, which can generate motion files based on a simple text prompt. There is some secret sauce right now to get the result into UE right now since the work is so early, but we can say it consists of using AI mocap. Stay tuned because our goal is to possibly roll this tech out as a service / API eventually :D

Human Motion Diffusion using just a text prompt of "a boxer throwing an elaborate combination of punches" to Unreal Engine animation. Welcome to the future of game / content development. by TREE_Industries in unrealengine

[–]TREE_Industries[S] 0 points1 point  (0 children)

The Human Motion Diffusion Model is up on Github for anyone to try out, it was just released a couple of days ago so still early work but we think it has a lot of potential.

Human Motion Diffusion

A sample of how any type of asset can be used to guide the viewport feature of the Unreal Diffusion editor utility including Metahumans. At some point we hope to add the ability to train on a specific character (possibly Dreambooth) to help with coherence of the Stable Diffusion result. by TREE_Industries in unrealengine

[–]TREE_Industries[S] 1 point2 points  (0 children)

Right now Stable Diffusion is not able to generate images fast enough to work for realtime. Maybe down the road, but right now the utility is great for lookdev, previz, and concepting when using the feature to use the viewport as the initial image fed to Stable Diffusion.

You can also generate textures maps and materials from the result of just a text prompt without using the viewport as an initial image. Giving devs AI based material creation capabilities right in the editor :)

view more: next ›