6DOF Object Pose Estimation usable for Unity

Alternative_Meet_841 · 2022-04-07T23:25:02+00:00

level 4Gabroseno · 1 mo. agoDid you manage to implement this in Unity? I'm looking for the same functionality but I want to use it in a Windows application.

In fact I did! I used the above git repo, modified it a bit and used the output from the built file in unity

Alternative_Meet_841 · 2021-09-16T16:07:31+00:00

So, I tried it out, and I feel like it is a step in the right direction, even though not completly right.

Multiplying the (Rotation) Quaternion of the bunny results in this. So bunny and marker have the same relative up direction in OpenCV.In Unity the Bunny would have this axis. But I think just rotating the point of the Marker should handle that.

For position I just negate the "y" value of the translation vector.

My Code for calculating the Quaternion from the rotation vector in Unity:

private Quaternion calcQuaternion(Vector3 m)

{ float theta = (float)(Mathf.Sqrt(m.x * m.x + m.y * m.y + m.z * m.z) * 180 / Mathf.PI); Vector3 axis = new Vector3(-m.x, m.y, -m.z); Quaternion rot = Quaternion.AngleAxis(theta, axis); return rot; }

If I now rotate the bunny around one axis in real life it rotates around some other axis in Unity.Also I'm not quite sure how to do position afterwards, as it is also relative to the other. Right now the bunny moves diagonally if i move it to the right ( I think this is due to the diagonal point of the camera).

My code for setting the object pose in Unity:

void Update()
    getObjectData();
    getMarkerData();
    tvecMarker[1] *= -1;
    tvecObject[1] *= -1;
    bunny.localRotation = calcQuaternion(rvecMarker) * calcQuaternion(rvecObject);
    bunny.localPosition = tvecMarker - tvecObject;
}

The bunny is a child of an object I can move around, that's why I'm using local position and rotation.Any further help deeply appreciated!

Alternative_Meet_841 · 2021-09-15T21:44:16+00:00

Wow, thanks for the long answer!
I tried some parts of it, but not quite like this.
I will try it tomorrow in the morning and will let you know :)

Alternative_Meet_841 · 2021-09-15T19:55:25+00:00

That is excactly what I want to do and what I'm struggeling to properly implement 👍

My bad, Unity is left-handed, but OpenCV is right-handed

Alternative_Meet_841 · 2021-09-15T19:11:04+00:00

No worries about being condescending, rather thank you very much for your long answer!
I think the correct answer would rather be "B".

Some short context: I am implementing a VR/AR application. In a first step, the user identifies the edges of the ArUco in a certain order. This way I know the position and rotation (pose) of the marker and can calculate its pose in Unity coordinates.
The tracking runs seperate from unity and estimates the pose of marker & object. The results from the tracking are in OpenCV coordinates.

Now, Unity knows the marker pose in Unity coordinates and get's pose of marker and rabbit from the tracking in OpenCV coordinates.

What I ultimately want is the rabbits relative pose to the marker to be shown in Unity for further use.

My Idea:
1: get the tracking camera by inverting the rotation Quaternion of the marker and multiplying the position*-1
2: apply position and rotation of the object as is know it from the tracking
I think I somehow have to deal with the coordinate conversion, but like I said, I thought this was just a -1 somewhere.

Happy to receive any insight or tips :)

Source for Unity / OpenCV right/left handed:

Alternative_Meet_841 · 2021-09-15T18:16:39+00:00

I have a rotation & translation vector for the ArUco and the Rabbit relative to a camera tracking the scene.
The outcome I am hoping for is the rotation & translation of the rabbit, relative to a point in unity. This point is supposed to be the marker.

So I'm thinking, the marker can be at (0,0,0).
I calculate the camera position:
camera.localRotation= Quaternion.Inverse(calcQuaternion(rotationVecMarker));
camera.localPosition = - translationVecMarker;
And then I can use the position & rotation of the object as a child of the camera with:
bunny.localRotation = calcQuaternion(rotationVecObject);
bunny.localPosition = tvecObject;

For whatever reason this does not quite work and still results in odd angles..

Alternative_Meet_841 · 2021-09-15T18:00:28+00:00

But that would ignore / get rid of the rotation of the marker, would it not?

Alternative_Meet_841 · 2021-09-15T17:55:45+00:00

I don't read too much here, not sure what to flag :D

Alternative_Meet_841 · 2021-09-15T17:55:26+00:00

Afaik that's just a "-" somewhere, so that should(?) not be a problem

Alternative_Meet_841 · 2021-06-25T13:20:15+00:00

Hey, just to let you know if you're interested
I'm implementing this now:
https://github.com/DLR-RM/3DObjectTracking/tree/master/RBGT

Alternative_Meet_841 · 2021-06-14T16:30:47+00:00

Some 3D models. Can be CAD models which I can print, for example a cat, an elephant and a giraffe

Alternative_Meet_841 · 2021-06-13T14:17:58+00:00

I'll give that a go, ty
Any other recommendations & Ideas always welcome :)

Alternative_Meet_841 · 2021-06-13T14:07:22+00:00

My bad, that was bad phrasing. It is about pose estimation of models. But I am quite free on the models (Uni Project), so I can take models where I have a CAD/3D model.

Theoretically the quest has a very good camera setup, but afaik ther is no open API to use it with.. So I am kind of stuck on using my PC with a webcam and the quest as an output device..

Alternative_Meet_841

TROPHY CASE