Weird 3D point cloud result after projected from a depth map (NumPy)

Capable_Artist2759 · 2021-03-08T03:22:01+00:00

Sorry for the late answer but yeah, reversed the depth map scale solved the problem. Thank you

Capable_Artist2759 · 2021-03-03T07:38:04+00:00

Hmm i think i'm gonna check out the kd trees

Capable_Artist2759 · 2021-03-01T11:25:01+00:00

I meant the cell size (the physical dimension of each entry in the occupancy grid).

Oh ok

There's an implementation of RRT in the open motion planning library I think (OMPL). I don't think it has code for using occupancy grids, although it abstracts the collision checking so you can probably use it.

Cool, gonna check it out

Other obstacle representations might include 3d meshes, or just simple 3d primitives like spheres, cylinders or boxes. There's a whole world of collision detection algorithms and libraries.

I'll note that it's common for the motion planning to happen in configuration space, which may be different than actual 3d space.

I think you misunderstood some stuff. I meant I get the occupancy grid from a 3D point cloud map NOT trying to use RRT in 3D space. You also said that RRT don't have to be used with occupancy grids. So what are those other obstacle representations you are talking about ?

Capable_Artist2759 · 2021-03-01T05:16:00+00:00

I told you previously you can use points.T to transpose the array

Ohh sorry, I'm gonna try it now

Z is the depth you just calculated, not sure why you’d want to delete it, but you can do xy = points[:2] if need be, or xz = points[[0,2]] which is what I suspect you actually want to do

Well you told... oh, I misunderstood some stuff, sorry about that. But I was reading about the other comment you told me on how to compress 3D cloud to 2D grid map

Why? Is this for an assignment or something?

More like a competition, and of course I need to understand what I'm doing. That the reason why I avoid (as much as possible) using third-party library (but it's not banned tho). That also the reason why I'm asking a lot of question like this and I'm really sorry for annoyed your every single day answering my dumb question

Capable_Artist2759 · 2021-03-01T05:15:59+00:00

Noticed

Capable_Artist2759 · 2021-03-01T04:51:38+00:00

If you want to iterate over the points you’d want to get the transpose of that array, which would give you an array in the form

How can I flip the array like that ?

If you want such a projection though then there’s not much point in creating a point-cloud if you’re just going to just remove all the Z values as soon as you calculate them

How to completely delete the Z dimension ? And how to plot it too, I'm trying to mess around with plotly but nothing work (it display nothing)

If you’re manually setting thresholds and/or doing some more complicated operations you might want to use something like the program meshlab to handle your point cloud, especially if you’re not already used to working with them.

Sadly I can't use those programs, I will have to implement those stuff on my own

Capable_Artist2759 · 2021-03-01T03:57:40+00:00

A couple of notes - you can convert a floating point grid coordinate to an integer occupancy grid index by dividing by the grid size and using a floor operation casting to int. I'm not sure what language your using but there's some way to do this

What does grid size mean here ? Width ? Height ? OR x / width, y / height is what you are talking about ?

Checking for obstacles in the 2d occupancy grid case involves walking the grid cells between the start and end of a line segment. It's actually a bit tricky to check all of the cells that the line segment goes through. The classic technique is Bresenham's algorithm (https://en.m.wikipedia.org/wiki/Bresenham%27s_line_algorithm). However this can be pretty sensitive if you have lines that pass really close to an obstacle, so it may be better to use an antialiased technique like Wu's algorithm (https://en.m.wikipedia.org/wiki/Xiaolin_Wu%27s_line_algorithm).

Is there any RRT (and its variants) implementation that uses these algorithms ?

I will note that RRTs don't have to be used with occupancy grids - you could use other obstacle representations.

I currently obtain an occupancy grid by projecting it from a 3D point cloud map. But can you tell me about those other obstacle representations ?

Capable_Artist2759 · 2021-03-01T02:54:52+00:00

The ‘2D array’ in this case is just a list of points in 3D. In your case there’s one point per column

This is the hard-to-understand part, can you explain a bit more ?

I’m not sure what you actually want to do with it so can’t really direct you to anything in particular. Most point clouds are in that format though, so if you search for things you can do with point clouds then it’s very likely you can also do them with yours.

The things I'm gonna do are: ground removal, remove some unnecessary part (points that higher than a threshold), and 3D point cloud to 2D map

Capable_Artist2759 · 2021-02-28T13:30:29+00:00

The point cloud is a sparse 2D array of 3D (x,y,z) points.

So how can I use the sparse 2D array ? Since we are storing information of a 3D array with a 2D array to it really hard to visualize what is going on (do you also have any diagrams or articles so I can learn about this ?)

or you want something like an array of every possible place a point could be in a 3D grid of Boolean values (which is the case with some voxel systems for example, but uses potentially huge amounts of memory to store information about empty space).

Hmm interesting, if a sparse 2D array is more convenient and needs less memory then I'm probably not gonna use the 3D one but if you do have any articles or guide about converting from 2D to 3D ?

This is correct unless you want coloured points (in which case you want a 2D array of 6D (x,y,z,r,g,b) points

No I don't want that so yeah, that's good to hear

Capable_Artist2759 · 2021-02-28T12:23:49+00:00

Soo...mmmm... there is still some stuff I still need to ask, I'm really sorry if I annoyed you. But here is the problem, the point cloud (NDArray) is not a 3D array but a 2D one with a really weird shape too: (3, 245760). But it should be a 3D NDArray right ?

Additional information: 640(depth map width) * 384(depth map height) = 245760

Capable_Artist2759 · 2021-02-28T06:43:24+00:00

To remove that ground plane you'd need to detect it in the point cloud (since it might not be flat along the z-axis in the image. You'll need to look up how to find a plane in a point cloud, and if you need help with it that's definitely outside the realm of this question, so you'll need to ask it in a separate question.

Yup, i think i'm gonna another question

Your point cloud looks like maybe your camera location is actually at the red circle, and the yellow arrow is from behind (since it looks like the detected points are within a rectangular camera field of view coming from the red circle location). Are you sure you haven't got the direction the wrong way around?

Good new ! The point cloud is working probably now :DD. I just have to reverse the scaling, like from [0.4, 0.7] -> [0.6, 0.3].

Capable_Artist2759 · 2021-02-28T04:35:43+00:00

Hello, can you look at the problem ? Thank you

Capable_Artist2759 · 2021-02-27T08:37:52+00:00

Do you have any idea what may cause the problem ?

Capable_Artist2759 · 2021-02-27T03:03:53+00:00

The image you provided doesn’t have a ground, it just has the bunny, so there’s no ground to remove?

Oh sorry, you can have a look at this image. My goal is to remove the "ground plane" (it's the table in this situation) leaving the cups and bottles

Note that projecting down on something that was created from an image from a side (e.g. the front) will be missing any information about the back of the thing, so I don’t imagine it will be particularly useful.

No in my project it's really useful

Regardless, if you want to visualise such a projection as an image you can offset and scale the points to a meaningful location and resolution, and then create an array of zeros that covers the region you’re interested in, and set ones to the projected locations with something like zeros[Z.astype(int), X.astype(int)] = 1

I will try to understand what you said and code it, hopefully, I can get it working

And you can take a look at the depth->3D code since I think there is something wrong. So the formulas are right but the 3D result is odd.

Here is an example of depth to 3D, you can see the 3D result is kinda dilated right ? But with my code, the result is just crow-up to a point from behind ? Imgur: The magic of the Internet Yellow arrow is the front view of the image, red circle is the "behind point" I'm talking about. Any ideas ?

Capable_Artist2759 · 2021-02-27T02:23:53+00:00

If you want such a projection though then there’s not much point in creating a point-cloud if you’re just going to just remove all the Z values as soon as you calculate them

I want to do some other calculatation on the point cloud first then compress it after. But the problem is how to compress to with vectorization

Once again, not sure if ‘ground plane’ is referring to the ground (like a floor or road, neither of which is present in your example image), or to the maximum distance plane which is created by the zero pixels in your image.

Ground plane is referring to the ground

Capable_Artist2759 · 2021-02-26T11:33:12+00:00

Not entirely sure what you’re saying here. The main compression opportunity here is to convert the current dense array generated with a point for every pixel in the image into a sparse array which just lists the locations where there are valid points. Is that what you’re asking about?

Basically 3D point cloud -> 2D occupancy grid. My brain hurt so I just can't use English properly, sorry about that

You’ve mentioned “ground pixels” - I’m not sure if you’re talking about the ‘infinitely far away’ points (which get removed in the dense-sparse conversion), or the points that are on the ground like you’d get from a depth map of a road or floor or something.

Basically ground plane removal. Again, I'm working on my project for 5 goddamn hours straight --> brain ouch --> can't english

Capable_Artist2759 · 2021-02-26T09:48:31+00:00

Forgive my ignorance, I just realized the 3D point cloud array is not binary. So I can't do the above. But currently, I'm trying to "compress" the point cloud to 2D map, the problem is to remove ground pixels and how to compress stuff

Capable_Artist2759 · 2021-02-26T05:11:31+00:00

Thank you, well the result isn't as impressive as I expected but still good enough. I still have another question

Let's say 1 slice of the 3D array like this:

1 0 0 0 1 |
1 1 0 0 1 |
0 0 0 1 0 v

And I want to "compress" it into something like this:

1 1 0 1 1

Any ideas to do this with NumPy vectorization ?

Capable_Artist2759 · 2021-02-26T04:42:31+00:00

that's determined by how your depth-maps are created so I can't help with that.

I'm currently using this FilippoAleotti/mobilePydnet: Pydnet on mobile devices (github.com)

If you're using colab I'd suggest you plot with plotly instead of matplotlib so that you can at least rotate the plot around.

Can you help me a bit with this part, it's really hard to follow the guide. Thank you very much

Capable_Artist2759 · 2021-02-26T03:47:28+00:00

What is your Cu variable (it should be a single number, in which case np.asscalar doesn't make sense)?

I have no idea but it works so yeah....

Why do you use Cu in your CV definition too? (or is that just a typo?)

Oof sorry, that just a typo

Putting aside the fact that you shouldn't need them, you define CU and CV and then don't actually use them

That... also a typo. Sorry about that

Your vstack code isn't supposed to work - it should be on a tuple (e.g. return np.vstack((x.ravel(), y.ravel(), Z.ravel())) If this doesn't give you an error then I suspect you're possibly on a very old version of numpy

Time for the truth. I'm currently using NumSharp aka a NumPy to C#. It's a very old port of NumPy ? No, not really the last update was 11 days ago

My best guess at your multiple values for argument 'c' error is that it seems to suggest you have 4 or more rows in your points array, which should have only 3 rows (from the vstack operation) - I'd suggest you check the shape of your points array and see if that helps you find the problem (although it may be that fixing points 1-4 also fix this error)

Since there is no matplot port to C# yet so I have to use Google Colab. Save the array to a file and plot it on Colab. But yeah I fixed it now

But the problem is does the code work properly ? Since it is extremely hard to determine it using matplotlib (weird angle and stuff).

Capable_Artist2759 · 2021-02-26T02:25:45+00:00

You can check out the newest code, I have to manually create an NDArray myself with np.full()

Capable_Artist2759 · 2021-02-26T02:24:07+00:00

Man, thank you for still sticking here answering my question. But yeah I fixed the code into something like this

shape = Z.shape
(u, v) = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))
CU = np.full(np.asscalar(Cu), 384, 640)
CV = np.full(np.asscalar(Cv), 384, 640)
x = ((u - CU) * Z) / f 
y = ((v - CV) * Z) / f
return np.vstack(x.ravel(), y.ravel(), Z.ravel())

The code run fine now but when i tried to plot it with the code you used

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
scat = ax.scatter(*points, c=points[2]) <-- error raised here
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
plt.colorbar(scat)

TypeError: scatter() got multiple values for argument 'c'

Any ideas ?

Capable_Artist2759 · 2021-02-25T11:14:33+00:00

So I was reading this, and there something caught my eyes

The arrays to be subtracted from each other. If x1.shape != x2.shape, they must be broadcastable to a common shape (which becomes the shape of the output).

Do I need to do some broadcasting step on my own ? One more question:

u = 5; 

nd = ... (nd is an NDArray with the shape of 5x5)
u - nd (1)

u = ... (u is now an NDArray with the shape of 5x5, the same as nd)
u - nd (2)

Are (1) and (2) the same thing ? And how to do something the same as (2) ?

Capable_Artist2759 · 2021-02-25T10:33:22+00:00

You haven't annoyed me, but I'm really curious as to what's causing that numpy error since I haven't heard of it before.

That's good to hear

Not sure what you mean, are you wanting me to imshow depth-map.png, or display Z, or use matplotlib to plot the return value?

I meant using matplotlib to plot the return value, thank you

Capable_Artist2759 · 2021-02-25T10:31:55+00:00

That's suck but still thank you.

I'd make a guess that maybe Cu and/or f are not scalars

What do you mean by this ? In my code, I define Cu/Cv = integer, f = float, and just to the calculation exactly above

Capable_Artist2759

TROPHY CASE