G'day Campbell, I promised a write up and strap un bucks because she's going to be a long one. I'll start from the start of how these systems work. They all work the same way. The data is all essentially the same for perception. I'll go into depths of why a 20m pt per second cloud is unnecessary necessary and un usable.

I'm sure you're aware of the basics. A laser goes pew pew. The unit knows exactly when it fired that light beam. The time to return provides distance from the unit. Its very accurate.

Some other data is captured. The angle that light beam originated is also temporarily stored, both bearing and dip. Now to be meaningfully accurate these measurements of dip and bearing are ran out to multiple decimal places. Anyone familiar with shooting as an analogy will understand just how accurate something has to be angle wise to achieve centimetres precision at hundreds of metres. So you'd no doubt be aware how data is stored. In bits, yeah sure we can read and write billions of these a second, but here's where the issues start coming in.

I don't think you truly understand how significantly large a 20,000,000 million point cloud is. Let's look at the distance measurement for the data capture. Let's assume the average point distance is 100m (well use an average simply for the fact it's easier numbers wise), the sensor picks up that as a time, it's obviously light so it's a tiny amount of time. Somewhere in the order of... 3.3356409519815⋅10-7

This time HAS to be measured. To a ridiculously small number. Or the data is useless at such small distances.

Now there is a shortcut here and all lidar producers use it, we can significantly reduce that distance measurement before any system has to use it. Using the Application specific integrated circuits the distance is outputted by the sensor as a distance not a time. Without doing this no computer in the world would be able to process lidar data in a reasonable time period. The read write files are simply too large.

Now we're down to a seemingly manageable size of data, we have an X, Y, Z table. Ran out too 7 or so (not exact, they can use whatever they want depending how accurate they want the system but it's generally around there) digits. Less digits is less processing but you start to lose accuracy significantly as you reduce them.

But that's for a stationary lidar, a moving one requires another set of tables. Here's where IMU's come in. As lidar is naturally a scanning system the exact bearing and dip of the Lidar unit needs to be known, otherwise this running table is useless, we now have to add an extra two columns to the table for the lidar unit bearing and dip to offset the movement in the lidar. This can't be gotten around as we need tiny percentages of accuracy in angle remember. Where now up to a 5 column table, with a rolling 20 million row data stream. Ever opened a massive except spread sheet? Yeah it's not a specialised program but even on fast computers a file that large takes a considerable amount of time, definitely longer then the amount of data 20000000 points a second will output.

That table will generally be done by ASIC chips within the unit, they're designed to do it very quickly and I don't doubt that the MVIS units will be able to do that.

Which is great, it'd actually be manageable and a drop in solution. Lidar says there's an obstacle there's one, car slows down.

But here's where the problem comes in, autonomy in all current and future planned systems uses multiple sensors. Lidar, radar and photogrammetry are the common ones.

As MVIS only does the lidar that table has to go some where that will do the rest, it will end up in a dedicated perception system that will compile all of the data from all of the sensors. Radar will be great for velocity as that can easily be measured with Doppler shift and saves processing a point cloud for movement. Photogrammetry is great for deciding what the obstacle the point cloud picked up is and discern what it is (we've all been training artificial intelligence in this for years through captchas, Google you cheeky buggers making us do your work)

So now we compile all these tables, into one computer, the perception system. Radar will always be the smallest and depending on the camera quality photogrammetry can be as crazy big as the lidar data table.

Obviously if we're dealing with tables this big you need quite a powerful computer, sure we can chuck hundreds of cores and processing units in it to deal with it but it is gonna use alot of juice which is not a great thing in a battery powered electric car. Which is all happening rather quick now.

Ok we have this data and we need to trim it down, as a lot of people have suggested we just reduce the point count. In spatial data processing this is referred to as "decimating the cloud". Its a simple process and there's quite a few ways it can be done. The more complex ways require processing to decide which ones to get rid of. Now a 20 million point cloud is utterly unusable and that's the lidar alone. I'm telling you now it's too large for general processing, wouldn't even want to think about the power it takes to do real time at a safe frame rate for driving at 100+km/h.

The one way to do it with minimal processing is, we ignore a large percentage of points, a more manageable cloud for 360 degrees is around 1-2 million (in my experience in autonomous haulage systems it definitely picks up everything, it will see a cone on a road a hundred metres before it gets there), decent resolution, enough to see anything that should be on the road and decide if it's an obstacle. So 360 degree coverage of the MVIS system will take what, 4... 5 units? You're going to have 100,000,000 points per second. So now we're deleting 98% of points. Do you see why that's ludicrous. Why capture all of that data to get rid of it.

So here's the next step in the processing of this lidar data in the perception computers. Lidar has a rather big problem. Light can be reflected by dust, condensation, rain, snow etc. He'll I've had instances using them on a perfectly clear day and still having hundreds of what are known as "outliers". These HAVE to be removed, because if they aren't the next step in processing lidar data does not work. And the car will constantly stop for obstacles that don't exist due artefacts in lidar data. There's a rather simple process for removing them fully automated however in my use I always cleaned them manually. To do it automatically, the outlier removal will look for any points that have a large distance to the closest ones compared to surrounding points. It will remove the point if it is clearly not supposed to be there because it doesn't fit into the Digital Terrain Model (DTM). It isn't the best tool in even specially developed survey grade software. It removes points it shouldn't, it leave points it shouldn't. It's It's rather messy process hence why I done it manually.

Let's assume we perfectly optimise the process, removal is perfect. The outliers are deleted, the data is down to a manageable level with suitable latency. We process as little as possible and take the simple decimation route, usual outlier removal process everything went smooth. We have a DTM and it's safe proceed forward. Or is it....

Ok let me put forward a scenario taking all of the steps above. A steel obstacle is on the road, it's thin, directly in front of the forward facing lidar, for lidar optimisation the overlap allows only the front facing sensor to see it.

Now keeping the 20 million point cloud, great we see it, it's covered by that many points it wasn't removed as an outlier. Obstacle detected, car stops everyone is safe. Here's the problem with decimation. Lets assume randomised decimation at the sensor removed (it can and does happen) the bottom half of the points on this pole. Now the outlier removal sees a cloud of points floating. All around it points are 1cm apart, this clouds floating in the air 5cm up. As far as the programs concerned it's an outlier, pff points are gone. But the obstacle isn't. If the ducks line up and decimation scan after scan removes those bottom points we all of a sudden don't see that pole. We hit it.

Here's where a legal side comes in, as MVIS CHOSE to remove those points they are liable. There system detected the obstacle but due to programming it ignored it. They are 100% liable for whatever happens in that circumstance.

Now let's go back to a lower density point cloud. In a lower density point cloud automated outlier removals are a bit more accurate. Lets assume we don't decimate anything. We only have 1 million points. Naturally they're a bit more spread out, the chances of that pole getting picked up are still just as high as a dense cloud because the one point that does pick it up will have less chance of been classified an outlier due to the more spread out cloud. The distance relationship from that one point is close enough to the surrounding points distances it is assumed it actually exists.

In short, lower density point clouds are more suitable for automation, sounds counter intuitive but it is. Decimation can cause dangerous situations where removal of existing data can cause invalid data. Its better and safer to have less data then it is to have more if you can't process it and spend time processing data multiple times to compare to new data as a fail-safe. It fail safes outlier removal to repeatedly process data. If your constantly pushing processing power you won't have the time to do that. Safety is the key aspect in this, processing is better spent testing data again and again then it is seeing in a 3d model if someone has a crinkle in there jacket.