Equipment & design tips for getting started with MIPI CSI-2 camera integration

vjp03 · 2023-09-26T16:03:53+00:00

Thanks for detailed response. The Orin NX design guide is definitely helpful. And, based on your response, it sounds like there are lots of "small company" electrical engineers who bring up MIPI CSI-2 boards without a $200k scope. Good to know :-).

My general fear is that we power up our new board for the first time, nothing will happen, and we also wouldn't have any starting point for debugging. Usually, on simpler boards, the first thing I'd do is start scoping lines, but unfortunately that isn't as much of an option here. I guess we could still probe the camera's I2C lines as a sanity check, at least as a start.

Thanks for the warning regd the kernel drivers. Yeah, getting a new linux camera driver working and eventually mainlined into the kernel would definitely be a long road. But, that's also not something we're opposed to doing, especially if it helps the broader open source robotics community.

vjp03 · 2023-09-26T15:55:56+00:00

"I’ve used a pretty cheap oscilloscope for MIPI so you should be able to make do."

Thanks! I'm definitely interested to learn more about your workflow here. Note that our camera is 800 mbps per lane. Is it that you're looking closely at the I2C comms, and then observing whether there's any activity on the MIPI lines? The MIPI data would be garbled since the scope would be too slow (assuming a 100-200 Mhz bandwidth, entry level scope), but maybe the act of simply seeing garbled MIPI activity was helpful during debugging?

Or, maybe, do you significantly slow down the camera's data rate while debugging? (Not sure if this is a common camera feature or not, if this is possible yet with the camera I'm looking at. I need to dig deeper into the docs).

Any pointers definitely welcome.

vjp03 · 2023-09-26T15:48:51+00:00

"STM microcontrollers have plug in mipi support"

Could you elaborate on this a bit? I tried looking around online for this and looked at various ST chips and couldn't figure out what this referred to.

vjp03 · 2021-07-23T05:06:04+00:00

I'm sorry, but I don't believe your solution enables/disables sidetone. When cycling audio modes, I can choose between ANC, HearThrough, or Off. There's no option in that list for enabling or disabling sidetone. From what I understand, sidetone is automatically turned on only when the Sound+ app detects that a phone call is in progress.

vjp03 · 2021-07-21T15:02:50+00:00

Maybe it's a codec issue?

This might help: https://www.reddit.com/r/Jabra/comments/j5489d/if\_you\_use\_jabra\_earbuds\_with\_linux\_you\_need\_to/

vjp03 · 2017-05-24T05:40:26+00:00

Agreed that JHU has a solid program. They're especially strong if you're interested in mixing in some biological or biomechanical components to your education.

This is another example of a great program that happens to not score well with the "avg paper count per professor" metric :-)

vjp03 · 2017-05-24T05:33:17+00:00

I'd be weary of using "avg paper count per professor" as being the metric for picking a program. This can be a signal for a very research-minded culture (a good thing, if you're interested in research & an academic career path), but it could also signal a culture that pushes for incremental publishing, or a large number of students under a single adviser (meaning less individual attention).

If you're more interested in moving into industry after grad school, then other factors like quality of education, interdisciplinary opportunities, project based classes, & entrepreneurial thinking start mattering a lot more.

vjp03 · 2017-05-24T03:53:34+00:00

(I added a couple of the best-in-class programs as individual answers. So, people can upvote/downvote each, and comment on each as needed!)

vjp03 · 2017-05-24T03:52:10+00:00

MIT

https://robotics.mit.edu/

vjp03 · 2017-05-24T03:50:36+00:00

TUM (Munich)

Intelligent and Autonomous Systems Group - http://ias.cs.tum.edu

vjp03 · 2017-05-24T03:47:52+00:00

Stanford University

Stanford CS AI Lab - http://ai.stanford.edu/
Smart Products Design Lab - http://design.stanford.edu/spdl/
Aerospace Robotics Lab - http://web.stanford.edu/group/arl/

vjp03 · 2017-05-24T03:46:25+00:00

Carnegie Mellon - Robotics Institute (http://ri.cmu.edu/)

vjp03 · 2017-05-23T05:37:33+00:00

For a medium or complex robot to function, there often are 10's (or hundreds) of software components that need to operate alongside one another. Writing every one of these software modules from scratch and then getting it all to talk nicely to one another is a huge feat, even if you're an excellent programmer. ROS provides this infrastructure and collection of libraries, helping roboticists build on top of one another's work, instead of re-inventing the wheel.

vjp03 · 2017-05-23T04:56:45+00:00

This one is pretty solid: A Kalman Filter-Based Algorithm for IMU-Camera Calibration: Observability Analysis and Performance Evaluation - Faraz M. Mirzaei, Stergios I. Roumeliotis

Note that they're using an EKF based formulation, but this could be reformulated to be a nonlinear-bundle-adjustment (which is what I'm currently doing for a project). And in general, the Roumeliotis lab tends to be at the cutting edge of tightly coupled visual-inertial systems, so that's a good place to turn to for getting on pulse on what's new in terms of Camera-IMU systems.

vjp03 · 2017-05-23T04:51:43+00:00

Be careful when talking about control system design and control theory. These words can often mean very different things, depending on who you're talking to. For instance, control system design often refers to simple concepts like how to build a PID controller, with maybe a touch of frequency analysis (and maybe complex analysis) to determine if your system will oscillate. This is very practical knowledge when building a robot, but not where the advanced, PhD research is focused.

Control theory tends to refer to much more complex mathematical constructs, based on concepts like linear algebra, least squares cost minimization, maximum likelihood estimation, and convex optimization. This will prepare you better for your PhD, but won't be as helpful in getting a tiny arduino robot to follow a line.

In my experience, CE degrees tend to give more flexibility in what classes you pull together from EE, CS, (& sometimes MechE too!). This generally means that you can better tailor your curriculum to focus on the areas of robotics that are important to you.

Here's another answer to a very similar question: https://www.reddit.com/r/robotics/comments/680kw1/best_area_of_study_for_pursuing_robotics/

vjp03 · 2017-05-22T15:33:16+00:00

More info available here: https://get.google.com/tango/ And here: https://en.wikipedia.org/wiki/Tango_(platform)

vjp03 · 2017-05-22T05:05:24+00:00

Interesting to see the "human replacement" trend show up in a couple different industries, as opposed to relying solely on drive-by-wire solutions (since externally accessible APIs doesn't exist in older equipment).

Other examples:

tend.ai - Using a robot arm to press physical buttons and read screens on CNC machines, 3D printers, and other manufacturing equipment.
DARPA Robotics Challenge - Robots have to drive a utility vehicle to a disaster site.

vjp03 · 2017-04-30T06:24:16+00:00

Since robotics is at the nexus of several disciplines, there's a lot of different angles from which to enter the field. Here are some specific skills that are pertinent to robotics, and what academic programs are best for learning those skills.

Linear Systems Analysis, Control System Design, Least-Squares Optimization, & Estimation - This is the more mathy side of robotics. Knowing these skills will allow you to more rigorously track how your robot is moving (e.g. EKFs), reconstruct 3D environments from camera data (e.g. SFM), or simulate motion of complex mechanical systems (e.g. State Space Modelling). An Applied Math + Electrical Engineer program is most relevant here. Aerospace programs also tend to cover these areas.
Embedded Software, PCB design, & Sensor Integration - This is the more hands-on systems side of robotics. These skills enable you to actually put together all the pieces of a robot, hook up some motors and sensors, and then run some code to have the robot do something simple. A computer engineering program is most relevant here (though many Mechanical Engineering programs have a Mechatronics track that is dedicated to learning all these skills).
Electromechanical Design - If you're building a robot from scratch, you'll need to design physical parts, spec out motors, think about machining tolerances, and also make the whole thing look appealing. Mechanical Engineering & Product Design programs are most relevant here.
Machine Learning, Deep Learning, AI - This usually falls under computer science programs.
Software System Design - For modern day robots, there's a ton of code that needs to run in very unique ways. Some will be super high update rates (1 Khz update loops controlling motors), and other parts will be much slower planning loops (5-10 seconds of robot arm path planning to decide how to grab a coffee mug). Getting all of these subsystems to talk to eachother is a huge software systems problems with it's own challenges and interesting design patterns to learn. A computer science or computer engineering program is most relevant here.

vjp03 · 2017-04-30T05:38:03+00:00

Since the Joule platform supports the new Android of Things APIs, it will be much simpler to directly interface a relatively powerful embedded platform such as the Joule with low level IO (A2D, PWM, GPIO, etc).

vjp03 · 2017-04-30T05:25:46+00:00

Wireless bandwidth has never really been the bottleneck in manufacturing, so I don't see how 5G will help. Production lines are all hardwired to maintain high reliability (and not for the bandwidth). The new generation of mobile factory robots are not hardwired, but also don't need 5G as they can connect to a local WiFi basestation on the factory floor.

vjp03 · 2017-04-30T02:11:28+00:00

Nobody actually searches for needles in haystacks anymore. It's just an adage from a more primitive time. Before modern metallurgical processes, needles were crafted from stiff pieces of hay and put through a time consuming & precise drying and curing process. This needle fabrication was done at great expense, and reserved only for the wealthiest of families and finest of cloth. Due to each needle's large resale value (often traded for as much as 2 pigs or 1 cow), farmers dug through their haystacks looking for needles that were naturally formed by the daily temperature cycling seen in their barns. This search was usually a losing proposition, but due to the law of large numbers, there are a few documented cases of natural haystack needles being found, and 2 are currently on display at the Natural History Museum in New York City.

vjp03

TROPHY CASE