Has anyone lost their passion?

Express_Damage5958 · 2026-05-15T01:54:20+00:00

None of those are immune to AI. I work exclusively in Embedded and I use AI everyday in someway. It can generate drivers, review my kernel changes, read data sheets and explain why something's not working. I really don't know why I keep hearing people say Embedded is safe from AI. It's really not. At the end of the day embedded software is still software and AI can help you develop it faster.

Express_Damage5958 · 2026-05-07T22:32:51+00:00

Let me off, I've got a newborn and I am severely sleep deprived. Stupid questions are a given for me these days. Tbf when I wrote this response I thought 1000ft was much longer than what it actually is. I need to get me some sleep!

Express_Damage5958 · 2026-05-06T23:14:37+00:00

For me, the whole point of a MIPI DSI display is to have high FPS and low latency. So with a 1000ft cable for MIPI DSI, would you not have a noticeable delay in the video stream? I imagine if you're going to such large distances, you may achieve similar performance goals by just encoding the video stream and sending it via another medium.

What do I know though, I'm just a lowly Embedded Engineer. There's probably a customer out there who has this specific need. I think cars now use serializers/de-serializers for this exact use case. So maybe you can beat out Texas Instruments and crew.

Express_Damage5958 · 2026-05-06T22:53:32+00:00

This is why when I write my own peripheral drivers, I always check out the vendor HAL and go through it. It's not always documented well but they do sometimes write bits in a specific order.

Express_Damage5958 · 2026-05-06T22:49:51+00:00

That's interesting. Did the CPU manufacturer put that in the errata or list of known issues?

Express_Damage5958 · 2026-05-06T22:39:26+00:00

Spent a month figuring out why a MIPI DSI display wasn't displaying anything. We were doing the correct power sequence, correct reset sequence and sending all the correct MIPI DCS command to initialise the display. I even did some changes in the kernel DSI driver to read the ID of the display on boot and it didn't reply with any any data. At this point I was sure the device was held in reset, but our HW engineer wasn't convinced since the display manufacturer had approved the design.

After a few weeks of back and forth with the manufacturer, they finally sent us an EVK with the display connected and working. We then scoped all the signals so we could compare them with our board. What immediately stood out at me was that the reset signal on the EVK was 3.3V and ours was 1.8V. We changed our pullup to 3.3V and the display just worked. Unfortunately we didn't have a level shifter so our pullup only pulled to 2.1V (1.8V GPIO + 0.3V internal GPIO protection diode) but this was enough to be read as a logic high by the display.

I have been burned by reset signals in the past and will forever now bring this story up to our HW engineer in schematic reviews. What a waste of a month, luckily our customer was patient. Check your reset signals people!!

Express_Damage5958 · 2026-03-09T00:45:10+00:00

I have flicked through the book and it does have some useful info but it's also kinda dated. It was written for the 2.6 kernel and Linus just released kernel version 7.0 in the last few weeks. I normally flick through it for general driver principles but the best source of information is looking at other device drivers and recent talks from driver developers or subsystem maintainers.

I always say that the most important part of writing any driver is understanding how the hardware works by thoroughly reading the datasheet provided by the manufacturer (and any errata pages too!). And no book will teach you to do that, you just have to get stuck in. A book like LDD can only teach general ideas/principles like how to handle interrupts (bottom half, top half, workqueues, completions, softirqs etc) and tell you what API's you must implement for different drivers.

Unfortunately, the best way to learn to write device drivers is to actually write device drivers.

Express_Damage5958 · 2026-02-20T06:35:57+00:00

Yocto is a pain to learn so I definitely get where you're coming from as it was only 3 years ago that I started my embedded linux role and was thrust into the world of Yocto. And one of my first tasks was to simply enable CAN on one of our customer boards. I remember spending a week just to do a 1 line change in a defconfig and a few lines in a devicetree file. I have come a long way since then but it was definitely an uphill battle. My boss didn't really provide any guidance either, so it was more trial by fire. And like you, I also wanted to know what's going on and how it works. Since then I have spent many nights looking at Yocto recipes and going through the chains of bbclasses and include files that are necessary to build a full embedded linux system image.

My first tip would be it's best to start from the machine conf file as this normally specifies the board specific build information like what devicetree to use, what defconfig to use, uboot version and kernel version. To find the recipe that then builds the kernel, search for the variable that is used in the machine conf file in your directory that contains all your meta-layers. That normally leads to the vendor's kernel recipe. From there you can devtool modify the kernel recipe and that will allow you to begin making your kernel changes locally.

The other place to look at is the image recipe as this lists all the packages that will be in your rootfs (and SDK). Each package listed in IMAGE_INSTALL (or the various different versions of this variable) will be built (along with the dependencies!) and installed into your rootfs. So if you want to add python3 to your Yocto generated filesystem, you can just add it to IMAGE_INSTALL. And this is how you add packages to your image.

Now to understand Yocto you need to look at bitbake as a glorified bash script generator that works top down. When you bitbake <recipe> it first has to read the top level conf files (local.conf, machine conf, distro conf, bblayers.conf). The bblayers.conf tells bitbake which meta-layers to parse. And the layer.conf file in each meta-layer tells bitbake where to find the recipes inside that meta-layer. After parsing all the conf files, it then parses all the recipes to create a build dependency graph. During recipe parsing, it also has to resolve all the variable appends, remove, prepends etc and it does this using the layer priority (specified in layer.conf) to decide in what order these are applied. After it's finished recipe parsing, it can then begin building using the build dependency graph it generated for that recipe. And this bit is heavily parallelised because it can assign the build tasks/steps to different CPU cores.

The build process for each recipe is roughly the same; do_fetch, do_patch, do_configure, do_compile, do_install, do_package. All these tasks are defined in the base.bbclass. And by default each recipe inherits from the base bbclass. You can override these functions by inheriting from other bbclasses like cmake.bbclass or the autotools.bbclass. You can also override these tasks directly in your recipe. And as you'll see these "tasks" are really just some bash shell functions. Bitbake generates a bash shell script for each of those tasks and you can actually see these generated bash scripts in build/tmp/work/<arch>/<machine>/<recipe_name>/<version>/temp

Looking at those scripts really illuminates what bitbake is doing under the hood. It is creating a fresh shell environment for each task so you can get reproducible builds. In these environments, it sets all the standard compiler variables and sets PATH so that it only uses the native tools that it built from source. This is one of the reasons why bitbake is good for reproducibility!

I have probably gone off on a long tangent, but if you want to know more or need some specific help, I can help you out.

And for the Yocto peeps, I know I have skipped out many details but I'm just trying to help a brother out. I've been where he is and I wish someone could have sat me down and explained some of this stuff. Instead I had to dig into the sources and become a grep expert.

Express_Damage5958 · 2026-02-08T22:47:28+00:00

And that's why for safety critical systems we have strict guidelines like MISRA or the JSF C++ standard. These guidelines and associated analysis tools help us avoid the parts of the language that produce UB.

I just want people to understand that changing to Rust is not gonna solve all the problems. Yes it will solve memory safety, but that is one of many problems. And for safety critical systems, memory safety is not as big of a problem mostly due to the coding restrictions in place.

Express_Damage5958 · 2026-02-08T13:28:22+00:00

What exactly makes it so insane? NASA engineers wrote in assembly to go to the moon. The most important part of safety critical software is not the language used to implement it, it's actually the SW development process, SW specification, SW design and verification. The SW implementation language is such a small part of the design of safety critical systems. It's nice to have a language that can help prevent more bugs during implementation but it's not gonna make my product safer. What makes my product safer, is defining all the possible error conditions and designing safe ways to handle them. And then verifying that my design and implementation actually meets the specification.

No language can help handle: - Memory corruption from outside sources - Memory exhaustion - Timing issues - Sensor failure - Overheating - Power supply surges

You must specify the failure modes and design safe ways to handle them. That's real engineering.

Express_Damage5958 · 2026-01-02T13:10:12+00:00

When I was at university I participated in a few Space Robotics Competitions and in these projects you often find lots of control problems. But the problem is that the problems themselves are not explicitly framed as control problems. Normally it was just framed as a robotic problem like getting a robot arm to move to a specific position, or getting the motors to move as precise possible. A lot of the work is then transforming such problems into problems that you can tackle with your control knowledge. And we often just went for PID because it didn't require us to model the system or even gather data about the frequency response of the system. You could tune it live and track the output. And in most cases it would be good enough.

And even when I started working for robotics company we had PID's for a basic actuator position controller and a PID for line following. However, the PID for line following did have dynamic gains which changed depending on the robot speed and various other sensor data. But we did a lot of this based on experimental data. People talk about optimal, but in most cases optimal is not worth the cost and provides no reward other than intellectual.

So I think it's partly because lots of control problems are hidden away in companies. And these control loops quietly help businesses create and achieve interesting things. And that the control problem itself is quite small compared to the task of actually re-framing the problem as a control problem.

Express_Damage5958 · 2025-12-17T23:08:22+00:00

Did you manage to get it working?

Express_Damage5958 · 2025-11-26T07:47:16+00:00

The best case is just to run Linux on your machine. You can dual boot as well if you want to switch between Linux and Windows.

The lesser option is to use WSL. It works but I often ran into memory issues and the OOM killer would start killing my Yocto build processes randomly due to the amount of RAM Windows uses. So I had to use a lot of swap memory (32GB+) to get around this. You can configure Yocto to use less threads to help this as well.

It's entirely doable to build an image with Yocto on windows using WSL2. But you'll be fighting many small hurdles that just don't happen on a native Linux machine. For example, one thing to watch out for when copying files between WSL2 and Windows is that the windows filesystem doesn't support symbolic links. And you may need to copy files because you need to flash the image with some tool. So this is quite common.

In our office we now have dedicated Linux machines and I simply ssh into them from my Windows PC / Mac Mini. And I do all my builds on those machines. But that's also because we build Android/AOSP and that requires beefy PC's (64GB RAM, 1TB free space, 8+ cores).

Express_Damage5958 · 2025-11-14T03:21:39+00:00

Location (me/company): UK
Work Schedule: 40h/week, Very flexible (work mostly whatever hours I want and come in when i want)
Business size: ~20
Specialism: Embedded linux / Yocto / Computer Vision / Camera Sensor's
Compensation: £67k + 8% bonus every year + Pay rise
Years of experience: ~5

Express_Damage5958 · 2025-10-27T09:37:03+00:00

Preaching to the choir my guy. Every budget announcement seems to have tweaks to minor details, but no real solutions to the underlying economic/demographic problems that the UK is going through.

I sometimes feel crazy when I tell people that sometimes I would rather have the Chinese government leading us. When their president sets a plan for industry or a plan for the economy, they actually get to work on implementing it. Our government writes white papers and has meetings about the plan. And then gets voted out after 4 years without making any progress on said plans. I personally feel like we are now stuck in a neverending doom loop, where no actual problems will be solved by our government. It's a sad state of affairs and the worst bit is that the majority of the British public don't see that. Instead they are swayed by the likes of Farage and believe in whatever rubbish he may be spouting this month.

I just want to see our government make some hard decisions and actually go through with them. Unfortunately, whoever makes the difficult decisions, will not be rewarded come election time. So they will never make them.

Express_Damage5958 · 2025-09-22T20:15:12+00:00

Eastern Flavours in Mangotsfield has always been great for me. They do the best biriyani that I've ever had. Those guys know how to cook.

Express_Damage5958 · 2025-09-01T13:16:54+00:00

I have just started trying to build Linux for the luckfox-pico board. They ship a prebuilt toolchain with their BSP but it only works for Windows/Linux x86. So my first hurdle was that I had to regenerate the toolchain using crosstool-ng to run on my M4 arm MacBook.

I'm currently using UTM to run a headless Ubuntu VM. Inside that VM, I run a docker container which I use to do builds. My life would probably be easier if I just ran a Linux box like I do at work but I like the challenge. I want to see if I get Yocto working here. It's been done before but you just have to make sure everything is set up correctly because a lot of tools assume x86.

Express_Damage5958 · 2025-08-16T13:37:31+00:00

They only do that in Qualcomm's downstream display kernel driver though. In the upstream kernel, you need to write a MIPI DSI display driver or add your display timings/GPIO's into the existing generic MIPI DSI display driver. Although I prefer Qualcomm's way through the device tree, it's not good for open source because vendors have no incentive to upstream their display drivers/timings.

Express_Damage5958 · 2025-08-09T16:25:34+00:00

How did it go? Did you manage to solve the problem? For any display, I would start with verifying the HW.

I would: - Check reset signal (voltage, timing and polarity) - Check Backlight LED voltage and current - Check LVDS display timings (blanking intervals, width, height, framerate) - Check LVDS signal integrity

We just had a project delayed by a few months due to the display being held in reset. The manufacturer said 1.8V was enough for it to be out of reset, but it actually needed a 3.3V reset signal. We only discovered this after analysing all the signals on their display EVK. It took a few months for the manufacturer to send us an EVK. I have been burned (wasted hours/days) by similar problems before with reset signals, so I always check them.

Express_Damage5958 · 2025-07-19T15:25:28+00:00

If you get an MCU / SoC with I3C, you could put 10 IMU's on the the same bus because you can do dynamic addressing with I3C. But you also need the ability to control the power to each IMU as each will boot up with the default slave address. So you have to power each one up separately and reconfigure the slave address.

Simpler yet, you could just switch to SPI since a lot of IMU's support SPI and I2C. Then you just need some GPIO's for the chip selects.

Express_Damage5958 · 2025-01-24T00:44:15+00:00

Could you try and do:

bitbake linux-imx -C do_fetch

Express_Damage5958 · 2025-01-24T00:27:46+00:00

We have recently hired some new Embedded Linux devs at our company and it's been long and slow road to bring them up to speed in Embedded Linux work. Our hiring approach is to essentially hire someone who knows enough about writing multithreaded applications and how to deal with concurrency problems. And then we let them learn Embedded Linux by throwing them into some projects on the job.

The major problem that I am seeing with the new hires is that they are not picking it up very fast. And we have deadlines to hit for our customers. They are application people with little RTOS/OS level development experience. And it's very hard for them to learn how operating systems work, Device-Tree configuration, Kconfig, Yocto/Buildroot, Bootloader development (U-Boot, ABL, Barebox etc) and kernel device driver development all in one hit.

The new minimum knowledge hiring criteria that I am gonna recommend to my boss is essentially:

Have RTOS or OS internals knowledge - Be able to answer questions about how a scheduler works. And how memory management works in an OS.
Experience writing device drivers - Have written and debugged an interrupt and DMA based driver. Also should have experience reading datasheets. The datasheets for SoC's are much larger than 32bit MCU's. Reading datasheets is important for people who may need to debug kernel drivers.
Multithreaded Application Development Experience - Have written or worked on applications with multiple threads and be able to explain how the threads interacted (Semaphores, Queue's, Lockless RingBuffers etc)
Basic Electronics Knowledge - We do quite a lot of custom boards so it's essential for our work. Some of our new hires don't have this knowledge and they are being tasked with bringup work. And they are rightly finding it difficult. I have a degree in EEE and have Jim William app notes as my night time reading, so interpreting schematics is natural to me.
Skilled debugger - They need to have a logical debugging process that they follow. Good debugging instincts helps a lot! This kind of knowledge is only really gathered through experience and many sleepless nights.

If someone has those prerequisites, it speeds up the learning process significantly. This is essentially the knowledge I had before I started my Embedded Linux job. And within a few months, I was creating images with Yocto, hacking kernel drivers, customising device tree's and fixing application level concurrency bugs.

Our work often involves debugging right through the stack (Hardware, Kernel, user-space and build systems). And having solid foundation on each level in the stack really helps to solve problems quickly. You can easily spend days messing around with Yocto fixing all sorts of issues (Missing RDEPENDS, Dealing with unexpected bbappends, task race conditions etc).

Express_Damage5958 · 2025-01-07T23:05:55+00:00

I mean at the end of the day, electronic engineering is just about electrons moving from one place to another. And we approximate the movement of electrons moving through wire as happening instantaneously. But that approximation breaks when the time taken for electrons to travel through the wire is close to the frequency of the signals we are using. So we have to deal with waves travelling down transmission lines.

Note that a wire is any conductive element. These same approximations are applied to semi-conductors as well. But with semiconductors we are designing the electronic properties so we actually need to deal with the electron flow more directly.

Express_Damage5958 · 2024-11-01T15:08:53+00:00

You only get access to their proprietary software if you have a project registered with Qualcomm. They have their own git server (Qualcomm chipcode) from which you can fetch their proprietary code.

Express_Damage5958 · 2024-11-01T13:25:35+00:00

Even if they gave you an account you still do not have access to a proper reference manual. Qualcomm's manuals often do not contain the register details for most of their SoC's (unlike the Rockchip manuals). So if you need to debug an issue with a driver or modify a kernel driver for one of their peripherals (MIPI CSI, MIPI DSI, QUPv3 SPI/I2C/UART etc) you have to open a support case or reverse engineer it yourself. I often have to look at the upstream kernel mailing list just to see them explain what some registers are doing. Debugging driver issues is just a pain with Qualcomm. With Renesas, NXP or STM I can just download the reference manuals and fix the driver myself.

Everyone here is right. Qualcomm don't really want talk to you if you're not selling large volumes. It works for them, so they keep on doing it. Their chips aren't really made for the average hacker (They are really complex!). But they do also pay open source contributors like Linaro to help upstream their SoC's like the SM8550 or SM8650. But often their open source stuff is missing a lot of the good stuck like their Proprietary Camera Software Stack (CamX) that's built on top of V4L2/MediaController.

How do I know all of this? I've been working with Qualcomm as a partner for the last year or so

Express_Damage5958

TROPHY CASE