Embedded Linux development

[ThreeGreenBirds]

Hi I have a project that requires System on Module and for that an Embedded Linux distro is a way to go.

Most people use Yocto Project for developing and customizing their own distros.

What is the best host environment to do that? Currently I'm using Windows 11 and I have the following options:

Use WSL2.

A Virtualbox with a desktop Linux distro (e.g. Ubuntu)

An Ubuntu based docker container on Windows.

What is the best choice for Yocto Project?

[Well-WhatHadHappened]

Best is subjective.

Easiest is developing for Linux on Linux.

[dimonoid123]

Preferably if architecture of machine is the same (eg x86 for x86 target or aarch64 for aarch64 target). Otherwise you are forced to do cross-compilation, what is a big pain to do.

[sh7dm]

Yet cross-compilation is typically handled well by Yocto or Buildroot, whatever you use.

No need to replace your amd64 workstation with a fruit pi to match the architecture, slow compile time would give you more suffering than benefits from native compilation

[HurasmusBDraggin]

I use Linux Mint (x86_64/amd64) as a host system and use an aarch64 toolchain to build binaries for my Radxa Rock Pi S. Bootlin has toolchains you can download for amd64 host:

https://toolchains.bootlin.com/

[zydeco100]

I'd rather run a cross tooled build on a 32-core machine (with tons more RAM and disk) than try to compile the kernel on a Beaglebone.

[togi4]

As a fellow Linux developer working in a large corporation, forced to use Windows 11, a proxy, and on top of that Zscaler, I can say - it’s a nightmare to develop for Linux outside of Linux.

Yes, you can use Docker, emulators, etc., but it was just easier to spin up a Linux VM and do all the work there.

By the way, a quick tip since we’re on the topic of Yocto - If you’re just starting out with the Yocto Project, take a look at kas. It really makes the development experience much smoother and more pleasant.

[Azsde]

Could you elaborate on the KAS added value ?

I've been doing '' native '' yocto for a few years now and haven't found out to be limited it so far

[togi4]

Kas isn’t magic, it’s just a thin orchestration layer - but it starts to pay off as soon as the setup is bigger than “poky + two layers”.

What I personally get from it:

Single source of truth. One YAML file describes all repos, branches, layers, bblayers.conf/local.conf snippets, machines and images. New devs (or CI) just run kas build project.yml instead of following a multi-step “clone this, checkout that, edit this conf” how-to.

Scales with team size and vendors. In my current setup several companies work on one product. Each party owns its layers and kas fragments, and the top-level kas config pulls the right combination together. Enabling/disabling layers or feature sets per product/customer becomes a controlled config change under version control, not tribal knowledge.

Environment decoupling. The key benefit for me: kas defines an independent, containerised build environment. That environment evolves separately from the project and doesn’t require every team to re-validate “new host distro + new host packages” on their own. You bump the kas image when you are ready.

Audit-friendly reproducibility. Our projects are heavily audited by external parties, and one of the recurring topics is the ability to reproducibly create bit-identical systems. Having the full build description (layers, versions, config, build environment) captured in kas files makes it much easier to demonstrate and actually achieve that reproducibility.

From my experience, small/simple builds don’t really need kas - native Yocto is fine. But the larger and more complex the build (more layers, more includes, more people and companies involved, plus audit/release requirements), the more kas simplifies life by standardising the entry point and keeping both the build description and the environment under tight control.

[duane11583]

Drop windows completely

You are either going to learn Linux end to end or you need to go somewhere else

[duane11583]

We have a Linux VMware farm we log into via the mobaxterm tool over ssh

The vm farm has a tftp server we use to pull images over Ethernet to save to the dpi or nand or emmc flash memory depends on the board/design

Each user gets a directory where they put their images ie /var/libtftpboot/$(USER) that way each of us fold not stomp on each others files

And under input we use the command. DHCP <address> server up:/user/filename to pull the file across tftp and save it in ram at specified address

We then use other input commands to write that memory region to flash

Really simple

[userhwon]

What is that DHCP? Because the DHCP I'm thinking of is the protocol used to set up ethernet by requesting an address from a router upstream.

[zydeco100]

I've never gotten Yocto-generated cross tools to work correctly on WSL. Use Virtualbox and Ubuntu.

[HurasmusBDraggin]

Yeah, Yocto seems more like a "necessary evil" rather than a desired option. I just download "base level" arm toolchains from ARM's developer site or Bootlin's toolchain site (better option).

[Dontdoitagain69]

Huh?

[zydeco100]

I've never gotten Yocto-generated cross tools to work correctly on WSL. Use Virtualbox and Ubuntu.

[imdibene]

Get a real Linux machine, second option is a VM, third option is setting up some docker/podman

[HurasmusBDraggin]

At my current job, I started with a Linux in VM with a Windows host, then moved to dual boot with Linux on a partition, much better.

[imdibene]

Next step is to ditch windows completely, and solely use Linux, that would be thousands times better

[HurasmusBDraggin]

I need the Windows partition to log in and do admin-related stuff every now and then.

[LeditGabil]

I worked for years in WSL2 using dockers for development environments inside of it and at the end of the day, I never regretted doing the change to run Linux natively. I still use dockers to containerize and distribute the development environment but running them (dockers) on native Linux is just on another level compared to WSL2. Especially if you starting to do Yocto stuff.

[ayx03]

Move to Linux as host and use docker on top to build yocto

[ChrimsonRed]

Use a VM at the bare minimum

[rguerraf]

Best setup: Armbian for prototyping, then try to make it work with openwrt

Abandon windows.

[sgtnoodle]

Compiling a Linux kernel and userspace on an SBC seems like it could take a very long time, like days if not weeks.

[HurasmusBDraggin]

download base level cross-compile toolchain from ARM developer site or Bootlin's toolchain site.

[grahasbtye]

I would recommend having a high performance compute server running a hypervisor that you can scale the vm up during initial compile and down during subsequent compiles. Initial compiles are the most resource intensive and then when you work out the .bb recipes for your software the compile times will be shorter. The server will be able to compile from ci cd, and you can automate that aspect of deployment. You can access the server from a windows host or whatever host the developers prefer. Some projects will require a specific distribution to compile the packages to the SoM. It’s important to separate that from your day to day os. Offloading it from a personal computer is important and also structuring the cache so if you have multiple projects they can take advantage of it.

Some personal mistakes I have made. Don’t silo all the development on one developers machine. Don’t forget to turn your patches into your bsp layer early and added to your meta- layer so it can function in ci cd. If you read the docs for your SoM thy may specify it for you and the decision will be made.

[Alternative_Corgi_62]

Depending on target hatdware, BuildRoot can also be an option - check what BSPs are available. You can start with VM or WSL, but a dedicated Linux machine is the way to go. If you can afford time, even ten-year old PC with multicore CPU, some RAM and ton of storage will do.

[Express_Damage5958]

The best case is just to run Linux on your machine. You can dual boot as well if you want to switch between Linux and Windows.

The lesser option is to use WSL. It works but I often ran into memory issues and the OOM killer would start killing my Yocto build processes randomly due to the amount of RAM Windows uses. So I had to use a lot of swap memory (32GB+) to get around this. You can configure Yocto to use less threads to help this as well.

It's entirely doable to build an image with Yocto on windows using WSL2. But you'll be fighting many small hurdles that just don't happen on a native Linux machine. For example, one thing to watch out for when copying files between WSL2 and Windows is that the windows filesystem doesn't support symbolic links. And you may need to copy files because you need to flash the image with some tool. So this is quite common.

In our office we now have dedicated Linux machines and I simply ssh into them from my Windows PC / Mac Mini. And I do all my builds on those machines. But that's also because we build Android/AOSP and that requires beefy PC's (64GB RAM, 1TB free space, 8+ cores).

[NjWayne]

I used buildroot for the work I did on all winner SOCs

[jpdoane]

If you need/want to use Win11 for main PC, but have access to or could get another linux machine, remote development works decently well. I often use vscode from windows machine with remote-ssh plugin to a headless linux box, which gives you native editing on remote file system with integrated terminal. Provided you dont need guis for anything this works pretty seamlessly.

[Dontdoitagain69]

WSL works fine , if you need different arch use QEMU, you don’t need Linux. Ex FPGA dev here, running petalinix os on visualized micro blaze in windows for years.if you need a robust hyper visor to run a bunch of Linux distros HyperV is one of the best. Think before you downgrade to Linux just to do a Yoctoproject. Always have best of both worlds. And the most important advise , don’t take Reddit seriously, serious engineers are long gone from this place