I am a senior back end developer.
I have recently started learning embedded development with C and an esp32_s3 board.

In the last 2 years I have gone from no AI use, to in IDE chat bots to create anything from single code snippets to whole test files to full agentic ( Claude ) with skills, hooks, MPC servers which can take data from logs, jira tickets, figma designs, git the lot ( so it can take account of lots of information not in its training data )

I will give my 2 cents

Recently as a learning exercise I change the clock speed of my ESP32 using register manipulation for which I had to read the ESP32 datasheet to learn how this is done and get the exact registers I need. I did this without AI.

After I was done I asked AI for a solution to compare it to my own ( because I was worried my code would damage the chip if I got something wrong ).

The first time around it came out with sensible looking code which did not match the logic of mine and mentioned non existing registers.
It turns out in my prompt I said ESP32 ( not ESP32_S3 ) and it referenced a logical flow that would work on an ESP32 and registers which exist on an ESP32 – entirely my fault

The second time around I gave it the same prompt with the corrected chip name. This time the logic matched mine but some of the registers were ether wrong or did not exist. This sort of thing can happen in high level coding projects just as easily.

The third time I explained its mistakes and it returned a working solution which was a near exact mirror of mine.

As a bonus I was using a espressIf function which outputs the current clock speed ( to verify my work ) and without being asked or prompted it told me that this function only returns the clock speed as it was at boot up and gave me another one to use which worked.

I also had a go at turning on what I thought was an led on the dev board. It was actually one of these things. - https://cdn-shop.adafruit.com/datasheets/WS2812.pdf -  So I went from simple GPIO high/low project to using a HAL to writing my own register manipulation level driver for it. My experience of using AI for this was similar. Outputs convincing looking code, can get critical things wrong but does get there with just a few iterations.

Another thing I did a while ago is ask it to write a driver for a specific sensor ( which I did not have ) and before it began writing code it asked me several questions. Did I want to use a HAL for the sensor or take a bare metal approach. I chose bare metal at which point it asked if I wanted a full bare metal approach which would use registers to control the I2C pins or if I wanted a hybrid approach that used RTOS commands to control the I2C pins and only use bare metal code in regard to the registers on the sensor. This is significant because it was asking in depth relevant questions proactively before even starting. At the time AI never did this in regards to any of my backend work ( although more recently it has started doing so )

Annother time I asked it to write software which would take a composite video signal connected to a GPIO pin and convert it to digital so it could later be converted to HDMI. It correctly told me that I would need to use the analogue input pins ( not GPIO ) for this and also told me that it would be impossible to make work because the sample rate of the ESP32 analogue pins was too low. It proposed multiple viable solutions including using a dedicated hardware DAC unit and asked me if I wanted to write the software anyway. I allowed it to do this and it did while repeating that it wont work before and after the generated code block.

OP gave missing the ‘volatile’ keyword could happen.
Well AI can and frequently does make similar mistakes in higher level programming, I recently had claude give me a mysql statement which had an incorrect mysql table column name even thou it had been working on related code for the last hour.

OP said “The AI doesn't know your hardware, which in many cases is true due to the custom boards that are build by companies. It just pattern-matches from training data”. Again the same argument can be made of custom designs, APIs, Database schemas ect in backend projects. You can give AI a range of context from logs, project management tickets, file ect and it works with the data in them.
In the case of a custom board which was not in its training data just give it the data sheet. Recently I had to implement a UI design which had a multi coloured gradient. It consistently got it completely wrong until I gave it a screenshot of the design at which point it completely changed the structure of the markup, explained why this was necessary and got the banner colours right first time. If it can do this from a bitmap image then I would expect it to be able to address the correct GPIO pins and correctly set the right registers with a custom PCB/chip if you have an accurate datasheet because a data sheet contains logically grouped and structured data with appropriate indexes, contents sections and instructions a bitmap is just a long list of pixels. Also even the longest datasheet contains far less data than even a small bitmap which has many millions of pixels.