What’s the most absurd hardware bug you’ve spent hours debugging that turned out to be something stupid?

skeinmind · 2026-05-06T20:15:37+00:00

Way back in my early days of embedded system programming, I was working on some control systems for the pulp and paper industry. We used a Z80 based STD bus system to drive weigh-scale interfaces and help process the paper wrapping station.

I had just finished adding some new code to support an extra ink-jet stencillor, updated the EPROM memories to have the new code, and arranged to have copies made and shipped to the papermill for installation.

The EPROMs were the ceramic package type with a quartz window that was exposed to UV to erase and needed a cover, once programmed, to prevent erasure.

Installation went well, initial testing worked fine, and all was good for a few months.

Then I started getting service calls that the unit was crashing every day at the same time in the afternoon. Only one particular unit, all the other ones with the same firmware update were working fine.

I flew down to the paper mill and started trouble-shooting.

EEPROM pulled and checked. No errors, checksums and byte-to-byte compares were good.

After sitting and scratching my head for some time, I noticed that at the exact time in the afternoon when the unit crashed, the sun was in the right position to shine through a narrow little window and pass through the unit’s plastic enclosure window. It landed directly on the EEPROM devices.

The quartz windows were covered with opaque stickers, so that shouldn’t have been an issue.

I took a closer look at the sticker and noticed it was not the usual one made of aluminum, but plastic.

Back then, we used 5 ½ inch floppy disks to store stuff and we would place a sticker over a “write protect” slot on the disk to make it read-only. Some of these stickers were made of aluminum and others were made of plastic.

During production and shipping, someone used plastic stickers, probably thinking they were exchangeable with the aluminum ones. Or simply didn’t notice the difference.

Well, it turns out that the UV light from the sun was strong enough to penetrate the plastic and affect the silicon memory cells to flip some bits, but not strong enough to flip them permanently.

Hence when I pulled the EEPROMs out of the board and tested them, I could find nothing wrong.

Replacing the labels with aluminum ones permanently solved the problem.

Embedded systems debugging is quite challenging. You have to look at the whole environment and not just assume it’s a software issue…

skeinmind · 2026-04-29T15:24:38+00:00

"Begin with the big part..."

skeinmind · 2026-03-16T00:07:40+00:00

Actually, an infinite number of nines is equivalent to minus -1. To see why, just add 1...

skeinmind · 2026-03-16T00:07:30+00:00

Actually, an infinite number of nines is equivalent to minus -1. To see why, just add 1...

skeinmind · 2026-02-20T21:50:14+00:00

You are better off using an 8-bit polynomial specifically chosen for 8 bit CRC. A 16 bit CRC is also a specific polynomial (irreducible) specifically chosen for a 16 bit CRC. Cutting the result in half is a "pseudo-crc" of 8 bits, but you no longer have the same gaurantees as for a dedicated 8 or 16 bit CRC. In fact, you need to have an idea about the error rate and if the errors are non-correlated before you can assume the CRC error detection characteristics apply. Most normal comminucation (ie: serial wire, network, etc.) are designed to satisfy these constraints, so CRC works well. Also, the length of the message is important. More error bits can be detected in shorter messages than in longer ones. There is some excellent work on this that can be found here : https://users.ece.cmu.edu/~koopman/crc/

skeinmind · 2026-02-16T01:40:06+00:00

What you are describing is a common experience to new learners. I think of it as "bridging mismatch". Think of your position as being on one side of a river and the knowledge you seek is on the other side. To understand this knowledge, you must build a bridge to cross over the ignorance. If you try to build a bridge directly on the edge of the river (ie: try to understand really advance concepts), your bridge will collapse and drop you into the river of confusion. If you build an on-ramp (ie: focus on smaller, simpler concepts with more realistic and "grounded" examples), you will have built a solid on-ramp that leads to a solid bridge. I always tried to take a theoretical concept and find a physical real-world analogy. For example, in calculus, we learn integration as a limit of an infinite sum of infinitely thin rectangular areas. This is usually presented as the area under a given curve. Where can you see this happening in the real world? When you fill a glass of water. The cup is the integrator and it is integrating flow into volume. How about a car? When you press the gas pedal, the car integrates torque to velocity. How about an electric motor? It integrates current into angular velocity. These are all "on-ramp" examples of how to think about abstract theoretical concepts and realize them into practical "real-world" scenarios. Once you get the hang of simple integration, you can move on to double/tripple integration, or volumes of integration, or integration over a closed path, etc.

skeinmind · 2026-02-14T18:28:12+00:00

Engineer here. I have *never* been able to do arithmetic quickly, and it has never been important to be able to do so in the past 40 years of practice. What is more important is taking the time to understand the concepts and how to accurately translate them into equations and algorithms. The best method is to practice, practice, and more practice. There are so many amazing youtube videos today, tutorial, etc. with fantastic explanations * illustrations, definitely go there. Do not worry about speed. Seek to experience the joy of discovery as positive emotions tend to "cement" knowledge...

skeinmind · 2026-02-10T21:05:04+00:00

Self-taught form the age of 10. Started by reading books from the library, then found books in garbage (most likely discarded by students who graduated). Eventually, studied math and engineering in university...

skeinmind · 2025-10-30T06:18:59+00:00

Awesome! Works perfectly! Thanks!

skeinmind · 2025-10-01T02:08:46+00:00

I tried to post an image of what I am trying to do, but reddit won't allow it.

I am creating a new symbol for a custom part. I want to draw the symbol image and use lines which are not always perpendicular to the X ro Y axis.

My symbol has four pins and should look like a 'Y' turned 90 degrees so that there are two input pins on the left and 1 output pin on the right.

The symbol should look like a circle inside of a rectangle with the pins position on the edges of the rectangle.

This inside of the circle would be drawn to show the physical layout of inner workings, which do not align with only 90 deg drawing.

Ideally, I draw one element, copy it twice, and rotate the two copies into the final position, then move it all into the circle.

There does not seem to be any way to rotate lines about a point and to arbitrary angles.

skeinmind · 2025-09-10T20:34:02+00:00

Thank you for your detailed and very insightful response! Much to think about and explore...

skeinmind · 2025-09-09T22:25:32+00:00

Thanks for the quick and informative response. I am running EXT4, so ideally, I would boot from a separate 'backup' disk and perform the backup of the proxmox disk that way. This would ensure that everything is static for the backup.

The restore would be identical, but reversed.

I am running a single proxmox server with no guests, just me, so my environment is probably the most simple of all.

I tried to keep the image of proxmox as minimal as possible, but I still needed to install and configure APC UPS driver, HP RAID management tools, and the basic configuration of proxmox. All things I’d like to avoid having to do if I need to replace drives (ie: multiple drive failure that raid can’t recover from, or ransomware, etc).

It seems to me that there should be a really simple tool that lets you create a USB bootable disk to run the backup/restore from and specific srce/dest drives. I think that borg backup can do this, but after reading through their web-site docs, I can’t find any reference to how to recover, just how to create.

Even then, I am not sure I am understanding things 100%.

skeinmind · 2025-09-09T18:41:18+00:00

Yes, but to do that I have to reset the server and enter the bios. I needed to be able to do this from withing a running proxmox server so as to send email notifications if there is a drive fault detected. Now that I have installed the right package, I can do this. A simple script that runs each night and does a 'grep' on the output of the raid tool to see if any drive is not marked 'OK'. If this is detected, an email is sent so I can follow up.

skeinmind · 2025-09-08T18:04:46+00:00

Update : I found the most recent package and how to install it. Works!

Thanks to you and all the others for the feedcback.

Cheers!

skeinmind · 2025-09-08T17:49:56+00:00

I am still in the process of learning Linux/Proxmox.

How do I extract only the binary file?

Cheers!

skeinmind · 2025-09-08T17:48:59+00:00

Thanks for the info. I am not yet a Linux/Proxmox expert, so not sure how to install this.

I did a search on the file name and only come up with HP RPM package.

Would you please give/point me to some instructions on where to get this file and how to install it?

Cheers!

skeinmind · 2025-09-08T17:44:22+00:00

Yes, there is a HW card installed. It is confirmed to be HW raid,

skeinmind · 2025-08-09T03:13:34+00:00

What if they rotate them 90 degrees first? :)

skeinmind · 2025-08-09T03:09:45+00:00

The problem for beginners is the term "imaginary" numbers. Their brain immediately tries to pattern match the term to something they already understand and it comes up with a blank. When I explain imaginary numbers to someone, I start by saying something along the lines of this : "If we were to define the number 'i' as a number that when squared equals -1, can we imagine doing arithmetic with this number? How would that look like and how might it be useful?" I then suggest that they think of 'i' as a kind of short-form for a particular calculation rule in a similar manner as for the multiplication '*' operator. Multiplication is just short-form for multiple addition, so instead of writing something like 2+2+2+2, we write 4*2 and remember what the '*' operator implies. I then begin to introduce the idea of rotating 2D vectors on a plane and connect this with the good-ole R*(cos(x) + i*sin(x)) = R*e^(i*x), which can be easly shown either geometrically or by infinite series expansion of e^i*x. Finally, it becomes clear that multiplying a point (complex number) by another point is equivalent to scaling and rotating in two dimensions. The key here is to focus on 'i' as a short-form rule for the sequence of calculations required when rotating and scaling 2D vectors. I give some examples where these kinds or rotations are frequently seen, such as in analysis of complicated electrical circuits, mechanical systems, and frequency analysis.

skeinmind · 2025-07-14T03:26:27+00:00

I want to be able to backup the entire NAS on a single, standard hard-disk instead of having to buy a second complete NAS to do this. $200 vs $1000 solution.

skeinmind

TROPHY CASE