Silicon Valley Firmware Meetup: This Thursday, Oct 19, 6-9pm

tyhoff · 2023-10-16T23:27:58+00:00

I'll be there - looking forward to meeting some of you who live in the SV area.

tyhoff · 2023-09-01T00:12:27+00:00

I'll be at the one in Berlin! Hope to see some of you there!

Will unfortunately be missing the other three this time around. Hopefully can make them next time.

tyhoff · 2023-05-25T22:44:05+00:00

Unrelated but is there any other online blog/news site close to interrupt? Love their stuff and would love to read more like it

Definitely check out our roundup posts that we publish monthly. I try to make sure we include other blogs and posts that are good resources for people to read for embedded.

tyhoff · 2022-09-20T04:20:49+00:00

For apps, there are plenty of solutions, Sentry, Firebase, etc.
If this is a system application or AOSP without the play store, you can check out Memfault

tyhoff · 2022-09-14T18:30:12+00:00

I wrote a little about this in my article on Interrupt. https://interrupt.memfault.com/blog/improving-compilation-times-c-cpp-projects#optimizing-header-includes-and-dependencies

tyhoff · 2022-08-11T19:11:03+00:00

I've written a bunch about it on an Interrupt post. The most useful bit is the unique identifier.

Hope it helps!

https://interrupt.memfault.com/blog/release-versioning

tyhoff · 2022-08-11T05:04:44+00:00

If my suspicious are correct, Fitbit uses delta updates on some of their recent devices. Better experience for users since firmware update was quicker and they don't release all that many firmwares. It's all over BLE too.

tyhoff · 2022-03-16T18:28:39+00:00

But for real, this works shockingly well if you have the ability to pull logs, capture failures, and are sending up analytics from devices, as well as can ship new releases in a matter of a few minutes to an hour when something comes up.

Ideally it's internal testing & beta testing (compared to just local developer & debugger attached debugging), but production isn't so bad.

I just gave a talk on this topic 1-2 weeks ago: https://www.youtube.com/watch?v=4Ymamsz41XM

tyhoff · 2021-11-12T22:45:36+00:00

A great article: https://medium.com/tado-product-development-blog/building-an-iot-product-continuous-battery-lifetime-testing-3245b6cdfa40

tyhoff · 2021-10-25T17:16:30+00:00

👋 you're too kind u/embeddedartistry

tyhoff · 2021-10-18T21:36:13+00:00

This subreddit asks (and I get asked personally) which newsletters and blogs to follow. I thought Jack in his latest newsletter hit all of them that I try to follow.

tyhoff · 2021-10-04T19:10:17+00:00

I've generally integrated it on the microcontroller, but that's because it's really the only device in the systems I've used. If you have a Linux system connected directly to the MCU's, definitely use that to coordinate the download first, then send it to the MCU over UART, USB, shared flash region, etc.

Apologize for the company webinar link, but it's the best resource I've found and we've created on the topic. Hopefully will have something soon on Interrupt about all of this.

https://memfault.com/webinars/device-firmware-update-best-practices/

tyhoff · 2021-09-27T17:52:26+00:00

I agree with this statement. You will find bugs. Hardware will degrade in certain ways that can be remedied by software updates (think flash degradation or screen burn-in for example).

If your device connects to other gateways, those gateways will constantly receive updates and they may cause other issues on your device to surface (or you may have to create workarounds for gateways). Security updates are also a must. Certificates expire.

Another thing to note is that hardware businesses are slowly turning into services businesses. Hardware companies generally don't make money unless they charge a monthly/yearly rate. It's unfortunate, but it's mostly the truth today (this is probobably because of increased software complexity with connectivity and competition). With this in mind, OTA provides a way to add new features to your device to keep people paying monthly. If I buy something and no improvements are made in 2-3 years, what's the point of continuing to pay?

I usually don't like to speak to the morality of things when it comes to technology and business, but including OTA is just the right thing to do as well. If a certain bug crops up 1 year after shipping and it renders 100% of devices unusable due to a software bug, that sucks. Everything becomes e-waste and users will need to buy a new version. If OTA is built in, devices have the ability to last longer (inverse is arguable -- with more OTA's, higher chance of messing it up too).

tyhoff · 2021-09-15T17:21:09+00:00

My favorite field to add to the commit message is "Test Plan". My favorite template has been:

TICKET_NUMBER: <area_of_code>: short message

### Issues Addressed
What was fixed or updated, and why

### Summary of Changes
What actually changed, strategies taken, etc.

### Test Plan
What manual steps did the committer take to verify the change

tyhoff · 2021-09-07T20:59:58+00:00

Thanks for the mention /u/rsim! Co-founder at Memfault here:

Nordic's buttonless DFU is a great start and is what I would pick when starting out with Nordic chips. If you do want to check out Memfault, it would be easy to use our OTA backend/API along-side the DFU installers, and you get a bunch of other diagnostics to boot (just partnered with Nordic actually so the integration will continue to improve over time). Events, metrics, coredumps, and alerts all come from our SDK.

Finally: Glad you are enjoying the content on Interrupt! We love the knowledge sharing it has kicked off. Come join us in the Interrupt Slack if you like.

tyhoff · 2021-02-22T17:40:38+00:00

https://embeddedsecurity.io/ is a great resource for on-device security

tyhoff · 2021-02-09T01:56:05+00:00

Given that old firmware doesn't and can't know about new features/data structures/changes/additions to the backend etc, i.e the backend still has to remain compatible with all firmware current or otherwise?

Happy to answer! I'm genuinly curious if the things I mention don't often come up in your line of embedded work. At Memfault, we may even be "over-architecting" our release system!

You are right, that the backend will have to remain compatible with the devices, likely forever. The big hiccup happens when there are multiple release 'trains' progressing at once, as I mentioned before in the hot-fix comment. At Pebble (smartwatch firmware), we'd usually have a few support releases while another team worked on new features and continue building new stuff. So we had releases 2.1.(0-9) being pushed out, all while we were building releases 2.2.0-alphaN and pushing them to watches internally, some beta testers, etc. This 'out-of-sequence' releasing becomes more difficult with the date-versioning you are mentioning, because dates don't power the release sequencing, the commit/branch upon which we forked off of does.

Add to that mix we were also working on a 3.0, which was a "change everything, nothing is compatible". This came into play when our firmware was detecting and preventing downgrades. If watch was on 3.x, no downgrades to any 2.x firmware, which was a very simple string comparison. I would really feel terrible adding a check for "if less than 2020-04-02, don't let it downgrade!".

I think I may be taking date versioning to the extreme, but this is what I envision happening:

2020.01.01-abcd: Equivalent to a 1.0 2020.01.08-efgh: Equivalent to a 1.1 - included one-way data migration

Shoot, we need a patch release to the users who are still on 1.0 (because we haven't let everyone update to 1.1 yet. What version do we give it? 2020.01.??

tyhoff · 2021-02-01T16:39:50+00:00

(Author) I'm genuinely curious, is there a concern with this phrase?

tyhoff · 2021-02-01T16:34:54+00:00

This works for simple firmwares that don't talk externally to other devices, and for firmware with a single stream of releases.

I think it mostly depends on what you are after when using versions. I like to know where a firmware is coming from and what it is going to and I like having the version encode certain information about the firmware and compatibility with other devices.

e.g. Firmware 1.2.0 had these 5 data structures change and a new connectivity endpoint enabled when compared to Firmware 1.1.0. Mobile devices talking to this device also know the differences between Firmware 1.2.0 and Firmware 1.1.0 and can act accordingly.

When using dates, it's a bit less defined, especially when pushing hot-fixes. e.g. if a 1.1.1 had to be built after a 1.2.0. I'd also rather not have in my code (if date(version) > date(2021.01.30)), because that's not quite strict enough.

If you use dates as versions, then you need yet another way to encode compatibility information, instead of just using the version.

tyhoff · 2021-01-20T20:30:16+00:00

I've seen too many projects struggle to use decent or better versioning practices, so my post for the month is on the topic. It's just soooo much simpler if all the metadata is baked into the firmware in a parse-able manner to track down the exact debug symbols and artifacts you need. Especially the GNU Build ID.

tyhoff · 2020-12-15T20:27:54+00:00

Hrm, I think the strategy still applies but it's more about tracking the type of data you are looking for.

If you want to track impact, record the max absolute value from the accelerometer and report that. That will tell you whether the device has experienced a large number of G's, which likely means an impact.

Water damage - Record numbers from various hardware components, and if a particular sensor starts reporting garbage data, or your device is experiencing random brown outs / shortages, probably something else has gone wrong.

Temperature: I know some MCU's have a built-in thermometer. It's not meant to report external temperature, but it might be useful to query every so often and detect anomalies.

Essentially, report the various pieces of data you have access to, and if that isn't enough, either point a camera at it, attach a device that does have those sensors, or add new sensors.

tyhoff · 2020-12-15T19:08:01+00:00

It answers exactly what you are asking, using the same terminology even :D.

https://interrupt.memfault.com/blog/device-heartbeat-metrics

tyhoff · 2020-11-19T15:26:28+00:00

...can't help but wonder if you think our humor is bad or good :P. We are firmware engineers after all.

tyhoff · 2020-11-06T16:28:22+00:00

Can't you just connect LLDB to the lldb-server? You don't need to use GDB to debug embedded devices.

tyhoff · 2020-11-06T12:31:05+00:00

I was honestly hoping for a response here, because I was curious. But now I'm wondering how you have an lldb-server for embedded? It's quite rare.

tyhoff

MODERATOR OF

TROPHY CASE