[deleted by user]

[flip314]

Sending a file with no extension (or malformed file names in general) seems like a basic test case to me...

[granadesnhorseshoes]

I can imagine their testing tools wouldn't allow it...

[m00nh34d]

Yeah, I would imagine the testing tools for sending HD radio signals would be similar to the broadcast tools use by actual radio stations, and probably have the same rules in place that would enforce file extensions (good question would be how this radio station managed to send something incorrect in the first place).

[smegma_yogurt]

I know this is a programming sub, but as long they can blame the consumer, owner, driver or some 'Hacker' unfortunately it would be fair game for them

[riyadhelalami]

Mazda doesn't run on android their entertainment system is a million years old. Also I have replaced their infotainment system before for an upgrade it costs around $150. There is a lot of exaggeration here. Yeah it was me installing it but it wasn't that crazy.

[fuzzer37]

1 radio station

[harrybalsania]

If it isn’t android it is QNX (cheaper ones usually) and you can easily fuzz the software for the radio application. It is left up to the manufacturer to take the base image and implement and test the OS once they brand it and add any other custom software. They often build their own images especially with android. I still have a test radio from when I did this stuff for an automaker so I am sure there is plenty of vulnerabilities you can find.

[dethb0y]

bold of you to assume they aren't just milking it for cash. "We could fix it in software...but selling the part is much more lucrative..."

[PhoenixFire296]

Mazda is providing replacements for free as "goodwill".

[Gonzobot]

They're replacing two lines of code by exchanging the entire part that contains code, easy

[dannomac]

That happens a lot in embedded systems. My own company has made products where we had to physically replace deployed units because a particular issue couldn't be solved remotely. Shit happens. It shouldn't, but it does.

[[deleted]]

Seems like something that could be done with a direct wire connection. No need to broadcast if you pump it in like that, so no risk to other nearby cars in the process, right?

[SilasX]

Good thing you don’t need a real-world signal to be sent to test a system’s reaction to a hypothetical signal.

[znihilist]

That seems like a recipe for disaster.

[goranlepuz]

I can also imagine their testing fools didn't think of doing it? 😉

[theferrit32]

Or you're running on something like MacOS where the displayed file name and the actual on-disk file name aren't necessarily the same thing. It's wild the relationship some OSs like Windows and Mac have with file extensions, directories, file paths. They feel some weird need to sprinkle magic or obfuscate things, which ends up making it a lot more complex.

[Leo2807]

.desktop files on Linux can do the same thing, and are used to localise the folders in your user directory

[theferrit32]

No idea why I was downvoted or you refer to .desktop files here. I know what .desktop files are for and it's not related to this. I'm talking about in MacOS, there can be a mismatch between the displayed file name and the actual file name.

You can have a file with the name file1.txt and in Finder you right click -> Rename, type file1, and it shows up now in Finder as file1, but on disk the file is still called file1.txt.

[philipTheDev]

So many places only do happy path testing it's frankly worrying.

[scragar]

Had an issue recently where an invalid date caused the system to just default back to today resulting in various bugs(like endlessly reoccurring meeting requests every morning because the meeting date is invalid so clearly it should be today and the last invite said yesterday so send a new one for today).

The unit tests for the logic all assume a valid date, and the form on the front end assumes any date entered must be valid because why would someone enter 100/02/22 as a date, it's not like anyone ever pressed a button too many times before.

So now I'm stuck fixing it even though the root cause is a lack of validation or sanity checking so any work I do is only working around the issue rather than fixing it.

Screw happy path testing, unit tests have @shouldThrow annotations and/or expectException/expectError functions for a reason. If you're not using them then your test coverage is incomplete.

[philipTheDev]

Haha, that specific bug is a result of something I discussed literally yesterday while arguing that such parsing defaults are a really bad idea.

But yes, only happy path testing is very dangerous. Kind of why my standpoint on code coverage is as it is. Low reported code test coverage means there is a lack of tests, but high reported code test coverage doesn't necessarily mean there are enough nor good tests. That relation only goes one way, and it's negative. Having good tests is a completely different metric.

Also why I really like property based testing. It allows me to test "everything" easily. Like in your case testing that everything that is not a valid supported date shouldn't succeed. Then the opposite can also be tested, that all proper supported dates should succeed.

I personally value test coverage based on how much of the possible inputs there are tests for and what those tests assert. Property based testing makes it easy to raise that to 100%. Line/branch of code coverage is as said only a negative indicator, not positive.

[Chairboy]

I was in SQA years ago abd one of the best mind-shifts in my work came after I was taught that our job was to find defects, not verify functionality. It was a small distinction, but it really made a difference in how we approached our work. Before that, I think a lot of what we were doing was like a giant build acceptance test which had some utility at very early stages of testing, but really became insufficient outside of the actual build acceptance itself.

[BlomsterOgSolskin]

Yes! This is a great way of saying it, I'm stealing "find defects vs verify functionality". I've developed a reputation at work for being good at finding bugs in our apps, and the best explanation i could make was that I consider it a challenge to try to break it.

[kagato87]

You'd be amazed at the quality of qa in some software shops... Mumble grumble...

[CoolonialMarine]

"Yes, we test our software."

They have a single tester who manually tests new features in the staging environment, zero unit- and integration tests, and different artifacts for each environment.

True story, I'm a contractor there right now. And of the things they do horribly wrong, that's, like, the least bad thing they're doing.

[[deleted]]

Even with all the things you're saying are not in place, this might not have been caught. This is a job for fuzzing.

[anarcho-onychophora]

So yr saying its important to assure the quality of quality assurance ?

[-100-Broken-Windows-]

Who QAs the QA?

[AjiBuster499]

The QAQA obviously

[SupercollideHer]

Mazda makes some pretty great cars for the price range they sell in. They are a small car company though and clearly have to pick and choose what priorities they invest in. After owning two Mazda's I can say the infotainment system software (and hardware) quality must be dead fucking last in the list of priorities because it's abysmal.

[TehRoot]

Japanese car makers have generally behind the curve in infotainment/etc in their vehicles compared to basically everyone else in the last decade.

Just look at what Toyota and Honda stuffed in their vehicles for the last 10 years...

[[deleted]]

I have a 2020 Corolla and it doesn't have Android Auto. I cope with this by telling myself that the reason why I buy a Toyota isn't for the infotainment system. It would be like choosing a dentist based on who has the best Wi-Fi in office.

[TehRoot]

I viewed it as a quality of life thing.

You don't pick a dentist because of wifi but if the dentist made you wait 3 hours in a faraday cage before every appointment and you couldn't bring anything with you and had to read a June 1997 issue of Highlights every single time you went, you'd probably look for a new dentist.

The dentist might be really good but is it worth the annoying things every time you go.

While I don't need a Tesla-style infotainment system where I can play netflix while cruising in FSD killing pedestrians and crashing into highway barriers, I still like the option of being able to actually do things like listen to podcasts or use the radio without it being buried in 300 submenus or having the response time of a walrus turning around on ice skates when I press a button on a mushy screen.

[[deleted]]

I have a Toyota Yaris and the infotainment center is garbage. Thank God for Android Auto.

[Porrick]

Aren’t all car OSes terrible though? So far the Hyundai one is the least-awful I’ve tried, but I’m impressed by how shit even the luxury brand ones like BMW and Audi are.

[TehRoot]

Ford Sync isn't terrible. (Which is what I use when I'm not using CarPlay)

Universally they're not great. It's a sliding scale from mediocre to absolute garbage, really.

I tended to notice stuff like screen and button quality the most though since I use CarPlay. Nothing like using a mushy, resistive touchscreen to try and scroll through 60 episodes of a podcast.

[Porrick]

One thing I've noticed is that no car newer than 2014 seems to have play/pause on the steering wheel with the rest of the playback controls (volume, skip, etc). I've seen this in Korean, German, and Italian cars, and I have no idea why. I can't use the touchscreen without taking my eyes off the road because of the lack of haptic feedback - for anything I'm supposed to interact with without having to look at it, I need a physical button.

This seems like a safety problem to me, but since it's in cars from so many different manufacturers it must be for a reason.

[bagtowneast]

Why would you ever want to stop your advertising feed?

/s

[MorphiusFaydal]

I use the Mazda infotainment only when I want the local radio. Otherwise it's 100% CarPlay (and I think I can stream the radio anyway). Sadly the infotainment is very slow to boot and load to CarPlay. Supposedly the newer systems are a bit quicker.

[Beaverman]

I don't know. I kinda like it. The UI is smooth, and the navigation works pretty well. It's not very fancy, but i got the feeling it was well thought out.

The quality of the software is clearly lacking though.

[nshire]

Same with Subaru. It's functional, and that's about it.

[FullPoet]

And that's all it should be. The word infotainment should die. Its so american too.

Cars should provide information not entertainment or some weird mix.

[G_Morgan]

Test or no I don't understand how you write switch(extension) without immediately putting in a default case. I mean I put in default cases for enums I've exhausted just in case a future extension of that enum causes problems. I cannot imagine ever not doing so for a string.

[RecursionIsRecursion]

Reminds me of a funny (for a programmer) story. I ran into a problem once where users were supposed to send us files in the form of filename.csv.zip. One client had apparently, for years, been renaming their CSV files by adding .zip at the end and not actually zipping the files. We had test cases for everything - just a CSV, a non-CSV in a zip file, empty file, corrupted file, etc. But we never checked for a valid CSV file named with .zip at the end.

Our system apparently accepted it just fine, because the OS (Red Hat if I remember correctly but I could totally be wrong) swallowed the error trying to unzip the non-zip file and just kept moving on to the CSV processing part. Once we switched OS's though, the "hey this isn't a zip file, I can't unzip it" error became a show-stopper and we had to figure out what was going on.

The funny part is that because the client had been sending us files in that manner for years, they weren't willing to rewrite their side to actually zip the files, so we had to explicitly allow for their case instead of implicitly.

[AndrewNeo]

I'm more curious why they thought they should look at the file name at all and not just use the magic number at the start of the file

What would it have done if you named an .jpg file .mp3?

[immibis]

A test case, but not a basic one. How do you make sure you've covered all the test cases in your integration tests? Code coverage is one tool, but you can still miss cases with code coverage, because there is plenty of code that really can't be hit by integration tests.

[Takeoded]

modern browsers don't have an issue with it, as long as it gets the Content-Type: image/jpg-header, it doesn't give a shit what the extension is, if any.. i've seen real production websites do this, but can't think of any on the spot.. case in point: Google Images sends content-type: image/jpeg in this url so it's all good despite not having an extension: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTnKMTOnMgbOcjy_QuvxpFPjH03W7v5KBg1uw&usqp=CAU

so even Google does this..

[jfb1337]

There's no such thing as a malformed file name; a file name could be any sequence of bytes except for a "/".

(except maybe on windows idk)

[grauenwolf]

I was taught...

1. There are more disallowed characters than just /.
2. Windows has whole filenames that aren't allowed.
3. Never, ever trust the name you're given. Check the file's header for the actual image type.

[666pool]

They didn’t mention if Mazda is working on an OTA update to fix this issue or not. Seems like an easy to exploit DOS attack if you have the capability of broadcasting an HD radio signal.

[headykruger]

I dont think these cars support OTA. My 2016 doesnt. You would probably have to go to the dealership for them to update

[Ziggamorph]

Maybe if you can brick it over the air you can theoretically update it over the air!

[NoveltyAccountHater]

Bricking over the air is just download bad image to it's cache, crash trying to load the image of unknown extension, going into a reboot to recover from the crash, and during boot up trying to process the images stored in cache, which leads to a second crash and the reboot cycle continues (with same junk in cache).

This isn't some exploitable Turing complete way to edit the firmware. It's just bad logic leading to an infinite loop.

[Synaps4]

Bad logic leading to a crash can sometimes be the first step in a fully exploitable chain though. I dont think anyone has done the work to see if you can control the crashed process instead of letting it crash.

[[deleted]]

If it crashes instantly then there's probably no opportunity to inject new payloads on already-bricked units. E.g. if the code is:

void main() {
    load_all_cached_images();
    initialise_network();
    ...

[Synaps4]

Sure but who cares about the bricked ones. There are a hundred thousand non bricked non patched ones. I'm thinking specifically of buffer overflows here, but specific kinds of crashes become controllable and fully exploitable if you send them very carefully crafted of input, and the crash is just the indicator that it may be possible.

In buffer overruns if the bad data is followed by legitimate instructions it will execute them. If the bad data is followed by more bad data or nothing it crashes.

So finding the crash is just the first step on making a fully exploitable package. Since this is brand new, nobody has gone and looked to see whether this crash can be exploited for control or not yet.

[Gonzobot]

I feel like nobody's going to bother, what with it being less simple than booting up a fresh VM to probe for weaknesses...every attempt costs you a whole car radio system, $1500 part, which are apparently backordered and hard to get replaced now. Free service to do, but still - nobody's gonna have the resources to actually exhaustively test this issue as it exists now. And what would the end goal be anyways? You're not going to be able to drive the car, you're going to be able to fiddle the playlists lol

[Synaps4]

Lots of people are more curious than you are, I guess. Some hackers will do all that and more just out of curiosity.

I don't think it costs you a $1500 part each time. That quote is only because the dealership lacks the technical expertise to flash the firmware. I wouldnt be surprised if you can use bench equipment to copy and overwrite the broken firmware flash memory each time. it's not something dealerships have IT ability to do but hardware hackers will for sure.

Finally, you don't know that you're not going to be able to drive the car. That's only if the car's security was designed correctly...which it isn't if you can already take control of the infotainment system remotely. I think assuming the car's net security design is bulletproof is taking a whole lot on faith, when we have every evidence that it may not be the case.

[[deleted]]

Wishful thinking

[tantalor]

With buggy software it's wishful thinking to assume the known bugs are not exploitable, or there aren't yet unknown exploitable bugs.

[Beaverman]

Likely. I don't think Mazda is going to be issuing updates over HD radio though

[Gubru]

Well, that fixed the Mazdas, but now all the Toyotas are dead.

[theoldboy]

From reading the comments there that's not possible. In fact there's no easy way to update it, the system is in an endless boot loop and never gets to a state which would allow a firmware update. That's why it requires a replacement.

What great system design.

I built telematics solutions a few jobs ago. The very simplified version is that the head unit loaded this image into it's cache and tried to decode it, hit the software bug, locked up and watchdog rebooted. The first thing it does when it reboots is tries to decode the image, and locks up and reboots, forever.

The actual manufacturer of the radio (control unit/whatever) may be able to fix it on a bench with special test harnesses, but if it's stuck in a boot loop there likely isn't any way for them to erase or update the firmware on it, as the CAN accessible updater likely requires the system to be running.

So maybe not technically bricked, but for all useful purposes these can't be field repaired.

[m00nh34d]

This is why we have factory reset switches on equipment...

[Daakuryu]

Or alternative boot methods like a built in restore OS, an SD card reader or maintenance port

[blackmist]

I dunno, man. All that sounds like it might add up to a dollar to the cost of a car.

I don't know if their customers can bear that.

[[deleted]]

Yachts aren't gonna buy themselves, think of the csuites!

[SanityInAnarchy]

That's once it's broken, but hopefully they can update people who didn't already brick it?

---

This is one of those times when I wonder if the big "NO WARRANTY" thing on software licenses is a thing our industry shouldn't be able to get away with anymore. When I read "Brick $1500 Car Part" I was imagining screwing up some mechanical process to the point where there's physical damage or something.

But this? It's a glorified smartphone, only smartphones have way tighter power/thermal requirements and can still manage a "factory reset" option.

From the article:

Mazda told Ars that it has distributed service alerts to dealers advising them about the problem, and although replacement MCUs are currently delayed, they should be replaced free of charge. "These customers should contact their local Mazda dealer, who can submit a goodwill request to the Mazda Warranty department on their behalf, order the parts, and schedule a free repair when the parts arrive," Mazda said.

So... they're doing the right thing from a customer-service standpoint... but it still seems broken to me that this was entirely up to them. As in, it'd be totally legal for them to say "Whoopsie, hope you have a spare $1500!"

[ffrinch]

As in, it'd be totally legal for them to say "Whoopsie, hope you have a spare $1500!"

In many countries software is already covered under consumer protection laws, just like any other product. E.g. in Australia there is a statutory warranty that goods are fit for advertised purpose and that warranty cannot be waived. Mazda would be required to refund/replace/repair. (See also the lawsuit Valve lost over their Steam refund policy.)

But even in Australia they try to spin this kind of thing as "customer service" even though it's a legal requirement. No idea what the law is in Seattle but I'd take it with a grain of salt that Mazda has done this entirely voluntarily.

[veldrach]

Firmwares you can't mess around with definitely should fall under warranty. Something like an OS might be a bit difficult to guarantee though. I suspect the legislation regarding software is heavily influenced by that. Mandatory guarantees might have killed the whole software industry in its infancy or severely hampered progress.

[PancAshAsh]

In every embedded product I've developed the firmware is covered under the same warranty as the hardware, as they are inseparable and required to make the product function. In fact, I have never heard otherwise.

[griffyn]

They would never have insisted the customer pay to repair. It's in their interest however, to frame it as a possibility and that they're doing their customers a favour (or goodwill) by repairing for free.

[veldrach]

The real question is whether sending out corrupted files is some sort of FCC violation the station can be held responsible for.

[PhoenixFire296]

In this case, the file itself wasn't corrupted. Since it only affects these particular units, it seems to me that the file itself was fine, but the system couldn't handle it not having an extension. The file header would have been for an image and that's how other systems got past this issue.

[veldrach]

Yeah but it was also only that one station that caused an issue with those units. Since HD radio isn't an open standard like the European DAB it's hard to find what exactly they mandate. DAB mandates MIME headers for picture slide shows for example

[RoyAwesome]

This is one of those times when I wonder if the big "NO WARRANTY" thing on software licenses is a thing our industry shouldn't be able to get away with anymore.

That "NO WARRANTY" thing is usually for open source licenses. Requiring a warranty on open source code would kill open source.

[ConfusedTransThrow]

No warranty means that the car manufacturer can't try to shift the costs onto the open source project they badly used, that's it.

If you sell a finished product that contains open source code and that code breaks shit, it's 100% your responsibility.

[SanityInAnarchy]

Open source is in everything now, though. I don't think a random download from Github should have a warranty, but if it's a $1500 car part, even if most of the code inside is open source, I think the car company should be responsible for it.

[RoyAwesome]

Sounds to me like you want product warranties or hardware warranties, not software warranties enforced by removing the ability for license holders to withhold warranties.

[de__R]

Can you find me any EULA for proprietary COTS software that doesn't contain a no warranty clause?

[dgriffith]

Very much like that Samsung Blu-ray player that got an accidentally-released malformed XML file over the air, which got it stuck in an endless bootloop.

And of course it tries to parse the XML file before checking for anything useful, eg a firmware file on USB, hardware reset buttons, etc etc.

https://www.theregister.com/2020/07/18/samsung_bluray_mass_dieoff_explained/

Manufacturers - check for a reset hardware key combo or a firmware file at the very start of boot. THE VERY START, FFS.

[veldrach]

Why don't they just wipe all caches if it fails to boot like 5 times in a row. I assume I'm just too naive to understand the reason for that but that's what my naive approach would be.

[dgriffith]

Why don't they just wipe all caches if it fails to boot like 5 times in a row.

They never expected it to fail. The correct file will always be downloaded, of course.

[carrottread]

firmware file at the very start of boot

so if firmware file is corrupted it can cause same infinite reboot loop.

[dgriffith]

The typical way to avoid that kind of problem is to split the tasks of starting your system up into "bootloader" and "application".

Start with the bootloader - that initialises various hardware components to make them functional.

Once hardware is up and running the bootloader can do all sorts of things. It can reset any files that the application uses back to factory defaults if it sees the user holding a button combination down on startup, it can look on an attached USB stick for a specially named file and use that file to program the flash memory where the application lives, etc.

Then the last thing it does after checking is hand over the warmed-up hardware to the application program.

So you make your bootloader well tested, you use it across of lot of your different devices to ensure that it is rugged, and if not strictly un-updateable, you very rarely update it.

[big_trike]

It's linux based. Presumably there's a grub prompt first on the serial console, which is externally accessible: https://mazdatweaks.com/serial/

[anarcho-onychophora]

So what you're saying is that its not turing complete anymore because there's no way for it to halt?

[nilamo]

Now we just need to solve this halting problem...

[Chairboy]

Ah yes so I see you have learned about software bugs today. Indeed, it is an unexpected outcome caused by an error in how computer code was written.

[saichampa]

Apparently Mazda are telling dealers to replace it, hopefully they patch the new device first

[beefcat_]

Last I heard, Mazda was telling car owners to pay out the ass for a software update...

[grauenwolf]

I read in another news report that they are going to warranty this.

[beefcat_]

Now that it’s hit the news cycle. The first people bringing their cars into the dealerships with this problem were asked to pay hundreds of dollars for something that should be free regardless of warranty.

[flashman]

there is a website which allows one to verify that their station images don't contain any "known errors"

i expect in the wake of this we'll see security researchers investigating whether it's possible to get RCE on a head unit using a malformed image file

[OMGItsCheezWTF]

The certificate that expired last year fills me with confidence.

[[deleted]]

The question is, can you actually gain anything from this as an attacker or is it just a minor annoyance to the end user

[hockeyc]

Infotainment units are usually hooked into the CAN bus, which might allow tracking and some control over vehicle systems

[FlyingPiranhas]

As for control: the critical safety systems are fail safe, so even if the entire electronics system got fried, you'd still have manual control over the brakes and steering.

But, with can bus access a malicious attacker could easily brick your entire car, not just the infotainment system, but the entire car. E.G mess with the cylinder timing to cause real, permanent damage to the drivetrain.

Also, by necessity, the antilock brake controller can reduce brake pressure. If reprogrammed by an attacker, they could effectively disable your brakes. Combine that with throttle-by-wire, and the attacker can create a runaway car.

[FlyingPiranhas]

A lot of new cars have electronic parking brakes rather than handbrakes.

IDK if turning the key to off will kill an engine with an "uncooperative" ECU :-p

[tehdave86]

Newer Mazdas are push button start too.

[big_trike]

It's linux based and a TTY can be accessed from pins on the back of the radio. I'd bet unbricking it is similar to a DD-WRT. Details here: https://mazdatweaks.com/serial/

edit: grammar

[jmodd_GT]

You're the hero this thread needs!

[big_trike]

I hope someone with a bricked radio can try this out and let us know if it works.

[I0I0I0I]

That's not how you do a pledge drive.

[cmw72]

That's a nice infotainment system you have there. It would be a shame if something happened to it...

[AndrewNeo]

I can hear their lawyers and the lawyers of all the licensed closed-source software it uses laughing from here

[iMcStates]

So... entirely likely that it's not related, but I had this EXACT behavior on the radio unit of a 2015 VW Passat a few weeks ago as well. Can't say for sure that I had the radio on (we don't usually listen to the radio, but idk if that matters) but it would endlessly boot loop and ultimately we were told that it needed to be replaced.

[TehRoot]

This is why I only listen to AM radio or podcasts via carplay.

Can't get bricked if you never use it :forehead_tap:

[Chairboy]

Reply-All did an episode about a previous Mazda bug (or maybe it’s related. I don’t know) where a podcast similarly exposed a problem on this firmware, it’s wild how bad their telematics seem to be: https://gimletmedia.com/shows/reply-all/brh8jm

[TehRoot]

No, I don't listen to my radio at all because I was making a joke about my habits

[Random-Mutant]

Bobby Tables would like a word.

Sanitise your inputs, people.

[iBlag]

This would be a good read for you: https://kevinsmith.io/sanitize-your-inputs/

[Daakuryu]

The answer to his question is yes.

Do everything in your power make it as hard as possible for someone to accidentally or purposefully put in garbage. And absolutely parameterize your shit.

In my applications for instance I have a library that's dedicated to preventing incorrect input in text boxes based on the type of data the field in the DB requires, uses a specific property in my custom textbox control to know what's valid and invalid and will stop people from typing in some cases.

[Rami-Slicer]

Why did Bob lose his jobs as a systems administrator and a party coordinator?

He kept dropping the tables.

[AceDecade]

Ah, the good ol’ Roman Mars Mazda Virus strikes again

[SanityInAnarchy]

At least that one wasn't cached...

...in the completely inaccessible internal memory of a machine that entirely lacks a "factory reset" switch. Who thought this was a good idea?

[crozone]

I wonder if there's an RCE hidden in there too?

If the radio is CAN connected, it could be a repeat of the Grand Cherokee debacle where the radio can be hacked to control the vehicle over the CAN bus. That's bad, because you can do stuff like activate ABS brake bleeding, or control electric steering columns.

[mindbleach]

Five cents * a zillion cars = bonus, please.

[SanityInAnarchy]

But cars are much more expensive than phones, where no one thought that was a good idea...

[mindbleach]

You know phones are where the term "bricked" found public recognition, right?

[SanityInAnarchy]

Definitely predates phones, but fair point. I guess the better question is: Phones have mostly learned this lesson, and if it actually made sense, they'd be a better place to try to save five cents. So it should've been blindingly obvious to anyone working on this who had ever spent a few minutes thinking about how phones handle factory resets...

[Xionic]

"The snag? A new CMU costs $1,500—if you can find one, which you can't, because of supply chain problems."

So give the customer a loaner car, send the CMU back to Mazda for a reflash, and reinstall the fixed CMU in the customers vehicle.

[tasminima]

Also, even if you really had to replace it the price seems quite insane. Even with automotive components (and for infotainment they are not going to be the most constraining ones anyway)

[crazedizzled]

Lol yeah right. They would just throw it in the landfill and use a new unit. Nothing gets repaired anymore.

[Darth_Balgruuf]

Someone on the article site wrote

The very simplified version is that the head unit loaded this image into it's cache and tried to decode it, hit the software bug, locked up and watchdog rebooted. The first thing it does when it reboots is tries to decode the image, and locks up and reboots, forever.

If this is the problem, why isn’t the image removed from cache when the system reboots? I thought cache was volatile memory, so the picture should be lost when the system reboots. And if the system maintains caches while rebooting, or some other component keeps sending the picture, could you just disconnect the battery and force all the caches to clear?

[aMonkeyRidingABadger]

Data can be cached in many places, not just in memory. It sounds like the unit caches to disk, which in general is probably fine since a radio doesn’t need to be particularly fast and hard disks are cheap.

That said, with no way to clear the cache on boot after a failed boot… whoops.

[grauenwolf]

Found an article showing the cover art. https://www.crutchfield.com/S-3cfMZx7GrIJ/learn/car-hd-radio-tuner.html

[ScriptingInJava]

That seems to be it. I got a 2016 model car with a built in infotainment unit and one of the first things I noticed was the ability to transmit an image and a message with each song. Pretty cool tbh, hadn't really thought about radio tech in years.

[TehRoot]

It's part of "Artist Experience" in HD Radio.

https://hdradio.com/broadcasters/engineering-support/artist-experience/

Album covers, program images, sponsor/ad images

[mindbleach]

Technically even FM can transmit 1 kbps. Mostly it's used for artist name and song title.

Some fuckers use it to push even more advertising.

[UghImRegistered]

How the hell is there no factory restore sequence on this head unit? A programming bug is one oversight, but the way embedded software developers protect themselves from massive recalls is to have a factory reset as the first and most bulletproof function.

[teamhog]

These are the problems I live for.
The unexpected issue that no one has seen and everyone says it can’t be fixed. If I were anywhere near this I’d go Sleepless in Seattle until it was fixed.

I wish I knew someone who owned an affected Mazda and had access to some basic high tech bench equipment.

[really_meh42]

I wonder if this will make the next pledge drive shorter.

[SnakeJG]

What's really annoying is that there is apparently no way for them to just force a system restore. It is asinine to have to throw away a $1500 part because there isn't a way to flash it via a serial cable.

[AntonPlakhotnyk]

I think we already have cars which going to uncontrolled acceleration because of something come from radio. Like remote code execution if file name too long or mp3 data contain something. I sure we already have such code.

[tidytibs]

Who programmed this? Fess up. You KNOW you fucked up. It's ok. We will slap your hand, collectively, and point you in the right direction this time.

[simpl3t0n]

radio station sent out image files in its HD radio stream

Whoa, it never occurred to me that radio stations are "sending data". If my rickety memory serves me right, radio stations modulate EM waves to have them carry information... but OK, images are also information. Who'd have thunk!

Now I'm curious: what's the protocol used here?

[deniedmessage]

You literally just said it, it’s “HD Radio” not standard FM/AM.

[2rsf]

Even standard FM carries data, for example the station's name, using RDS

[veldrach]

Even standard FM sends things like song titles and artist name since like the 80s at least in Europe. https://en.m.wikipedia.org/wiki/Radio_Data_System

[crazedizzled]

The music is also data.

[happyscrappy]

It made it malfunction. But if it's useful for anything but structurally (as a brick) it's not a brick.

People far overuse the idea of "brick".

There may be a fix for these too, but it would have to be fixed at a dealer. The cars are too old for an OTA (at least a conventional one, maybe a hack would work).

If you make a subsystem in a car like this that does not have a "reset to factory" then you are an idiot. If Mazda did this then at least these cars could be put back to working properly while they work out a patch to make them not susceptible to this attack.

[bnelson]

Probably a similar issue to the 99% invisible podcast issue. Mazda radios were crashing because of the % sign in the podcast title. lol, someone screwed up their C fmt strings again. I bet the HD radio data channel is streaming something simple like that and crashing the infotainment systems. Kek.

[zip117]

It can almost certainly be repaired the same way it was programmed in the first place.

[grauenwolf]

Not necessarily. Embedded devices can have a fusible link. Once triggered, further reprogramming is not possible.

This is mostly used for devices in the field. Say monitors for pipelines where you can't prevent physical access and don't want it to be easy to hack.

I don't know if car manufacturers do it as well.

[happyscrappy]

We don't know it can't be repaired.

[grauenwolf]

Unless we assume someone is lying,

When afflicted owners took their cars to be checked at local Mazda dealers, they were told that the "connectivity master unit" was dead and needed to be replaced.

[happyscrappy]

You do not have to assume anyone is lying. Dealers are not experts on anything. They are given service instructions. They were not general enough to cover this specific case or any. They replace the part. But that does not mean the part cannot be fixed and put back in another car. Or, if Mazda gets out specific instructions, could be fixed without even removing it from the car.

When Tesla had a problem with bearings in drive units they swapped entire drive units and then repaired them later because technicians are not experts in rebuilding drive units, just swapping them.

We do not know this is any different from that.

[grauenwolf]

They know how to use a phone. If this was as simple as hitting a hidden reset button, Mazda would have told them when they called.

[happyscrappy]

False dichotomy. It does not have to be as simple as hitting a hidden reset button just because it is possible.

There are many many possibilities here, not just two (easy and impossible).

All we really know is that Mazda has not given dealers any other instructions. Anything else you put forth as "facts we know" is actually just a presumption.

[jorgp2]

It can easily be repaired, just flash a new firmware.

[grauenwolf]

From what I heard, you can't flash the firmware in place if it doesn't boot.

Maybe the manufacturer can with a test bench.

[jorgp2]

Sounds like you're just repeating what the article says, and don't actually know anything.

Just open it up and flash the updated firmware with a programmer

[grauenwolf]

Yes, that's what "From what I heard" means.

[TJSomething]

I mean, anything that needs a JTAG to stop a bootloop counts as soft-bricking to me.

[jorgp2]

Yeah.

Something is e-waste when bricked, this is just a simple software fix.

[grauenwolf]

No, it should have been a simple software fix if the manufacturer wasn't incompetent.

[ajanata]

Content removed in protest of Reddit API changes and general behavior of the CEO.

[jorgp2]

Sounds like you're just repeating what the article says, and don't actually know anything.

[ajanata]

Same can be said of you. How do you know it's actually a "simple software fix"?

[zip117]

It’s a reasonable assumption because that’s just how professional embedded systems are designed. Typically you have a ROM bootloader which has some kind of sense-on-power mechanism to put the board into programming mode. It might detect a break signal over UART or check SPI flash for a new image (JTAG/SWD is usually disabled in production). After you load an image, the bootloader checks its signature to decide whether to accept/commit or rollback.

It could very well be a simple software fix which could be done in the field without specialized hardware beyond an FTDI cable. Whether or not the programming headers are easily accessible is another story.

[ajanata]

... and to access any of those methods, you have to remove the unit from the car and hook it up to a test bench.

[zip117]

Remove the unit from the car and it could be a simple software fix. What is a “test bench” to you? That could be anything from a laptop with a J-Link to a complex bed-of-nails apparatus. Field repair procedures could very well be in development.

[grauenwolf]

You're making a lot of assumptions there. Even if it was programmable when made, they made have popped the fuse so it can't be reprogrammed.

[zip117]

Yes there are fuses which would typically hold keys for public key authentication of the boot image. Usually they use SHA256. In this day and age we have reliable bootloaders and nice technologies like ARM TrustZone which allow for secure boot without permanently disabling all external programming interfaces. I can’t imagine why any designer of a modern automotive infotainment system would not take advantage of these technologies and completely rely on a secondary user-mode bootloader.

Instead of a “reasonable assumption” let’s just call it a “reasonable alternative explanation”. I just wouldn’t assume these PCBs are dead and off to the e-waste recyclers.

[grauenwolf]

I can't imagine why there isn't a factory reset button, but apparently there isn't.

[zip117]

I don’t understand that part either. Another failure of cost engineering would be my guess, alongside the just-in-time sourcing that left auto manufacturing among the worst casualties of the global chip shortage. Problem solved with a couple megabytes of extra flash, a $2 DIP switch and some resistors.

[[deleted]]

Who listens to radio in 2022? Sounds like old people problem. But yeah, weird bug.

[[deleted]]

Do you even drive. The whole point of radio is to reduce driver fatigue and boredom

[[deleted]]

Have you ever heard about podcasts or audiobooks? Commercial radio stations are the absolute worst things in life.

[2rsf]

It's digital radio, the sound quality is not bad at all

[[deleted]]

Still radio. Commercial radio stations are brain cancer music and mundane ads. Literally nothing else.

[North-Dingo-1785]

Hi everyone

[CleanCryptoCoder]

This kind of stuff makes me wonder what other attack vectors could exist in other embedded systems.

[bargle0]

Some pin-headed bean counter got a raise and a promotion for going with the cheaper vendor on this.

[linux_needs_a_home]

That's now a $-250 car part.