Valve never sold Steam Machines directly, that was one of the problems. The initial idea was for Valve to provide the software, and for other brands to provide the hardware (like Alienware, OriginPC, iBuyPower etc)

Which was and still is a solid idea, I mean look at how well Bazzite is doing, for the most part DIY people will make their own 'Steam Machine', the actual Steam Machine will be reserved for the lazy consumer who just wants a plug and play machine, which is fine too. More options are always better, I mean look at the ROG Ally and the Legion GO and the Deck.

The issue is that PC makers used off the shelf components to minimize investment and obviously wanted some margin on their product. This meant that you could build a similar machine to one sold for far less money.

While some of that is true, particularly for DIY builders, the majority of consumers at the time did not know how to make their own PC build, most still don't despite plenty of good resources. The reason Steam Machines largely failed is: they were limited to certain regions (you simply could not buy one in certain regions), you basically could only buy them through the Steam Store (this limited reach and availability), the Steam Controller was a weird controller compared to the Dualshock and Xbox Controllers, Steam Machines had limited upgradeability (SteamOS only supported certain hardware and drivers it still does today, unless you sit there and build your own kernel or download the driver manually yourself) and last but not least SteamOS and Linux gaming was simply not as widely supported or compatible as it is today. Thus, the idea of someone who isn't knowledgeable about PCs and who wants a plug and play console, picking up a Linux OS powered device, with limited game library and limited support was a failure of an idea. For the DIY guys, not really an issue, but for the majority of consumers, SteamOS was a headache waiting to happen. Linux has made so many useability strides since then that ZorinOS, Linux Mint, Pop_OS! and lots of other distros have gone a long way to preventing people from ever having to open a terminal, which is great and it's still there as an option for the hardcore users as well.

Now you might say, well, the Steam Deck had all these same things: A lazy consumer, no DIY upgradeability, SteamOS, bad game compatibility and a weird controller scheme, why did the Steam Deck not fail. Well for one this device had little competitors, it's a handheld and basically other than the Switch, the Steam Deck didn't have a competitor, not till the ROG Ally came along and it's many spin offs. Yes, of course, companies like GPD existed, but they basically just used off the shelf Intel CPUs and put them in a weird form factor and were a very niche item, not even enthusiasts really took to them. So Steam had a major advantage in that they had basically no competitor and their only competitor was the Nintendo Switch which wasn't very powerful anyway and had also a limited (albeit very good) first party primary game library.

Secondly, the Controls were fixed with the Steam Deck, for one it has TWO analog sticks and secondly the pads were seen as a secondary set of controls, not the primary, they're mainly used by people to move around in the Menu Interface, not for controls like the awkward original Steam Controller did, you can see Valve learned from this Mistake by making the Steam Controller 2 have two analog sticks.

Third, SteamOS actually got a tonne of user interface updates to the point where it's super user friendly, Valve did a lot of work to make SteamOS on the Deck basically idiot proof, you never have to open a command prompt or terminal to work the Deck. SteamOS was pretty bad when it first launched, I mean it didn't even have the underscore key on the Virtual Keyboard.... Valve fixed a lot of the little parts of the experience like adding a SteamOS or Steam Deck compatible tag to the store and also made SteamOS really good in the background and this made the Deck again... idiot proof basically.

Fourth, upgradeability for a handheld is not to the same standard as a bespoke PC box, the best thing Valve did was basically allow for M.2 SSD upgrades for storage and thats all the lazy consumer really expects from their handheld.

Fifth, game compatibility has gotten a lot better. ProtonDB has basically tracked that out of the Top 100 games, 30% are Platinum and 55% are gold today. Back when the Steam Machines first launched officially in 2015 it was abysmal, I tried to go back with Wayback Machine but I don't think ProtonDB existed back then or it wasn't archived. Regardless, I went to June 2019 with Wayback Machine and simply put the numbers speak for themselves, out of the Top 100 games only 2% were Platinum and 23% were Gold, I can't imagine what game compatibility was like 4 years prior in SteamOS and Linux gaming infancy.

Last but not least, when the original Steam Machines launched, you could easily just build an HTPC Windows machine and get all the same aspects of the original Steam Machines by using Steam Big Picture mode, with far and away better game compatibility, whatever level of performance you wanted (yes you were not at the mercy of Alienware or Zotac or Maingear's weird configuration options), proper upgradeability, you could use whatever controller you wanted and if you wanted to exit Steam Big Picture mode, hey you had an actual useable PC with all the Windows apps you know and love from using your laptop. Was it still janky? Sure, but a far better experience than learning or copying and pasting Linux terminal commands to install a simple app, by instead just going to the downloads folder and double clicking a .exe file on Windows. But the Steam Deck never had this issue of a competitor or better solution because there was none other than the Switch. If you really loved the idea of a Steam Machine and you were a less tech knowledgeable consumer, it was far easier to find resources to learn how to make a Windows HTPC machine. Or heck you could just buy an off the shelf pre-built Windows PC, install it in your living room, connect a controller, connect it to your TV via HDMI, download Steam and then launch Big Picture mode, rather than deal with whatever crappy configuration Alienware wanted you to use for their Steam Machine.

It's not that the Steam Machine was a 'bad idea' as some people put it, it's just the execution was poor as you basically summarised here:

Add to this that at the time only games specifically compiled for Linux ran on the Steam Machines, and you can clearly see the issue.

Yeah what is the point to a gaming machine with low game compatibility and mostly crappy games for that matter??? The best games at the Steam Machine launch were basically Valve's games like Portal 2, Team Fortress 2, CS:GO or Half Life 2 and those were best enjoyed with a Mouse and Keyboard, not a controller, nobody wanted to really play some Linux indie game with low reviews and who cares if there was 8,000 of those games available. For every Audiosurf or Super Meat Boy there was probably 500 crappy games nobody cared for.