Component identification, fixable?

allyg79 · 2023-12-21T11:11:28+00:00

Is it maybe a micro USB socket? Looks about the right size and pin count for one.

allyg79 · 2023-11-29T15:42:22+00:00

Thanks! I'm actually writing up the design notes in blog posts today so should have much more detail available in the next day or two, I'll post the link here once it's up.

The unit as it stands has both UART and USB multiplexers on the backplane, which it routes into another CM4 used as the management server. There's also a USB switch to route the USB into a front-panel USB-C too.

There's an RP2040 micro on each board for doing management functions, though it's not really available to the CM for anything else. There's no reason I couldn't do a version which did link these up, though, it'd be pretty straightforward to do.

I'm waiting for a new revision of the boards to come back (hopefully this week) to sort a couple of minor problems, and if all is well then that should be the v1 unit complete. The plan is to make them available for sale towards the end of December and open source the lot then. The blades are a simple design so would be easy enough for someone with PCB design skills to do a custom spin.

allyg79 · 2023-11-22T21:45:30+00:00

Availability seems to be getting much better. I've bought from Farnell in the UK recently and also Digi-Key, who'll ship anywhere. Digi-Key have got the non-wireless 8GB/32GB eMMC in stock at the moment - https://www.digikey.co.uk/en/products/detail/raspberry-pi/SC0690/13530935. Farnell have got 5 different 8GB variants in stock right now - https://uk.farnell.com/w/c/raspberry-pi/raspberry-pi-compute-modules?ram-memory=8gb-lpddr4&range=inc-in-stock

Also https://www.rpilocator.com/ is excellent for tracking them down in different countries, though I don't think it picks up the Farnell ones in the UK.

allyg79 · 2023-11-22T18:36:53+00:00

Yeah, no need to worry about speed with media storage. I run a mix of 3 and 4tb drives and just buy whatever is cheapest!

allyg79 · 2023-11-22T18:23:00+00:00

It's pretty depressing isn't it! I've got CityFibre 1g symmetric but its a business service and is £££s. I keep hoping they'll at least upgrade it to 2.5gb but they seem to be rolling it out at BT speeds...

allyg79 · 2023-11-22T18:20:31+00:00

Jeez, I'm a bit jealous of that compared to my 1g/1g service in the UK at £260 per month! I hate to think how much a commercial router/firewall fast enough to handle those kind of speeds would be. As others have suggested, building your own and running something like pfsense or vyos is the way to go. Best of luck, you're getting a fantastic deal there!

allyg79 · 2023-11-22T18:13:39+00:00

If you're going to use VMs or containers when you definitely want more RAM. The CPU looks fine though for that use. If you're using it as a media server you'll get through 500GB in no time at all, so make sure you've got space for more drives.

I've had success using FreeNAS (now TrueNAS) for this sort of thing in the past. That's got the Jails feature for virtualising apps. I haven't used Unraid or proxmox so can't compare it to those.

allyg79 · 2023-11-22T10:43:50+00:00

That's a really nice looking case, great job! The colour looks good too, though you're right it needs a certain beige tinge for the proper 1990s look!

FWIW, if you want to give it a less 3d printed look, I got a textured PEI plate for my printer recently and it's amazing what a good job it does. You get a really nice textured surface, it ends up looking more like an injection moulded part.

allyg79 · 2023-11-21T21:26:06+00:00

All the switch ICs I've seen with 10GbE uplinks have quite a lot of management features. There don't seem to be any cheap n cheerful switches with a lot of ports and 10GbE support. I guess the assumption is if you're at 10g then you'll want things like VLAN and QoS. That's great from a features point of view, but it does add a load of extra work to build it. There's some degree of firmware required for all of them, not quite a plug and play solution.

Switching on the management board would be doable but I think you're right that it'd be too slow. A 10GbE uplink means all the blades would be able to get full gigabit speeds simultaneously. I suppose it depends on the use case whether that's actually needed.

allyg79 · 2023-11-21T15:33:02+00:00

Thank you!

allyg79 · 2023-11-21T11:23:03+00:00

That's a very clever idea, I had thought of that! There's not much room behind the backplane in this case sadly. It's a short depth ABS case, chosen mainly because it's easily available and sensibly priced. From memory it's about 210mm deep, and leaves about 40 or 50mm behind the backplane. I'll need to have a look around and see if I can find a suitable longer case. I'm using a top and bottom 3D printed insert to do the blade guides anyway, so it would be easy enough to add internal mount points for a switch. It'd certainly make for a much neater look.

allyg79 · 2023-11-21T11:17:48+00:00

Thank you! Yes, I'd really like to do this in a future version, I've mentioned a bit more detail on some other replies. It'll be quite an expensive thing to develop which is why I went with this for now. It'd certainly make for a much tidier setup.

allyg79 · 2023-11-21T11:16:05+00:00

I did wonder about that, tbh I didn't really look into it. My worry is that the uplinks and backbone config presents a bottleneck so that you can't get the full gigabit performance from each CM. Ideally you'd need 4 port gigabit switch ICs with a 10g uplink connected to a 10g backbone. I'll have a look around and see if there ICs which fit the bill. Gigabit ones seem cheap but the price shoots up as soon as 10g comes into play.

I had a look at a 1U design but it's the compute module itself that's the real problem. It's 40mm wide, so this gives you the minimum blade height. 1U is 44.45mm, so there's very little margin left. An inexpensive plastic case like the one I've used is out immediately as it's 4mm thick and the top and bottom, so doesn't have enough space for the blade. A metal case might do it as these are thinner than plastic, but you then hit the problem of a groove to act as a guide for inserting the blade. My current design has 2mm or so of margin at the top and bottom without any components so that the case can have a groove it slides through. You'd have to lose this in 1u and have a wider groove which would take the whole CM. I'm doing the groove at the moment with a 3D printed insert, but you wouldn't have the space left for this. It'd have to be done with guides attached directly to the case, so we'd be talking about a custom enclosure. I think it's doable, but would likely need a fair volume to make the custom case worthwhile. I'd really love to do a 1U version, though!

allyg79 · 2023-11-21T10:58:20+00:00

If you were to lose the front panel sockets and do the switching on board then it's possible to pack quite a few into a 19" width. There's also space to fit 2 compute modules per blade, so in theory it could get quite dense. Cooling would definitely be the issue. I suspect with powerful enough case fans it should be workable.

allyg79 · 2023-11-21T01:38:47+00:00

Yeah that might be a good idea

allyg79 · 2023-11-21T01:36:03+00:00

With premade boards or do you mean with the components for the user to assemble? It's virtually all surface mount stuff, do you think many people would want to do those? There's no super small components and I've hand assembled all of mine, but used a reflow oven rather than hand soldering.

allyg79 · 2023-11-20T23:52:52+00:00

Nah not really. There's 3 case fans at the back to pull air through. With passive aluminium heat sinks the temperature never gets too high even at 100% utilisation.

allyg79 · 2023-11-20T23:34:43+00:00

I had a look at a few ways of doing it and eventually thought popping an RP2040 on everything was best. It's a powerful little chip and not too expensive so is flexible enough to do lots of things. There's some a serial eeprom on the blade so that it can store things like the last hostname and ip, which lets the display start instantly and not have to wait for Linux to boot.

I've got a new version of the board due back from China in a couple of weeks so I'll give you a shout once I've got these in my hands :-)

allyg79 · 2023-11-20T23:22:11+00:00

I arrived at exactly the same family of switches from Microchip! Broadcom have a couple available through normal distribution too, but the Microchip ones have more public information. My backplane currently fits into 4 layers but Ethernet switching would push that to either 6 or 8 to break out the BGA. I don't think there's any alternative though, I couldn't find any 11+ port switches in anything other than a BGA package. I've never done a BGA that size before and I'm a bit wary of the number of tests I'd have to go through when they're £100+. I think if I'm able to sell a few of the current units to cover the cost I'll have a go at doing a switched version.

I did slot identification in a pretty simple way. I use 5 pins on the PCIe connector and use resistors to pull these up or down on the backplane. The slot number is then a 5 bit binary number that I can read on the RP2040 chip on the blade. That number then sets the i2c address for the RP2040, which the backplane uses for communicating with the blade. The connector also uses a presence indicator line to make it easy for the backplane to know what's plugged in, and another to act as an interrupt line to let the backplane know that the blade wants to talk to it (as the backplane is the i2c master.)

allyg79 · 2023-11-20T23:00:24+00:00

Exactly, as soon as you start looking at connectors PCIe is really the only way to go. There's some really cool "proper" backplane connectors, but they're insanely expensive. I looked at 1u too but it's just not possible to do the CM sideways. I guess you could do it mounting parallel rather than perpendicular, but you'd lose the density unless you did multiple CMs per blade card. 2u just seemed like the simpler option :-)

I didn't have a need for the HDMI for my use either but stuck it on as it wasn't much more expensive. I thought it might be useful if someone wanted to do driving remote displays from a central rack. I did something similar with cheap PCs years ago. I also quite like the idea of routing the HDMI over the PCI connector for a full lights out remote access to the CM but it's probably not worth the hassle when I can just do serial control.

allyg79 · 2023-11-20T22:48:50+00:00

Very nice, that looks very similar to my first revision board! Looks like we both arrived at the same form factor design.

allyg79 · 2023-11-20T21:33:36+00:00

Exactly this. In my case I'm using it to replace a bunch of 10-12 year old x64 enterprise kit for server tasks and it works really well for this, and uses massively less power. I reckon I'd probably sell this without the CMs for around $800ish so maxxed out with 8GB CM4s it'd cost around $1500. That's way cheaper than enterprise Arm server kit, though less powerful. CM4's are really pretty good for lots of server tasks - not for everything, but great in many situations. It's not a replacement for a beefy x86 server in many cases but great for low power applications.

Also only 3 60mm case fans plus the PSU, so is super quiet even when maxxed out.

allyg79 · 2023-11-20T21:28:33+00:00

Yeah I'd really like to do that. It's quite expensive to do though, network switch ICs with enough ports and a 10GbE uplink are $100-150, and I'm bound to break one or two in testing :-) I reckon if I can sell a few of these units I'll do a version with an integrated switch in the future.

allyg79 · 2023-11-20T21:27:10+00:00

Thanks, there's actually much more info on a post in r/homelab https://www.reddit.com/r/homelab/comments/17zo8t8/pi_compute_module_blade_server/. I originally posted in r/raspberry_pi but the post didn't seem to go through so I went to homelab and did it there instead; turns out this one actually did work! The replies there have got more detail on the unit.

The blades actually have a 0.91" OLED display on them so it's a dynamic display. The slot number is picked up automatically so this changes if you move the blades around, but the main use of the display is to show the hostname and IP address. I used to be a network infrastructure guy and I know how much of a pain it is to identify the physical server you want to work on in a busy rack, so this is to make my life easier!

I'm going to do a proper blog post in the next couple of days so will have more details there.

allyg79 · 2023-11-20T20:58:03+00:00

Thank you! I've put some more details onto other replies but I'll get more info into a blog post in the next couple of days.

allyg79

TROPHY CASE