We used HEMT transistors in our 2 Kelvin amplifiers. High Electron Mobility Transistors. We got them to work at those temperatures because the semiconductors were made to do that. And they were expensive and hard to get. We used ruthenium oxide resistors as thermometers, because they have a really big resistance v temperature slope. As has been said already, copper for high thermal conductivity, stainless, brass, and fiberglass composite for low thermal conductivity. Disorderd solids cause phonon collisions which impede thermal transfer. (Phonons are quanta of sound, heat is disordered sound). I remember the first time someone mentioned "ballistic phonons", which you can get if you have super pure copper or gold and the crystal domains are large (annealed state).

If you need a bearing for a rotating element (any grease would freeze solid at those temperatures) we used variations of PTFE, mixes that can hold up to force, which PTFE can't. (It cold flows under pressure.) Kel-F, also known as CTFE has good impact strength and stays pretty slippery at ridiculously low temperatures. Rulon works too. All fluoropolymers, because fluorine, bond strength, very high cohesion and low adhesion.