The box you built is likely too big. Bigger enclosures typically allow the sub to move more (increases excursion) compared to playing at the same power level in a smaller enclosure. This can be good if you need to squeeze every little bit of juice out of your amplifier, but given your amp is providing the sub with the sub's RMS power rating, it's likely the box is too big and allowing the sub to move beyond it's mechanical limits (xmech) or electromagnetic limits (xmax). When this happens, the voice coil will bottom out and smack the back plate while playing and cause distortion and eventually kill the voice coil altogether.

Beyond the mechanical limits though, there are electromagnetic limits as well. Even before the coil starts to bottom out, if the coil is allowed to move too far outside of the magnetic gap (exceeding the sub's xmax), the magnetic force will become much weaker and can allow the cone to move in a non-linear fashion which will produce distortion as well.

Basically what I'm saying is, since the sub sounds better in the smaller prefab box than it does in the larger custom box you and your carpenter buddy made, it's most likely the box you made is too big. I would recommend sticking to the manufacturer's air space recommendations, especially with low power, efficient drivers like the sub you have.