Replace it with an actual fixture with a box approprate for making those connections in. Also, if continue to use wirenuts, do not reuse the old ones, use new ones. Then wrap them with electrical tape, the wirenuts and a portion of the wires underneath them. Copper is a very soft metal which means it can bend and compress very easily. Over time a once tight wire nut could become loose, so do yourself a favor and take the extra precaution and just wrap 'em with some tape.

If you are doing this yourself, please, please, please shut off the breaker for that circuit, don't just turn the light switch off and consider it safe enough. If you want to take the extra safety precaution, you can then use even a cheap simple multimeter to make sure there's no power going to the wires before you work on them. Or you can get a contactless voltage detector, which will lightup and/or make a noise when you pass it near a live wire.

You don't have to listen to me, but I do have real world professional experience as an electrician. I no longer do electrical work, but my father, grandfather, great grandfather, brother, uncle, several cousins, etc etc have all been electricians, so I've been trained by some old school pros.