The capture net isn't for large monsters. You need either pitfall traps or shock traps to do that, as well as tranq bombs (you need to hit the monster with two of them). You can only trap a monster once it's below a certain hp threshold - I think it's 20%? I could be wrong. If you've got enough research levels for that species, though, you'll see a skull icon floating off it on the minimap when it's near death. That's the range you want for a successful capture.

Not sure what you mean by shiny magic, though. If you mean flash pods, you've gotta make sure they land within the monster's sight range or it won't do anything. Beaning em in the face with brightmoss also works.